A heterogeneous war game deduction decision-making method, device and equipment based on LLM-MARL collaborative driving and a storage medium

Abstract

This invention provides a heterogeneous wargame simulation decision-making method, apparatus, device, and storage medium based on LLM-MARL collaborative driving. By extracting features and quantifying military terminology from the battlefield situation, and generating structured situation text according to a protocol template, the LLM strategic planner outputs a tactical intent vector, achieving high-level semantic mapping to a tensor space. Using the tactical intent vector and local agent observations as query and key-value sources respectively, an intent-guided feature is generated via a cross-attention network, driving the policy network to output actions, thus guiding the underlying actions based on strategic intent. The actions and observation trajectories are mapped to a shared semantic space, and joint training combining consistency loss and environmental rewards constrains the convergence of tactical trajectories towards strategic intent. Furthermore, reflective feedback is generated based on execution deviation, driving the LLM to update the tactical intent vector, forming a two-way collaborative optimization closed loop.

Description

Technical Field

[0001] This invention relates to the field of wargaming, and in particular to a heterogeneous wargaming decision-making method, apparatus, device, and storage medium based on LLM-MARL collaborative driving. Background Technology

[0002] Wargaming, as a key means of studying combat patterns and assisting command and decision-making, has gradually become a core direction in the field of military simulation through its intelligentization. Currently, decision-making systems applied to heterogeneous wargaming are mainly developing along two technical paths: rule-based expert systems and end-to-end policy networks based on deep reinforcement learning (DRL, such as PPO, QMIX, MAPPO, etc.). The former relies on prior rule bases for reasoning, offering strong interpretability, but struggles to handle dynamic adversarial situations with incomplete information and high-dimensional action spaces; the latter drives policy iteration through environmental rewards, possessing strong nonlinear fitting capabilities at the micro-operational level, and has been widely used in multi-agent cooperative control.

[0003] To alleviate the shortcomings of purely data-driven solutions in long-term planning, a two-layer architecture has emerged in recent years, combining a Large Language Model (LLM) as a high-level planner with a low-level reinforcement learning executor. Specifically, the LLM reads the situation description and outputs strategic instructions in natural language form, which are then executed by the policy network within the environment. While this architecture introduces high-level semantic priors to some extent, a difficult-to-reconcile coupling exists between the high-level semantic planning and the low-level numerical control in heterogeneous wargame scenarios: the high-level LLM abstracts the battlefield situation using discrete symbol sequences, and its output is natural language text oriented towards human reading habits; while the low-level multi-agent policy network samples and updates gradients in a continuous high-dimensional tensor space, and its optimization objective is strictly constrained by the immediate reward function of the environment. The two are from different sources in terms of representation carrier, optimization gradient and time scale. Even after the LLM output is mapped into an instruction vector by a text encoder and concatenated to the observation of the policy network, the action distribution of the bottom agent will still drift freely along the reward gradient direction in the process of pursuing the maximization of environmental rewards. This causes the actual tactical trajectory formed after multiple executions to gradually deviate from the strategic intent initially issued by the LLM in the latent space. At the same time, failure states such as physical inaccessibility, sudden increase in battle losses and time delay generated in the bottom execution process reside in the policy network in the form of high-dimensional tensors. They cannot be re-perceived and utilized by the high-level LLM in the symbol space. This makes the "high-level planning - bottom execution" link appear as a one-time unidirectional open loop in a single game simulation.

[0004] In view of the above, this application is hereby submitted. Summary of the Invention

[0005] This invention discloses a method to address the problem in heterogeneous wargame simulations where LLM strategic semantics and multi-agent policy networks gradually deviate from tactical trajectories and strategic intentions in the latent space due to heterogeneous representation spaces, misaligned optimization targets, and unidirectional open-loop execution feedback links.

[0006] The first embodiment of the present invention provides a heterogeneous wargame inference and decision-making method based on LLM-MARL collaborative driving, including: The battlefield situation in the current round is feature-extracted to obtain a situation feature matrix. Key situation indicators are extracted from the situation feature matrix and quantified into military terminology levels. They are then assembled into structured situation text according to a preset protocol template. The structured situational text is input into the LLM strategic planner, which then outputs a tactical intent vector. For each of the multiple heterogeneous agents, the tactical intent vector is used as the query source and the agent's local observation at time t is used as the key value source. An intent guidance feature is generated through a cross-attention network and the intent guidance feature is input into the policy network corresponding to the agent. The policy network then outputs the action. The action output by the policy network at multiple consecutive time points and the corresponding local observations are collected to form a trajectory segment. The trajectory segment is mapped to the semantic latent space shared with the tactical intent vector to obtain tactical trajectory features. The policy network is trained with the consistency loss Lalign between the tactical trajectory features and the tactical intent vector and the environment reward. The execution deviation generated by the action is monitored. When the execution deviation exceeds a preset threshold, the failure features at the corresponding moment are extracted and transformed into reflection text. The reflection text is fed back to the LLM strategic planner to update the tactical intent vector for the next round, thus forming a two-way collaborative loop between the LLM strategic planner and the policy network.

[0007] The second embodiment of the present invention provides a heterogeneous wargame simulation and decision-making device based on LLM-MARL collaborative driving, comprising: The semantic engine unit is used to extract features of the battlefield situation in the current round to obtain a situation feature matrix, extract key situation indicators from the situation feature matrix and quantify them into military terminology levels, and assemble them into structured situation text according to a preset protocol template. The tactical intent vector generation unit is used to input the structured situation text into the LLM strategic planner, and the LLM strategic planner outputs the tactical intent vector. The heterogeneous execution unit is used to generate intent guidance features for each of the multiple heterogeneous agents, using the tactical intent vector as the query source and the local observation of the agent at time t as the key value source, through a cross-attention network, and input the intent guidance features into the policy network corresponding to the agent, so that the policy network outputs the action. The consistency alignment unit is used to collect the actions and corresponding local observations output by the policy network at multiple consecutive time points to form trajectory segments, map the trajectory segments to the semantic latent space shared with the tactical intent vector to obtain tactical trajectory features, and train the policy network with the consistency loss Lalign between the tactical trajectory features and the tactical intent vector and the environment reward. An asynchronous feedback unit is used to monitor the execution deviation generated by the action. When the execution deviation exceeds a preset threshold, the failure features at the corresponding moment are extracted and converted into reflection text. The reflection text is fed back to the LLM strategic planner to update the tactical intent vector for the next round, thus forming a bidirectional collaborative loop between the LLM strategic planner and the policy network.

[0008] The third embodiment of the present invention provides a heterogeneous wargame simulation and decision-making device based on LLM-MARL collaborative driving, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to realize the heterogeneous wargame simulation and decision-making method based on LLM-MARL collaborative driving as described in any of the above embodiments.

[0009] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement the heterogeneous wargaming simulation and decision-making method based on LLM-MARL collaborative driving as described in any of the above embodiments.

[0010] Based on the heterogeneous wargame simulation decision-making method, apparatus, device, and storage medium driven by LLM-MARL collaboration provided by this invention, the battlefield situation is extracted by features and quantified by military terminology, and then assembled into a structured situation text according to a preset protocol template. This allows the LLM strategic planner to read and output tactical intention vectors, thereby mapping the high-level semantics of the symbolic space to the tensor space. Using the tactical intention vector as the query source and the agent's local observations as the key source, the intention-guided features generated by the cross-attention network drive the policy network to output actions, allowing strategic intentions to directly intervene in the generation of underlying actions. The trajectory segments composed of actions and observations at multiple consecutive moments are mapped to the semantic latent space shared with the tactical intention vector to obtain tactical trajectory features. The consistency loss Lalign between the tactical intention vector and the tactical trajectory features is used in conjunction with environmental rewards for training, constraining the tactical trajectory to converge toward the strategic intention. By monitoring the execution deviation and transforming failure features into reflective texts to feed back to the LLM strategic planner to update the tactical intention vector for the next round, a bidirectional collaborative loop between the LLM and the policy network is formed. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a heterogeneous wargaming simulation and decision-making method based on LLM-MARL collaborative driving, provided in the first embodiment of the present invention. Figure 2 This is a schematic diagram of a heterogeneous wargaming simulation and decision-making device based on LLM-MARL collaborative driving, provided in the second embodiment of the present invention. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0013] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0014] This invention discloses a method to address the problem in heterogeneous wargame simulations where LLM strategic semantics and multi-agent policy networks gradually deviate from tactical trajectories and strategic intentions in the latent space due to heterogeneous representation spaces, misaligned optimization targets, and unidirectional open-loop execution feedback links.

[0015] Please see Figure 1The first embodiment of the present invention provides a heterogeneous wargame simulation decision-making method based on LLM-MARL collaborative driving. The method is applied to a decision-making system composed of an LLM strategic planner and a policy network of multiple heterogeneous agents. The method includes: S101, extract features from the battlefield situation of the current round to obtain a situation feature matrix, extract key situation indicators from the situation feature matrix and quantify them into military terminology levels, and assemble them into structured situation text according to a preset protocol template. In this embodiment, firstly, features are extracted from the battlefield situation of the current round to obtain a situation feature matrix. Specifically, a situation map is constructed using each combat entity in the battlefield as a node and the spatial adjacency relationship between entities as an edge. The node attributes in the situation map include basic information such as the entity's position coordinates, remaining resources, and current state. Then, the situation map is processed through multiple rounds of message passing and feature aggregation using a graph neural network, and finally, a situation feature matrix is ​​output. The situation feature matrix contains topological relationship information between entities, terrain obstacle information, and fire coverage information, thereby completing the transformation of the original battlefield situation from a discrete set of entities to a structured vector representation. Subsequently, key situation indicators are extracted and discretized from the situation feature matrix. Specifically, a predefined effectiveness evaluation operator extracts the ratio of friendly and enemy firepower density within a preset grid as a win rate evaluation indicator from the situation feature matrix. A state retrieval operator extracts resource indicators, including ammunition and fuel reserves, from entity nodes as resource reserves as resource reserves. An intent recognition operator extracts the movement trajectory of enemy entities in historical rounds and their orientation in the current round as deployment tendency indicators from the situation feature matrix. Then, a preset discretization mapping function maps the win rate evaluation indicator, resource reserves indicator, and deployment tendency indicator to corresponding military term levels. For example, the win rate evaluation indicator is mapped to "advantage / equilibrium / disadvantage", the resource reserves indicator is mapped to "sufficient / tense / critical", and the deployment tendency indicator is mapped to "assembly / transfer / retreat", so that continuous numerical values ​​are converted into discrete semantic symbols that can be directly read by the language model. Finally, the quantified military terminology levels and key parameters are assembled according to a preset protocol template to obtain a structured situation text. Specifically, the structured situation text includes a round identifier field, a core mission field, a local situation field, and a heterogeneous service pool field. The round identifier field is filled with the current round number, the core mission field is filled with the core combat mission identifier of the current round, the local situation field is filled with the quantified military terminology levels, and the heterogeneous service pool field is filled with the scale of currently available reconnaissance and strike units, ground armored units, and electronic warfare units according to the service performance constraint type. This outputs a structured situation text that can be directly read and used for thought chain reasoning by the subsequent LLM strategic planner, completing the end-to-end conversion from numerical situation to semantic situation.

[0016] S102, The structured situation text is input into the LLM strategic planner, and the LLM strategic planner outputs a tactical intent vector; In this embodiment, structured situational text is input into the LLM strategic planner. The LLM strategic planner is based on a pre-trained large language model and further fine-tuned with domain corpus, enabling it to perform semantic parsing of each field in the structured situational text and multi-step reasoning based on battlefield context. Specifically, the LLM strategic planner first performs field-by-field semantic parsing of the round identifier field, core task field, local situational field, and heterogeneous unit pool field in the structured situational text to identify the current simulation stage, the required mission objectives, the situational comparison between the enemy and friendly forces, and the scale of available heterogeneous combat units. Then, based on the thought chain reasoning mechanism, it gradually simulates the current situation according to the logical chain of "situational analysis, objective establishment, force deployment, and action sequence," thereby deriving the relative situation of the enemy and friendly forces, the sub-objectives that need to be prioritized in this round, the collaborative allocation scheme of various heterogeneous units, and the corresponding action directions. After completing the above thought chain reasoning, the LLM strategic planner maps the reasoning results into tactical intent vectors. The tactical intent vector The system includes at least three dimensions: target area encoding, task priority, and coordination constraints. The target area encoding indicates the battlefield area to be prioritized in this round; the task priority indicates the relative weight of each sub-target in the execution order; and the coordination constraints indicate the temporal and spatial cooperation relationships between different unit types. To further constrain the search range of the underlying strategy network, the LLM strategic planner also generates corresponding sub-instructions G for each type of unit performance constraint. j j is the index of the service type performance constraint type, and the sub-instruction G jThe specific action requirements for the corresponding unit type in this round are expressed in natural language, and then the sub-instructions G are extracted by a semantic feature extractor. j Encode it as a fixed-length command vector Eintent, such that the fixed-length command vector Eintent is related to the tactical intent vector. Together they form a policy-available network The high-level semantic guidance signals are directly consumed, thereby completing the end-to-end conversion from structured situational text to tactical intent vectors.

[0017] S103, for each of the multiple heterogeneous agents, using the tactical intent vector as the query source and the agent's local observation at time t as the key value source, an intent guidance feature is generated through a cross-attention network, and the intent guidance feature is input into the policy network corresponding to the agent, and the policy network outputs the action. In this embodiment, for each agent i among multiple heterogeneous agents, a tactical intent vector will be used. Local observations of agent i at time t The common input is sent to the policy network corresponding to agent i. The local observations mentioned above This includes the location coordinates of the intelligent agent i itself, remaining resources, current terrain, and information on friendly and enemy entities within its field of vision; Specifically, the policy network Internally, a multi-head cross-attention network is configured to achieve semantic alignment between high-level strategic semantics and low-level local observations. This multi-head cross-attention network then integrates the tactical intent vector... The query matrix is ​​obtained through linear transformation. The local observation The bond matrix is ​​obtained through the first linear transformation. The local observation The value matrix is ​​obtained by the second linear transformation. Then calculate the intended guiding features using the following formula:

[0018] in The key matrix The column vector dimension is , softmax(·) represents the normalization exponential transformation applied to each row of the input matrix, and T represents the matrix transpose operation; the multi-head cross-attention network queries the matrix AND key matrix The transpose of the matrix is ​​used to perform a dot product operation and then normalized by softmax to obtain the attention weights. These attention weights are then combined with the value matrix. Multiplication results in the amplification of the portion of the local observations of agent i that is highly relevant to the current tactical intent and the suppression of the portion that is irrelevant to the current tactical intent, thereby obtaining an intent-guided feature focused on strategic intent. Furthermore, the intent-guided features and the local observations are... The hidden state of the historical actions of the intelligent agent i at time t. and the fixed-length instruction vector corresponding to the performance constraint type of the unit to which the intelligent agent i belongs. Common input to the policy network The policy network Output action by pressing the formula :

[0019] The historical action hidden state Used to carry the historical decision context of agent i before time t, so that the policy network Timing consistency is considered when generating the current action; the fixed-length instruction vector Then, through the gated loop unit and the policy network The internal local features are fused to make the policy network The search range in the action space is limited to sub-instruction G. j Within the allowed subset, actions are output that conform to strategic intent and unit performance constraints. This completes the semantic transformation from tactical intent vectors to specific actions.

[0020] S104, collect the actions and corresponding local observations output by the policy network at multiple consecutive time points to form trajectory segments, map the trajectory segments to the semantic latent space shared with the tactical intent vector to obtain tactical trajectory features, and train the policy network with the consistency loss Lalign between the tactical trajectory features and the tactical intent vector and the environmental reward. In this embodiment, a sliding window of length k is taken with the current time t as the endpoint, and the data is collected by the policy network. The actions output within the sliding window and the corresponding local observations constitute a trajectory segment. Where k is the preset sliding window length, and the trajectory segment The state-action sequence of the agent i over the most recent k time intervals has been fully recorded; Subsequently, the trajectory segment is processed by a trajectory encoder. Multimodal temporal coding is performed; specifically, the trajectory encoder encodes the trajectory segment. The local observation sequence is compressed using state semantics to obtain a state embedding sequence, which is then used to analyze the trajectory segment. The action intent sequence is obtained by aggregating the action sequences in the data. Then, the state embedding sequence and the action intent sequence are fused and encoded in the time dimension to output the original trajectory vector that can characterize the recent tactical behavior pattern of agent i. The trajectory encoder can be implemented by a gated recurrent unit or a temporal convolutional network; furthermore, through a projection function... The original trajectory vector Projected onto the tactical intent vector The shared semantic latent space yields tactical trajectory features. :

[0021] The projection function fproj is implemented by a multilayer perceptron, which enables the tactical trajectory features to... With the tactical intent vector They possess directly measurable semantic comparability within the same latent space; After obtaining the tactical trajectory features Then, the tactical trajectory features are calculated using the following formula. With the tactical intent vector Consistency loss between :

[0022] in For the tactical trajectory features With the tactical intent vector The inner product, ‖·‖, represents the L2 norm of the vector; the consistency loss The smaller the value, the closer the actual tactical behavior pattern of agent i in the most recent k time intervals is to the strategic intent direction issued by the LLM strategic planner; conversely, the larger the value, the closer the actual behavior has deviated from the strategic intent in the latent space. Finally, the consistency loss is... The overall training objective of the policy network is formed by weighting and summing the environmental reward terms obtained by the policy network based on environmental feedback. The parameters of the policy network are then updated using gradients according to this overall training objective, so that while maximizing environmental rewards, the policy network also generates tactical trajectory features. Continuously moving towards the tactical intent vector in the latent space The strategic directions represented converge, thereby achieving implicit alignment between high-level strategic intentions and low-level tactical execution.

[0023] S105, monitor the execution deviation generated by the action. When the execution deviation exceeds a preset threshold, extract the failure features at the corresponding moment and convert them into reflection text. Feed the reflection text back to the LLM strategic planner to update the tactical intent vector for the next round, thus forming a bidirectional collaborative loop between the LLM strategic planner and the policy network.

[0024] In this embodiment, the heterogeneous intelligent agent executes the output action. During the process, the actual displacement generated by the intelligent agent i is continuously monitored to determine whether the current execution process deviates from the issued tactical intent vector. Specifically, at a certain moment in the current round Taking a monitoring window of length ΔT as the starting point, the deviation POR is calculated within the monitoring window using the following formula:

[0025] in The start time of the monitoring window is [time]. The duration of the monitoring window, For the agent i at time t, based on the action The resulting actual displacement vector, The tactical intent vector The indicated predetermined direction vector of motion, Let |t| be the expected displacement vector at time t, where |·| represents the absolute value for the inner product operation and the L2 norm for the vector; when the monitoring window... When the execution deviation (POR) within the specified range is continuously greater than a preset threshold of 0.9, it is determined that the execution deviation exceeds the preset threshold, and a feedback interruption is triggered. After triggering the feedback interruption, the failure features at the corresponding moment are extracted from multiple dimensions and transformed into reflective text. Specifically, the spatial conflict dimension features are extracted from the failure features at the corresponding moment using a collision detection operator and transformed into path obstruction semantic text, such as "the maneuvering unit cannot pass through the designated area for multiple consecutive steps". The effectiveness loss dimension features are extracted from the failure features at the corresponding moment using a damage increment assessment operator and transformed into rapid battle damage semantic text, such as "the remaining resources decrease by more than a preset proportion per unit time". The temporal offset dimension features are extracted from the failure features at the corresponding moment using a mission time counter and transformed into... Timeliness violation semantic text, such as "the actual arrival time of the predetermined task node is later than the planned time"; at least one of the path obstruction semantic text, the battle damage too fast semantic text, and the timeliness violation semantic text is encapsulated in the order of an anomaly type field, agent identifier field, quantization reason field, and environment label field to obtain reflection text. The anomaly type field is filled with the category identifier of the dimension to which the triggered failure feature belongs. The agent identifier field is filled with the identifier of agent i that triggered the failure feature. The quantization reason field is filled with the quantization value of the triggered failure feature. The environment label field is filled with the environment type identifier where the triggered failure feature is located. Finally, the reflection text is input as a feedback prompt into the LLM strategic planner. The LLM strategic planner further combines the reflection text with the original structured situation text to perform re-reasoning, thereby outputting a revised tactical intent vector for decision-making in the next round. This forms a two-way collaborative loop between the LLM strategic planner and the strategy network, enabling the entire decision-making system to have the adaptive ability to dynamically reshape strategic intent based on actual deviations in long-term adversarial simulations.

[0026] Please see Figure 2 The second embodiment of the present invention provides a heterogeneous wargame simulation and decision-making device based on LLM-MARL collaborative driving, comprising: The semantic engine unit 201 is used to extract features of the battlefield situation in the current round to obtain a situation feature matrix, extract key situation indicators from the situation feature matrix and quantify them into military terminology levels, and assemble them into structured situation text according to a preset protocol template. The tactical intent vector generation unit 202 is used to input the structured situation text into the LLM strategic planner, and the LLM strategic planner outputs the tactical intent vector. The heterogeneous execution unit 203 is used to generate intent guidance features for each of the multiple heterogeneous agents, using the tactical intent vector as the query source and the local observation of the agent at time t as the key value source, through a cross-attention network, and input the intent guidance features into the policy network corresponding to the agent, so that the policy network outputs the action. The consistency alignment unit 204 is used to collect the actions and corresponding local observations output by the policy network at multiple consecutive time points to form trajectory segments, map the trajectory segments to the semantic latent space shared with the tactical intent vector to obtain tactical trajectory features, and train the policy network with the consistency loss Lalign between the tactical trajectory features and the tactical intent vector and the environment reward. The asynchronous feedback unit 205 is used to monitor the execution deviation generated by the action. When the execution deviation exceeds a preset threshold, the failure features at the corresponding moment are extracted and converted into reflection text. The reflection text is fed back to the LLM strategic planner to update the tactical intent vector for the next round, thus forming a bidirectional collaborative loop between the LLM strategic planner and the policy network.

[0027] The third embodiment of the present invention provides a heterogeneous wargame simulation and decision-making device based on LLM-MARL collaborative driving, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to realize the heterogeneous wargame simulation and decision-making method based on LLM-MARL collaborative driving as described in any of the above embodiments.

[0028] The fourth embodiment of the present invention provides a computer-readable storage medium storing a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to implement the heterogeneous wargaming simulation and decision-making method based on LLM-MARL collaborative driving as described in any of the above embodiments.

[0029] Based on the heterogeneous wargame simulation decision-making method, apparatus, device, and storage medium driven by LLM-MARL collaboration provided by this invention, the battlefield situation is extracted by features and quantified by military terminology, and then assembled into a structured situation text according to a preset protocol template. This allows the LLM strategic planner to read and output tactical intention vectors, thereby mapping the high-level semantics of the symbolic space to the tensor space. Using the tactical intention vector as the query source and the agent's local observations as the key source, the intention-guided features generated by the cross-attention network drive the policy network to output actions, allowing strategic intentions to directly intervene in the generation of underlying actions. The trajectory segments composed of actions and observations at multiple consecutive moments are mapped to the semantic latent space shared with the tactical intention vector to obtain tactical trajectory features. The consistency loss Lalign between the tactical intention vector and the tactical trajectory features is used in conjunction with environmental rewards for training, constraining the tactical trajectory to converge toward the strategic intention. By monitoring the execution deviation and transforming failure features into reflective texts to feed back to the LLM strategic planner to update the tactical intention vector for the next round, a bidirectional collaborative loop between the LLM and the policy network is formed.

[0030] Exemplary examples show that the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in implementing a heterogeneous wargaming simulation and decision-making device based on LLM-MARL collaborative driving. For example, the apparatus described in the second embodiment of the present invention.

[0031] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the aforementioned heterogeneous wargame simulation and decision-making method based on LLM-MARL collaborative driving, connecting various parts of the method through various interfaces and lines.

[0032] The memory can be used to store the computer program and / or modules. The processor, by running or executing the computer program and / or modules stored in the memory, and by calling the data stored in the memory, implements various functions of a heterogeneous wargame simulation and decision-making method based on LLM-MARL collaborative driving. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, text conversion function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0033] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0034] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0035] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A heterogeneous wargame simulation and decision-making method based on LLM-MARL collaborative driving, characterized in that, include: The battlefield situation in the current round is feature-extracted to obtain a situation feature matrix. Key situation indicators are extracted from the situation feature matrix and quantified into military terminology levels. They are then assembled into structured situation text according to a preset protocol template. The structured situation text is input into the LLM strategic planner, which then outputs a tactical intent vector. For each of the multiple heterogeneous agents, the tactical intent vector is used as the query source and the agent's local observation at time t is used as the key value source. An intent guidance feature is generated through a cross-attention network and the intent guidance feature is input into the policy network corresponding to the agent. The policy network then outputs the action. The action output by the policy network at multiple consecutive time points and the corresponding local observations are collected to form a trajectory segment. The trajectory segment is mapped to the semantic latent space shared with the tactical intent vector to obtain tactical trajectory features. The policy network is trained with the consistency loss Lalign between the tactical trajectory features and the tactical intent vector and the environment reward. The execution deviation generated by the action is monitored. When the execution deviation exceeds a preset threshold, the failure features at the corresponding moment are extracted and transformed into reflection text. The reflection text is fed back to the LLM strategic planner to update the tactical intent vector for the next round, thus forming a two-way collaborative loop between the LLM strategic planner and the policy network.

2. The heterogeneous wargame simulation and decision-making method based on LLM-MARL collaborative driving as described in claim 1, characterized in that, The process of extracting features from the battlefield situation in the current round to obtain a situation feature matrix is ​​as follows: A situational graph is constructed using a graph neural network with battlefield entities as nodes and spatial adjacency relationships between battlefield entities as edges. The situational graph is then used to extract features to obtain the situational feature matrix, which includes topological relationship information between entities, terrain obstruction information, and fire coverage information.

3. The heterogeneous wargame simulation and decision-making method based on LLM-MARL collaborative driving as described in claim 1, characterized in that, The process of generating intent-guided features using the tactical intent vector as the query source and the agent's local observations at time t as the key source, through a cross-attention network, specifically involves: The cross-attention network is a multi-head cross-attention network, which obtains the query matrix by linear transformation of the tactical intent vector. The local observation The bond matrix is ​​obtained through the first linear transformation. The local observation The value matrix is ​​obtained by the second linear transformation. The intent-guided features are calculated using the following formula: in, Let Kenv be the column vector dimension. This indicates that a normalization exponential transformation is performed on each row of the input matrix, and T represents the matrix transpose operation.

4. The heterogeneous wargame simulation and decision-making method based on LLM-MARL collaborative driving as described in claim 1, characterized in that, The process of collecting the actions and corresponding local observations output by the policy network at multiple consecutive time points constitutes trajectory segments. These trajectory segments are then mapped to a semantic latent space shared with the tactical intent vector to obtain tactical trajectory features. Specifically: Taking the current time t as the endpoint, a sliding window of length k is selected, and the actions output by the policy network and the corresponding local observations within the sliding window are collected to form the trajectory segment; A state embedding sequence is obtained by performing state semantic compression on the local observation sequence in the trajectory segment using a trajectory encoder. An action intent sequence is obtained by aggregating the action sequences in the trajectory segment. The state embedding sequence and the action intent sequence are then fused and encoded in the time dimension to output the original trajectory vector. ; Through projection function The original trajectory vector Projecting the vector onto the semantic latent space shared with the tactical intent vector yields the tactical trajectory features. : Wherein, the projection function It is implemented using a multilayer perceptron.

5. The heterogeneous wargame simulation and decision-making method based on LLM-MARL collaborative driving as described in claim 1, characterized in that, The process of training the policy network using the consistency loss Lalign between the tactical trajectory features and the tactical intention direction, combined with the environmental reward, specifically involves: The consistency loss Lalign is calculated using the following formula: in, For the tactical trajectory features With the tactical intent vector The inner product of the vectors, ‖·‖, denotes the L2 norm of the vector; The consistency loss Lalign is weighted and summed with the environmental reward term obtained by the policy network based on environmental feedback to form the overall training objective of the policy network. The parameters of the policy network are then updated according to the overall training objective.

6. The heterogeneous wargame simulation and decision-making method based on LLM-MARL collaborative driving as described in claim 1, characterized in that, The monitoring is based on the execution deviation caused by the action, specifically: Taking a monitoring window of length ΔT as the starting point, the execution deviation POR is calculated within the monitoring window using the following formula: in, For the agent i at time t, based on the action The resulting actual displacement vector, The tactical intent vector The indicated predetermined direction vector of motion, Let be the expected displacement vector at time t; When the execution deviation POR within the monitoring window ΔT is continuously greater than a preset threshold, it is determined that the execution deviation exceeds the preset threshold.

7. The heterogeneous wargame simulation and decision-making method based on LLM-MARL collaborative driving as described in claim 1, characterized in that, The extraction of failure features at corresponding moments and their transformation into reflective text specifically involves: The spatial conflict dimension features are extracted from the failure features at the corresponding time by the collision detection operator and transformed into path obstruction semantic text; the efficiency loss dimension features are extracted from the failure features at the corresponding time by the damage increment assessment operator and transformed into battle damage too fast semantic text; the temporal offset dimension features are extracted from the failure features at the corresponding time by the mission time counter and transformed into timeliness violation semantic text. The reflection text is obtained by encapsulating at least one of the path obstruction semantic text, the battle damage too fast semantic text, and the timeliness violation semantic text in the order of an anomaly type field, agent identifier field, quantification reason field, and environment label field. Specifically, the exception type field is filled with the category identifier of the dimension to which the triggered failure feature belongs, the agent identifier field is filled with the identifier of agent i that triggered the failure feature, the quantization reason field is filled with the quantization value of the triggered failure feature, and the environment label field is filled with the environment type identifier where the triggered failure feature is located.

8. A heterogeneous wargame simulation and decision-making device based on LLM-MARL collaborative driving, characterized in that, include: The semantic engine unit is used to extract features of the battlefield situation in the current round to obtain a situation feature matrix, extract key situation indicators from the situation feature matrix and quantify them into military terminology levels, and assemble them into structured situation text according to a preset protocol template. The tactical intent vector generation unit is used to input the structured situation text into the LLM strategic planner, and the LLM strategic planner outputs the tactical intent vector. The heterogeneous execution unit is used to generate intent guidance features for each of the multiple heterogeneous agents, using the tactical intent vector as the query source and the local observation of the agent at time t as the key value source, through a cross-attention network, and input the intent guidance features into the policy network corresponding to the agent, so that the policy network outputs the action. The consistency alignment unit is used to collect the actions and corresponding local observations output by the policy network at multiple consecutive time points to form trajectory segments, map the trajectory segments to the semantic latent space shared with the tactical intent vector to obtain tactical trajectory features, and train the policy network with the consistency loss Lalign between the tactical trajectory features and the tactical intent vector and the environment reward. An asynchronous feedback unit is used to monitor the execution deviation generated by the action. When the execution deviation exceeds a preset threshold, the failure features at the corresponding moment are extracted and converted into reflection text. The reflection text is fed back to the LLM strategic planner to update the tactical intent vector for the next round, thus forming a bidirectional collaborative loop between the LLM strategic planner and the policy network.

9. A heterogeneous wargaming simulation and decision-making device based on LLM-MARL collaborative driving, characterized in that, The system includes a memory and a processor. The memory stores a computer program that can be executed by the processor to implement a heterogeneous wargaming and decision-making method based on LLM-MARL collaborative driving as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The device contains a computer program that can be executed by a processor of the device where the computer-readable storage medium is located, to implement a heterogeneous wargaming simulation and decision-making method based on LLM-MARL collaborative driving as described in any one of claims 1 to 7.

Images