A smart holographic war game method and system based on a wearable device

Abstract

This invention discloses an intelligent holographic wargaming method and system based on wearable devices, belonging to the fields of artificial intelligence and wargaming technology. The method collects multimodal data of the commander's gaze, voice, and gestures through wearable devices, employs adaptive weighted fusion and sliding time window algorithms for spatiotemporal alignment and fusion, and generates multimodal cue words. A large language model is used to parse the cue words and convert them into structured instructions. Based on a two-layer coupled intelligent wargaming architecture, the instructions are processed: the macro-tactical planning layer uses the large language model to decompose tactical sub-tasks, and the micro-action execution layer uses a deep reinforcement learning model to execute micro-actions, generating data on the evolution of the future battlefield situation. Finally, the evolution data is mapped to a holographic display terminal for virtual-real overlay rendering and generates a natural language tactical analysis report. This invention solves the problems of low interaction efficiency, lack of interpretability in decision-making, and the disconnect between holographic display and wargaming logic in existing systems.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and wargaming technology, specifically to an intelligent holographic wargaming method and system based on wearable devices, and more particularly to an intelligent holographic sandbox wargaming method and system that integrates multimodal interaction, large language models and deep reinforcement learning. Background Technology

[0002] Wargaming systems are simulation and reasoning tools used to model adversarial scenarios. They typically consist of three parts: an interactive module, a simulation engine, and a display. In terms of interaction, these systems primarily rely on a graphical user interface (GUI), leading to excessive dependence on the mouse, keyboard, and two-dimensional screen during actual operation. When processing tactical commands, this interaction mode requires commanders to mentally translate complex, three-dimensional battlefield information into multi-level menu clicks and parameter inputs. This interaction mode requires active adaptation, resulting in long and inefficient operation chains, increased cognitive burden on commanders, and an inability to focus all their energy on core tactical decisions. Current research indicates that traditional wargaming systems also suffer from a mismatch between operation and intent. Furthermore, issuing commands via menus exhibits significant time lag, failing to meet the demands of real-time simulations.

[0003] With the development of artificial intelligence technology, current wargaming systems have incorporated algorithms or agents such as Deep Reinforcement Learning (DRL). With the support of DRL algorithms, the efficiency and depth of wargaming systems have been further improved. However, there are still two major challenges for AI decision-making models or agents applied to wargaming systems: First, the decisions given by the models or agents lack interpretability. Current wargaming models exhibit "black box" characteristics, meaning that the decisions they provide are often simply data; the models cannot explain the tactical logic behind the decisions in natural language. Commanders find it difficult to trust and adopt the model's recommendations, leading to a break in the trust chain between humans and machines. Second, existing models widely suffer from a lack of semantic understanding, meaning they struggle to directly understand vague, colloquial commands from commanders. In practical applications, models also cannot directly process multimodal inputs of "voice + gesture + visual intelligence," requiring extensive human collaboration in traditional wargaming systems to address this issue. This gap between human and machine collaboration makes true intent alignment difficult to achieve.

[0004] Existing technologies struggle to address the mapping between unstructured instructions and structured reasoning logic. For example, how to precisely align specific spatial information (gestures) with their semantic information (spoken instructions) on the timeline and transform them into precise parameters executable by the inference model. Current systems face multiple challenges in this regard. The system needs to capture and understand these two asynchronous pieces of information—gestures and spoken language—in real time, align them on the timeline through specific operations, synchronously interpret their combined information, and finally transform them into a set of specific parameter instructions. Current inference systems still have shortcomings in the process of translating human intentions into machine language.

[0005] In terms of display technology, holographic projection technology has achieved three-dimensional static visualization of the battlefield situation. However, in current sand table wargame simulation systems, holographic display and intelligent simulation are disconnected at the system architecture level. Holographic devices currently only serve as passive display terminals, lacking in-depth semantic interaction capabilities; while the simulation algorithms run on servers, and the dynamic battle situation simulation data they generate is difficult to map onto the holographic space in real time and intuitively, resulting in insufficient interactivity between display effects and reasoning logic.

[0006] In summary, there is currently a lack of a system that can deeply integrate multimodal natural interaction, interpretable intelligent inference algorithms, and holographic immersive display. This has become a key bottleneck restricting the transformation of existing sand table wargame simulation systems towards intelligence. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide an intelligent holographic wargaming method and system based on wearable devices, so as to solve the problems of low human-computer interaction efficiency, poor interpretability of decision logic and insufficient holographic dynamic visualization in existing sand table wargaming systems.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following solution: A smart holographic wargaming simulation method based on wearable devices includes: Step S1: Collect multimodal interaction data from commanders; Step S2: Perform fusion processing on the multimodal interaction data to generate multimodal prompt words; Step S3: Use a large language model to parse the multimodal prompt words and convert them into structured instructions; Step S4: The structured instructions are processed based on a dual-layer coupled intelligent simulation computing architecture to generate battlefield situation evolution data for a future time period. The dual-layer coupled intelligent simulation computing architecture includes a macro-tactical planning layer and a micro-action execution layer. Step S5: Map the battlefield situation evolution data to the holographic display terminal for virtual-real overlay rendering, and generate a tactical analysis report in natural language.

[0009] Furthermore, the multimodal interaction data mentioned in step S1 includes gaze data, voice data, and gesture data; gaze data is acquired through an eye-tracking sensor inside the wearable device; voice data is acquired through a near-field microphone array built into the wearable device; the gesture data is acquired through an adaptive bimodal acquisition strategy: if the commander is wearing a force feedback data glove, the high-precision posture data of the force feedback data glove is read first; otherwise, the visual tracking algorithm of the wearable device's camera is invoked to acquire the gesture data.

[0010] Further, step S2 involves fusing the multimodal interaction data to generate multimodal prompts, including: An adaptive weighted fusion algorithm is used to dynamically weight and fuse the confidence levels of the gaze data, voice data, and gesture data, with the weights of each data being dynamically adjusted according to the environmental and hardware conditions.

[0011] Furthermore, the fusion processing of the multimodal interaction data in step S2 also includes: A sliding time window algorithm is adopted, which takes the triggering time of the voice keyword as the center and performs spatiotemporal feature alignment of the gaze data, gesture data and voice data within a preset time window.

[0012] Further, in step S3, the multimodal prompt words are parsed and converted into structured instructions using a large language model, including: The confidence score of the output of the large language model is calculated. If the confidence score is lower than a preset threshold, a loop of visual highlighting and voice questioning feedback is triggered to generate question feedback to prompt the commander to supplement the instruction information.

[0013] Furthermore, in step S4, the macro-tactical planning layer decomposes the overall tactical instructions into multiple tactical sub-tasks based on a large language model; the micro-action execution layer controls virtual combat entities to execute micro-actions in the simulation environment under the constraints of each of the tactical sub-tasks based on a deep reinforcement learning model.

[0014] Furthermore, the processing of the structured instructions based on the dual-layer coupled intelligent inference computing architecture in step S4 also includes: It will simultaneously call the wargame rules calculation module to accurately calculate the specific battle losses, ammunition consumption, and morale changes at each step, ensuring that the final result complies with military rules.

[0015] Furthermore, step S5 involves mapping the battlefield situation evolution data to a holographic display terminal for virtual-real overlay rendering, including: A virtual combat entity representing a future scenario is rendered with a semi-transparent and edge-glowing effect using a custom holographic ghost shader, and then overlaid on a real holographic sand table model.

[0016] The present invention also provides an intelligent holographic wargaming system based on a wearable device, used to perform the method described in any of the above-mentioned embodiments, the system comprising: The multimodal data acquisition module is used to collect multimodal interaction data from commanders. The semantic parsing module is used to fuse and process the multimodal interaction data to generate multimodal prompt words, and to use a large language model to parse the multimodal prompt words and convert them into structured instructions; The dual-layer coupled intelligent simulation computing architecture includes a macro-tactical planning layer and a micro-action execution layer, which is used to generate battlefield situation evolution data for future time periods based on the structured instructions. A holographic display terminal is used to map the battlefield situation evolution data onto the holographic display terminal for virtual and real superposition rendering, and generate a tactical analysis report in natural language.

[0017] Furthermore, the multimodal data acquisition module is configured with a high-risk misoperation secondary confirmation mechanism, which is used to trigger secondary confirmation logic when the fused instruction is a high-risk operation or the confidence level of the intent is in the edge range.

[0018] The beneficial effects of this invention are as follows: 1. By using a sliding time window alignment algorithm and a multimodal adaptive weighting mechanism, the physiological time difference between the commander's line of sight, gestures and voice can be eliminated, thus eliminating the asynchronous problem of heterogeneous data. The adaptive weighting model dynamically adjusts the signal source weights according to environmental factors, and can maintain high-precision intent locking even in complex combat environments. This overcomes the problem of command loss or misjudgment caused by traditional interaction methods and achieves enhanced interaction robustness.

[0019] 2. The LLM introduces a thought chain analysis and tactical report generation module based on the large language model; it not only converts natural language instructions into structured tasks, but also presents their understanding and decomposition process; it converts the underlying simulation data into natural language that conforms to military logic, enabling commanders to understand the basis of model decision-making, establish a human-machine trust chain, and realize the transparency of the decision-making process.

[0020] 3. A two-layer coupled architecture of LLM macro-planning and DRL micro-execution was constructed, with an embedded wargame rule adjudication engine. LLM handles fuzzy macro-instructions, DRL handles path avoidance and formation control in physical simulation, and the rule engine ensures the rigor of damage calculation, avoiding the numerical illusion of pure LLM and the local optimum problem of pure DRL, ensuring that the simulation results conform to both tactical logic and physical reality, and realizing the closed-loop simulation strategy.

[0021] 4. By employing virtual-real overlay rendering technology, dynamic marching trajectories and combat predictions for future time periods are overlaid on the real static model of the holographic sand table. This allows commanders to intuitively compare the potential consequences of different tactical plans and conduct low-cost, zero-risk hypothetical analyses, greatly enhancing the ability to predict the evolution of the battlefield situation and realizing forward-looking situational awareness.

[0022] 5. During the instruction parsing and execution phase, logic such as confidence threshold judgment, high-risk instruction security interception, and holographic secondary confirmation pop-up is set up. For instructions that violate security rules or are semantically ambiguous, the system automatically blocks the automatic execution process and requires manual intervention for confirmation through multimodal feedback. This ensures that the commander always has the final decision-making and control authority, guarantees the security and controllability of the wargaming exercise, and achieves system control security. Attached Figure Description

[0023] Figure 1 This is a flowchart of an intelligent holographic war game simulation method based on wearable devices according to the present invention.

[0024] Figure 2 This is an interactive diagram of an intelligent holographic war game simulation system based on wearable devices according to the present invention. Detailed Implementation

[0025] like Figure 1 As shown, Figure 1 This is a flowchart of an intelligent holographic wargaming method based on wearable devices according to the present invention. An intelligent holographic wargaming method based on wearable devices includes: Step S1: Collect multimodal interaction data from commanders; Step S2: Perform fusion processing on the multimodal interaction data to generate multimodal prompt words; Step S3: Use a large language model to parse the multimodal prompt words and convert them into structured instructions; Step S4: The structured instructions are processed based on a dual-layer coupled intelligent simulation computing architecture to generate battlefield situation evolution data for a future time period. The dual-layer coupled intelligent simulation computing architecture includes a macro-tactical planning layer and a micro-action execution layer. Step S5: Map the battlefield situation evolution data to the holographic display terminal for virtual-real overlay rendering, and generate a tactical analysis report in natural language.

[0026] Furthermore, the multimodal interaction data mentioned in step S1 includes gaze data, voice data, and gesture data; gaze data is acquired through an eye-tracking sensor inside the wearable device; voice data is acquired through a near-field microphone array built into the wearable device; the gesture data is acquired through an adaptive bimodal acquisition strategy: if the commander is wearing a force feedback data glove, the high-precision posture data of the force feedback data glove is read first; otherwise, the visual tracking algorithm of the wearable device's camera is invoked to acquire the gesture data.

[0027] Further, step S2 involves fusing the multimodal interaction data to generate multimodal prompts, including: An adaptive weighted fusion algorithm is used to dynamically weight and fuse the confidence levels of the gaze data, voice data, and gesture data, with the weights of each data being dynamically adjusted according to the environmental and hardware conditions.

[0028] Furthermore, the fusion processing of the multimodal interaction data in step S2 also includes: A sliding time window algorithm is adopted, which takes the triggering time of the voice keyword as the center and performs spatiotemporal feature alignment of the gaze data, gesture data and voice data within a preset time window.

[0029] Further, in step S3, the multimodal prompt words are parsed and converted into structured instructions using a large language model, including: The confidence score of the output of the large language model is calculated. If the confidence score is lower than a preset threshold, a loop of visual highlighting and voice questioning feedback is triggered to generate question feedback to prompt the commander to supplement the instruction information.

[0030] Furthermore, in step S4, the macro-tactical planning layer decomposes the overall tactical instructions into multiple tactical sub-tasks based on a large language model; the micro-action execution layer controls virtual combat entities to execute micro-actions in the simulation environment under the constraints of each of the tactical sub-tasks based on a deep reinforcement learning model.

[0031] Furthermore, the processing of the structured instructions based on the dual-layer coupled intelligent inference computing architecture in step S4 also includes: It will simultaneously call the wargame rules calculation module to accurately calculate the specific battle losses, ammunition consumption, and morale changes at each step, ensuring that the final result complies with military rules.

[0032] Furthermore, step S5 involves mapping the battlefield situation evolution data to a holographic display terminal for virtual-real overlay rendering, including: A virtual combat entity representing a future scenario is rendered with a semi-transparent and edge-glowing effect using a custom holographic ghost shader, and then overlaid on a real holographic sand table model.

[0033] like Figure 2 As shown, Figure 2 This is an interactive diagram of an intelligent holographic wargaming system based on a wearable device according to the present invention. The present invention also provides an intelligent holographic wargaming system based on a wearable device, used to execute the method described in any of the above claims, the system comprising: The multimodal data acquisition module is used to collect multimodal interaction data from commanders. The semantic parsing module is used to fuse and process the multimodal interaction data to generate multimodal prompt words, and to use a large language model to parse the multimodal prompt words and convert them into structured instructions; The dual-layer coupled intelligent simulation computing architecture includes a macro-tactical planning layer and a micro-action execution layer, which is used to generate battlefield situation evolution data for future time periods based on the structured instructions. A holographic display terminal is used to map the battlefield situation evolution data onto the holographic display terminal for virtual and real superposition rendering, and generate a tactical analysis report in natural language.

[0034] Furthermore, the multimodal data acquisition module is configured with a high-risk misoperation secondary confirmation mechanism, which is used to trigger secondary confirmation logic when the fused instruction is a high-risk operation or the confidence level of the intent is in the edge range.

[0035] The present invention will be further described in detail below with reference to specific embodiments: This invention provides a multimodal interactive intelligent wargame simulation system that integrates multimodal interaction, large language models, and deep reinforcement learning technologies. This system does not improve the hardware itself for holographic display or basic interaction; instead, it focuses on intelligent data flow processing and closed-loop feedback. Utilizing multi-source data such as eye-gaze, gestures, and voice provided by wearable devices, and relying on a two-layer coupled architecture of large language models (LLM) and reinforcement learning (RL), it achieves accurate understanding of multi-source heterogeneous data and dynamic simulation of the battle situation. In summary, this method addresses the problems of fragmented interaction and the black-box nature of AI decision-making in existing systems by constructing a logical closed-loop link of "holographic perception - semantic parsing - two-layer simulation - dynamic feedback."

[0036] The innovation of this invention lies not in a single algorithm, but in addressing the three major challenges mentioned earlier through specific optimizations of the process and architecture, utilizing existing mature technologies (such as LLM and DRL). The specific implementation steps are as follows: Step S1: Collect multimodal interaction data from the commander; This step uses high-precision sensors in wearable devices (such as gesture recognition devices, AR or VR devices) to directly capture the commander's interactive behavior, and uses an adaptive weighted algorithm to solve problems such as visual occlusion or spatiotemporal asynchrony in traditional systems, thereby achieving accurate alignment of multimodal intentions.

[0037] 1.1 Real-time acquisition of multi-source data streams: The system connects wearable devices such as holographic interactive glasses to the server via a low-latency wireless protocol to acquire streaming data in the following three dimensions in real time: Eye Flow ( ): Acquired and recorded in real time by eye-tracking sensors inside wearable devices (such as AR / VR glasses), capturing the commander's three-dimensional gaze points in the holographic space. .

[0038] Audio Stream (D) voice ): Collects voice signals containing tactical commands through the built-in near-field microphone array of the wearable device.

[0039] Gesture Flow (D) hand The system supports an adaptive dual-modal acquisition strategy. If the commander is wearing force feedback data gloves, the system will prioritize reading the high-precision attitude data from the sensors; otherwise, it will use the visual tracking algorithm of the wearable device's camera to acquire data.

[0040] Step S2: Perform fusion processing on the multimodal interaction data to generate multimodal prompt words; 2.1 Multimodal Adaptive Weighted Algorithm and Error Correction: To address the signal quality fluctuation problem in wargaming systems, this invention introduces an intent confidence function to achieve adaptive weighted fusion:

[0041] S in the formula gaze S voice and S hand These represent the confidence levels of various data signals; w1, w2, and w3 refer to dynamic weighting coefficients. (1) Dynamic weight adjustment mechanism: The weights w1, w2 and w3 of the data stream are not fixed, but are dynamically adjusted according to the environment and hardware status. When the visual sensor detects that the ambient light is dark or the hand is occluded (i.e., gesture confidence), the weights are adjusted accordingly. When the gesture weight w3 is reduced, the system will automatically reduce the gesture weight w3 and significantly increase the gaze weight w1 (i.e., the intention determination is mainly based on the "eye gaze point") to ensure the robustness of the intention determination.

[0042] (2) Hardware Adaptive Mechanism: When a high-precision data glove is detected to be connected, the gesture weight is automatically increased (i.e., the intention determination is mainly based on "finger pointing").

[0043] (3) Cooperative Trigger Logic: The "gaze lock + voice activation" logic is adopted. When the line of sight gazes at a virtual unit for more than the threshold time (such as 300 ms), and a valid voice keyword (such as "attack") is recognized within the time window, the system will determine it as a valid operation instruction and automatically lock this unit as the operation object.

[0044] (4) High-risk Misoperation Secondary Confirmation Mechanism: If the fused instruction belongs to a high-risk operation (such as "bomb our area"), or when the intention confidence level F of the fused instruction is in the marginal range (such as 0.6 < F < 0.8), the system will not execute directly but trigger the system's secondary confirmation logic. The system will pop up a red warning box on the AR or VR display interface and ask via voice: A high-risk instruction is detected. Please confirm the password 'execute' or 'cancel'.

[0045] 2.2 Spatiotemporal Feature Alignment: Since people's habit is usually "look first, then point, and finally speak" or "point and speak at the same time", this may cause delays in visual, gesture, and voice signals on the time axis. This system introduces the Sliding Time Window algorithm: This algorithm sets a time interval [T0 - α, T0 + β] (for example, 300 ms before and after each), and takes the trigger moment of the voice keyword (i.e., T0) as the center. The system only searches and matches the most stable fixation point coordinates and gesture vectors within this interval, and aligns the weighted corrected spatial coordinates with the time stamp of the voice instruction, and finally splices and generates an enhanced multimodal prompt word to eliminate the time lag caused by heterogeneous data.

[0046] Step S3: Use the large language model to parse and convert the multimodal prompt word into a structured instruction; In this step, the large language model (LLM) is used as the semantic understanding center to convert fuzzy natural language instructions into precise machine code.

[0047] 3.1 Semantic Parsing and Entity Element Extraction: Input the prompt word generated in step S1 into the LLM. The model uses the Chain of Thought reasoning ability to perform entity element extraction and intention classification on the unstructured natural language instruction, and identifies the combat subject (Subject), tactical action (Action), target object (Target), and execution mode (Mode).

[0048] 3.2 Structured Instruction Generation: LLM converts the extracted features into standardized instructions that the inference engine can directly call, usually in JSON format (e.g., {Action: "Fire_Support", Target_Grid: [120, 45],Duration: 10}).

[0049] 3.3 Ambiguity Closure Processing: To address potential ambiguities in LLM (Local Management Model), the system uses confidence scores to assess command reliability, enabling closed-loop processing and enhancing system robustness. The detailed process is shown below: The system calculates the confidence score of the LLM output.

[0050] Judgment criteria: If the confidence level is greater than or equal to the preset threshold (e.g., 0.85), the instruction is valid and will automatically proceed to step S3; otherwise, exception handling will be performed.

[0051] Anomaly Handling: If the confidence level is less than the preset threshold (indicating that the instruction is ambiguous or key parameters are missing), the system triggers a feedback loop of "visual highlighting + voice questioning". The system generates a question based on the missing parameter slot and sends it to the commander for further information confirmation (e.g., highlighting two suspected targets, A and B, on the holographic map at the same time and asking the voice question: Please confirm whether the target is point A or point B?). At the same time, all suspected targets are highlighted on the holographic map, and the system returns to step S1 after waiting for the user to provide additional instructions.

[0052] Step S4: Process the structured instructions based on the dual-layer coupled intelligent inference computing architecture to generate battlefield situation evolution data for future time periods; This step employs a hybrid architecture, where LLM handles macro-level planning and DRL handles micro-level execution. This hybrid architecture resolves the disconnect between the capabilities of a single model in terms of overall strategic vision and micro-management, making the simulation more realistic at the physical level and more reasonable at the tactical level, thus facilitating subsequent tactical interpretation.

[0053] 4.1 Macro-tactical planning layer (LLM Agent): Based on the instructions in step S2 and combined with the current battlefield situation map (including terrain, enemy and friendly troop strength comparison, and supply line status), LLM ultimately decomposes the overall tactical instructions into several coherent tactical sub-task sequences. For example, the overall instruction "capture the high ground" is decomposed into [1. Drone reconnaissance] → [2. Artillery suppression] → [3. Infantry assault].

[0054] 4.2 Micro-action execution layer (DRL Agent): The system invokes the Deep Reinforcement Learning (DRL) inference engine (using the PPO proximal policy optimization algorithm). DRL receives each subtask from the macro-level and, within a simulation environment built on a physics engine, controls each virtual combat entity (such as a vehicle or soldier) to perform specific micro-actions such as path planning, terrain obstacle avoidance, and simulated combat.

[0055] 4.3 Rule Decision Engine: During the micro-simulation process of DRL, the wargaming rule calculation module (such as the Lanchester equation) is simultaneously invoked to accurately calculate the specific battle losses, ammunition consumption, and morale changes at each step, ensuring that the final result conforms to military rules. Ultimately, the future time period T is quickly generated. future The evolution of the battlefield situation within the country.

[0056] Step S5: Map the battlefield situation evolution data onto a holographic display terminal for virtual-real overlay rendering, and generate a tactical analysis report in natural language; To break down the barrier between intelligent logic and immersive display, this step transforms the previous data into intuitive spatiotemporal pre-visualization and white-box interpretation. This requires the cooperation of client-side rendering, server-side simulation, and the large model. The rendering module of the holographic presentation layer is developed based on a mature commercial 3D graphics engine (such as Unity3D or Unreal Engine). The specific implementation logic is as follows: 5.1 Data-driven and analytical approaches: Holographic display clients (such as AR / VR devices) establish a long-lived connection with the simulation server via the WebSocket protocol. The client receives and parses the situational data stream serialized in Protobuf (Protocol Buffers) format in real time, and immediately parses the data packets to extract key state parameters for each virtual combat unit, including: timestamp, three-dimensional coordinates (x, y, z), and rotation quaternions, providing a foundation for subsequent dynamic rendering.

[0057] 5.2 Implementation of the "Holo-Ghost" shader: To achieve a clear visual distinction between the current reality and future projections, this system employs a customized "holographic ghost" shader in its rendering engine. Specific technical parameters are as follows: (1) Alpha Blending: Set the alpha channel threshold of the model material to around 0.4 to 0.6. After testing, this range can ensure that the simulation image is clearly visible and achieve a semi-transparent effect, ensuring that it will not obscure the real sand table model and achieving a true combination of virtual and real.

[0058] (2) Rim Light: To achieve this effect, the Fresnel Effect algorithm can be used. When the angle between the line of sight and the normal of the model surface is close to 90°, the algorithm can enhance the brightness of the model edge. This method effectively enhances the stereoscopic effect of virtual objects in AR or VR view and solves the problem of holographic images floating.

[0059] (3) Dynamic color coding: Establish a mapping mechanism between state and color. The shader dynamically changes the main color of the material according to the received state information (for example, red highlights represent combat or damage state, and blue represents marching state), thereby providing intuitive feedback on the deduction results.

[0060] 5.3 Construction of the server-side physical simulation environment: To ensure the physical realism of the simulation process and results, the system runs a Headless mode (i.e., no rendering mode) without a graphical rendering interface on the cloud simulation server. Taking Unity Headless as an example, the system utilizes its built-in NVIDIA PhysX physics engine to perform rigid body collision detection and navigation mesh pathfinding calculations for virtual tanks and vehicles. This mechanism ensures that the marching routes simulated by the AI ​​strictly follow physical laws (e.g., tanks will not pass through walls, and vehicles will not enter impassable areas), thereby guaranteeing the credibility and physical realism of the simulation.

[0061] 5.4 Ghosting: Based on the aforementioned technologies and data, the system performs core virtual-real overlay rendering (Ghosting). The holographic display terminal reads the generated evolution sequence data and renders "ghost forces" in real time alongside the real troop models. These unit models move and engage in combat along a future timeline according to the data stream parsed in section 4.1. By comparing reality and the future within the same space, commanders can intuitively examine the potential consequences of different decisions, achieving a leap from static to dynamic situational awareness.

[0062] 5.5 Explainable AI for Battle Report Generation: To address the black-box nature of decision generation, the system invokes LLM again, this time acting as a tactical analyst. By reading detailed data from the simulation process (such as win rate curves, high-risk area markers, and battle damage data), it generates a natural language tactical analysis report (e.g., recommending the adoption of Plan A because the resistance encountered by the left-wing assault group after the feint attack attracted the enemy's main force was 40% lower than expected). This report is then broadcast to the commander via an AR or VR device's display interface and bone conduction headphones, enabling the commander to understand the specific results and basis of the instructions and bridging the trust gap between humans and machines.

[0063] Compared to traditional inference systems based on keyboard and mouse interaction and rule calculation, this invention systematically reconstructs the interaction chain, decision logic, feedback mechanism, and security. The differences between the old and new systems are not only reflected in the addition or deletion of specific steps, but also in the improvements in functionality and user experience. The differences between the old and new processes will be elaborated in detail from four specific technical aspects below.

[0064] 1. Interaction Paradigm: From "Menu-based Discrete Operations" to "Multimodal Intent-based Direct Access" (1) Existing process: In traditional systems, issuing commands typically involves multiple thought-action transitions. First, a visual search is performed, locating and selecting combat units on a two-dimensional screen or sand table using a mouse / gamepad. After selecting a unit, the specific operation must be navigated through menus, requiring navigating through multiple levels to find the appropriate tactical command (e.g., "Action" → "Attack" → "Fire Coverage"). Secondly, parameters must sometimes be manually input via keyboard, such as coordinates and duration. This process is a typical example of a human adapting to a machine, characterized by a long operational chain, heavy cognitive load, and a disconnect between the commander's tactical thinking and battlefield focus, making real-time command impossible in emergency situations.

[0065] (2) The process of this invention: This invention reconstructs the input process (corresponding to the multimodal perception and fusion of step S1 in the technical solution), realizing direct intent delivery and background fusion. Direct intent delivery means that the commander only needs to look at the target unit and verbally state the tactical intent (such as "suppress Hill B"); the system will automatically complete signal alignment and intent extraction in the background.

[0066] (3) Difference analysis: Compared to the old process, this invention eliminates all menu searching and manual parameter input steps, reducing operation time by more than 70%. Although it increases the complexity of the backend algorithm (spatiotemporal alignment), it achieves zero-menu interaction on the front end, allowing commanders to always focus on the battlefield situation itself, reducing the barrier to entry and training costs.

[0067] 2. Decision Generation: From "Manual Parameter Translation" to "Semantic Intelligent Parsing" (1) Existing technical process: The intelligence of traditional systems is mainly reflected in rule calculation, while command input still relies entirely on manual intervention. Commanders must manually break down vague tactical concepts (such as "intercept the enemy") into precise parameters that the system can recognize (such as "move to X,Y coordinates, set alert mode, firing distance 500 meters"). This process is prone to errors, and the system lacks the ability to understand and correct these errors. Once the parameters are set incorrectly, the system will mechanically execute the wrong command.

[0068] (2) Technical process of the present invention: The system introduces an LLM-based semantic understanding layer and a two-layer inference architecture of "LLM macro-planning + DRL micro-execution" through steps S2 and S3. The system can directly understand fuzzy natural language instructions, automatically complete entity extraction and parameter filling, and finally automatically decompose the instructions into reasonable sub-task sequences, which are then handed over to reinforcement learning agents for execution in physical simulation.

[0069] (3) Difference analysis: The new process enhances the semantic understanding and task decomposition capabilities of unstructured instructions. It transfers the work of "human translation" to algorithms. Compared to the old process, this invention avoids errors from manual parameter input, increases the ability to understand and process unstructured instructions, and achieves accurate interpretation and execution of complex and ambiguous tactical intentions. This allows commanders to use natural language that conforms to military conventions for command, lowering the barrier to entry for the system.

[0070] 3. Results Feedback: From static results display to dynamic process simulation (1) Existing technical process: Traditional game simulations typically provide static or instantaneous feedback. For example, once an instruction is issued, the system directly calculates and predicts the final outcome. On the display terminal, the pieces are modified directly based on the instruction's result. This means the commander cannot anticipate the risks and details during instruction execution and can only review the game after the final result has occurred.

[0071] (2) Technical process of the present invention: This invention adds a "holographic ghost" virtual-real overlay rendering technology in step S5, based on which functions such as virtual-real overlay and dynamic trajectory on the holographic sand table are realized. Virtual-real overlay refers to the system automatically generating a semi-transparent "ghost force" near the real static model based on time-series data. It can demonstrate the execution process and results of relevant commands in a future time period. Dynamic trajectory refers to the "ghost force" dynamically demonstrating the marching route and firefight situation in a future time period in the holographic space.

[0072] (3) Difference analysis: Compared to the old process, this invention adds a visual feedback over time. The new process no longer simply shows what is happening now, but can also demonstrate what might happen in the future. This feature allows commanders to shift from observing the present to rehearsing the future, helping them to intuitively compare the dynamic consequences of different plans and achieve a leap from result verification to process prediction.

[0073] 4. Security Mechanisms: From "Open-Loop Execution" to "Closed-Loop Verification with Human In-Loop Monitoring" (1) Existing technical process: Traditional processes have always followed a linear open-loop structure of input → execution → output. The system trusts all inputs by default and lacks mechanisms to identify and intercept high-risk operations or ambiguous instructions.

[0074] (2) Technical process of the present invention: In steps S2 and S3 of this invention, an embedded closed-loop design based on confidence threshold judgment and rule base is implemented to achieve functions such as ambiguity detection and high-risk operation interception. Ambiguity detection means that when the data collected by multimodal acquisition conflicts or has too low confidence, the system will automatically suspend and ask the operator again for the operation content; high-risk interception means that when the system identifies a high-risk instruction, it will forcibly interrupt the execution of the command and require the commander to confirm it again through multimodal interaction.

[0075] (3) Difference analysis: Compared to the old process, this invention adds a verification step before instruction execution. Although this "human-in-the-loop" design only adds a confirmation interaction in very rare cases, it fundamentally transforms the system's blind execution into controllable operation, eliminates catastrophic operations, ensures the system's security and robustness, and guarantees the commander's ultimate control.

[0076] In summary, this invention is not an optimization of existing system processes, but rather a construction of a completely new inference paradigm centered on "human-machine collaboration" by introducing innovative modules such as multimodal interaction, semantic understanding, two-layer intelligent inference, dynamic pre-simulation feedback, and security closed loop. This achieves a systematic generational improvement in efficiency, intelligence, insight, and security.

[0077] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Based on the technical essence of the present invention, any simple modifications, equivalent substitutions, and improvements made to the above embodiments within the spirit and principles of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for intelligent holographic wargaming based on wearable devices, characterized in that, include: Step S1: Collect multimodal interaction data from commanders; Step S2: Perform fusion processing on the multimodal interaction data to generate multimodal prompt words; Step S3: Use a large language model to parse the multimodal prompt words and convert them into structured instructions; Step S4: Process the structured instructions based on the dual-layer coupled intelligent simulation calculation architecture to generate battlefield situation evolution data for future time periods. The dual-layer coupled intelligent simulation calculation architecture includes a macro-tactical planning layer and a micro-action execution layer. Step S5: Map the battlefield situation evolution data to the holographic display terminal for virtual-real overlay rendering, and generate a tactical analysis report in natural language.

2. The intelligent holographic wargaming method based on wearable devices according to claim 1, characterized in that, The multimodal interaction data mentioned in step S1 includes gaze data, voice data, and gesture data; Eye-tracking data is acquired through an eye-tracking sensor inside the wearable device; voice data is acquired through a near-field microphone array built into the wearable device; and gesture data is acquired through an adaptive bimodal acquisition strategy: if the commander is wearing a force feedback data glove, the high-precision posture data of the force feedback data glove is read first; otherwise, the visual tracking algorithm of the wearable device's camera is used to acquire gesture data.

3. The intelligent holographic wargaming method based on wearable devices according to claim 2, characterized in that, Step S2 involves fusing the multimodal interaction data to generate multimodal prompts, including: dynamically weighting and fusing the confidence levels of the gaze data, voice data, and gesture data using an adaptive weighted fusion algorithm, wherein the weights of each data are dynamically adjusted according to the environmental and hardware conditions.

4. The intelligent holographic wargaming method based on wearable devices according to claim 3, characterized in that, Step S2, which involves fusing the multimodal interaction data, further includes: using a sliding time window algorithm to align the gaze data, gesture data, and voice data with spatiotemporal features within a preset time window, centered on the triggering time of the voice keyword.

5. The intelligent holographic wargaming method based on wearable devices according to claim 1, characterized in that, Step S3 uses a large language model to parse the multimodal prompt words and convert them into structured instructions. This includes: calculating the confidence score of the output of the large language model; if the confidence score is lower than a preset threshold, triggering a loop of visual highlighting and voice questioning feedback to generate question feedback to prompt the commander to supplement instruction information.

6. The intelligent holographic wargaming method based on wearable devices according to claim 1, characterized in that, In step S4, the macro-tactical planning layer decomposes the overall tactical instructions into multiple tactical sub-tasks based on a large language model; the micro-action execution layer controls virtual combat entities to execute micro-actions in a simulation environment under the constraints of each of the tactical sub-tasks based on a deep reinforcement learning model.

7. The intelligent holographic wargaming method based on wearable devices according to claim 1, characterized in that, Step S4, which processes the structured instructions based on a dual-layer coupled intelligent simulation computing architecture, also includes: synchronously calling the wargame rule calculation module to accurately calculate the specific battle losses, ammunition consumption, and morale changes at each step, ensuring that the final result conforms to military rules.

8. The intelligent holographic wargaming method based on wearable devices according to claim 1, characterized in that, Step S5 maps the battlefield situation evolution data to a holographic display terminal for virtual-real overlay rendering, including: using a customized holographic ghost shader to render virtual combat entities representing the future simulated situation with semi-transparent and edge-glowing effects, and overlaying them next to a real holographic sand table model.

9. A smart holographic war game simulation system based on wearable devices, characterized in that, The system for performing the method according to any one of claims 1 to 8 comprises: The multimodal data acquisition module is used to collect multimodal interaction data from commanders. The semantic parsing module is used to fuse and process the multimodal interaction data to generate multimodal prompt words, and to use a large language model to parse the multimodal prompt words and convert them into structured instructions; The dual-layer coupled intelligent simulation computing architecture includes a macro-tactical planning layer and a micro-action execution layer, which is used to generate battlefield situation evolution data for future time periods based on the structured instructions. A holographic display terminal is used to map the battlefield situation evolution data onto the holographic display terminal for virtual and real superposition rendering, and generate a tactical analysis report in natural language.

10. The intelligent holographic war game simulation system based on wearable devices according to claim 9, characterized in that, The multimodal data acquisition module is equipped with a high-risk misoperation secondary confirmation mechanism, which is used to trigger secondary confirmation logic when the fused instruction is a high-risk operation or the confidence level of the intent is in the marginal range.

Images