Multi-modal data perception and intention intelligent interaction control method for masking tasks

By employing a multimodal data perception and intent-based intelligent interactive control method, the problems of insufficient adaptability of smoke masking systems to dynamic environmental changes and low human-machine collaboration efficiency were solved. This method enables high-confidence data acquisition and rapid response adaptive masking strategy generation, thereby improving the execution efficiency and reliability of masking tasks.

CN121680187APending Publication Date: 2026-03-17AEROSPACE INTERNET OF THINGS TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202511854363.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-12-10
Publication Date
2026-03-17

Smart Images

  • Figure CN121680187A_ABST
    Figure CN121680187A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-modal data sensing and intention intelligent interaction control method for a shielding task, and belongs to the field of intelligent control and sensing interaction, and the method comprises the steps: collecting optical, electromagnetic and environmental multi-source heterogeneous data through a multi-modal data sensing module, and carrying out the time-space synchronization; performing data fusion and lightweight CFD model reasoning based on an edge calculation module, and predicting a smoke screen diffusion path; voice, gestures and AR instructions are analyzed through an intention intelligent interaction module and converted into structured control instructions; based on the intelligent control core module, an optimal smoke generation strategy is generated by adopting CFD simulation and reinforcement learning mixed decision; safe and low-delay distribution of instructions and multi-equipment cooperative control are realized based on a DDS protocol through the heterogeneous middleware module; and performing multi-modal evaluation and model online calibration according to the execution effect to form closed-loop optimization. According to the method, the environment perception dimension, the decision real-time performance, the resource utilization efficiency and the system self-adaptive capability are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of intelligent control and perception interaction technology, and in particular relates to a multimodal data perception and intention intelligent interaction control method for occlusion tasks. Background Technology

[0002] In modern battlefield environments, smoke masking systems serve as a crucial protective measure, widely used for the concealment and protection of personnel, technical weapons, and key targets. Existing technologies include: intelligent control methods for smoke generators based on environmental perception, which integrate basic sensors (such as temperature, humidity, and wind speed sensors) to achieve initial adjustments to smoke parameters, thereby enhancing the targeting of smoke screen release; multi-sensor information fusion methods applied in unmanned combat platforms, which integrate heterogeneous data sources such as optics and radar to enhance battlefield situational awareness; and augmented reality-based battlefield situational display and interaction systems that map the physical environment into a virtual sand table, assisting commanders in visual command and instruction issuance. These existing technologies, to a certain extent, improve the efficiency of masking missions and the level of human-machine collaboration, achieving initial adaptation and response to the battlefield environment.

[0003] However, the aforementioned existing technologies still have significant shortcomings. First, traditional smoke masking systems are insufficiently adaptable to dynamic environmental changes, lacking real-time sensing and closed-loop feedback mechanisms for key parameters such as wind speed, temperature, and humidity. This leads to deviations in the smoke diffusion path from expectations and unstable masking effects. Second, the system suffers from low human-machine collaboration efficiency, relying excessively on manual input of commands by operators, resulting in delayed command response and a disconnect from tactical coordination. Furthermore, the sensing dimension is limited, employing only optical or single-modal sensors, making them susceptible to interference under complex electromagnetic interference and weather conditions, resulting in low data confidence and difficulty in supporting precise decision-making and control. These shortcomings severely restrict the execution effectiveness and reliability of masking missions in dynamic battlefield environments. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention proposes a multimodal data perception and intention-based intelligent interactive control method for occlusion tasks, thereby resolving the issues present in the prior art.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a multimodal data perception and intention-based intelligent interaction control method for occlusion tasks, comprising: The multi-modal data sensing module collects multi-source heterogeneous data of the battlefield environment in real time, including optical data, electromagnetic data and environmental data, and performs spatiotemporal synchronization processing on the multi-source heterogeneous data. Based on the edge computing module, localized fusion and lightweight inference are performed on spatiotemporally synchronized multimodal data to generate environmental state vectors and predict smoke diffusion paths. The intent-based intelligent interaction module parses the commander's voice, gestures, or AR commands and converts them into structured machine-executable instructions. Based on the intelligent control core module, and combining the environmental state vector with the machine executable instructions, the optimal smoke generation strategy is generated through a hybrid decision-making process of CFD simulation and reinforcement learning. The optimal smoke generation strategy is distributed to the execution end through a heterogeneous middleware module based on the Data Distribution Service (DDS) protocol, thereby realizing distributed collaborative control of multiple equipment. Multimodal evaluation is performed based on the performance, and online calibration and closed-loop optimization of the prediction model and control strategy are carried out based on the evaluation results.

[0006] Preferably, the spatiotemporal synchronization processing of multi-source heterogeneous data includes: The PTP precision clock protocol is used to perform hardware-level clock synchronization between the lidar and the optical camera, with a synchronization accuracy of ±1μs. Based on the UTM unified transverse Mercator projection, the data from each sensor are mapped to a unified geographic coordinate system; Timestamp alignment is achieved using Kalman filtering and the Network Time Protocol (NTP), with a time synchronization accuracy of ±1ms.

[0007] Preferably, the localized fusion of multimodal data includes: The DS evidence theory is used to weight and fuse optical, electromagnetic, and environmental data to output a comprehensive threat level matrix; The order of the full-order model of computational fluid dynamics (CFD) is reduced by intrinsic orthogonal decomposition (POD) to construct a low-dimensional surrogate model. The prediction error of the full-order CFD model is dynamically corrected using a long short-term memory network. The input of the long short-term memory network includes historical airflow data and real-time environmental parameters.

[0008] Preferably, the parsing of the commander's voice, gestures, or AR commands includes: Semantic slots in voice commands are extracted using a BERT model finely tuned for military applications, and control commands in JSON format are generated. Based on the spatiotemporal convolutional network ST-GCN, key points of gesture skeletons are identified, and the intention of dividing and scaling operations is analyzed. The augmented reality (AR) sandbox maps touch operations in real time to geofence GeoJSON format and overlays them with optical video streams.

[0009] Preferably, the method further includes: when multiple interactive commands have overlapping areas, performing conflict arbitration based on a preset priority strategy, wherein the priority marked by the commander is higher than the priority suggested by the AI.

[0010] Preferably, the step of generating the optimal smoke generation strategy through a hybrid decision-making process combining CFD simulation and reinforcement learning includes: Constructing an initial reinforcement learning policy based on CFD simulation results; Design a reward function that optimizes occlusion coverage, duration, and resource consumption. The reward function is expressed as: ; in, Indicates coverage rate. Indicates duration gain. This represents resource consumption, where α, β, and γ are weighting coefficients and satisfy α+β+γ=1; The combination of smoke generation parameters, including emission elevation angle, dose, and frequency, is iteratively optimized using the Q-Learning algorithm.

[0011] Preferably, the method further includes: introducing a generative adversarial network (GAN) to generate adversarial examples and robustly training the reinforcement learning strategy.

[0012] Preferably, the optimal smoke generation strategy is distributed to the execution end based on the Data Distribution Service (DDS) protocol: Multiple QoS policy templates are predefined, including a RELIABLE reliability mode for control commands, a deadline of 10ms for maximum latency, and a LatencyBudget of 20ms. The instruction payload is encrypted using the national cryptographic algorithm SM4-CTR mode and verified using the SM3 hash algorithm.

[0013] Preferably, the online calibration of the prediction model based on the evaluation results includes: monitoring the smoke screen obscuring effect using an infrared thermal imager and millimeter-wave radar to obtain attenuation rate and signal-to-noise ratio data; The turbulence coefficient in the full-order CFD model is dynamically corrected based on the gradient descent method, and the update formula is as follows: ; Where, ΔR (k) Let η be the error between the CFD prediction and the measured data, and let η be the adaptive learning rate.

[0014] Secondly, the present invention also provides a multimodal data perception and intentional intelligent interaction control system for occlusion tasks, used to implement the method described in the first aspect, the system comprising: The multimodal data sensing module is used to collect multi-source heterogeneous data of the battlefield environment in real time. The multi-source heterogeneous data includes optical data, electromagnetic data and environmental data, and performs spatiotemporal synchronization processing on the multi-source heterogeneous data. The edge computing module is used to perform localized fusion and lightweight inference on spatiotemporally synchronized multimodal data, generate environmental state vectors, and predict the smoke diffusion path. The intent-based intelligent interaction module is used to parse the commander's voice, gestures, or AR commands and convert them into structured machine-executable instructions. The intelligent control core module is used to combine the environmental state vector with the machine-executable instructions to generate the optimal smoke generation strategy through a hybrid decision-making process combining CFD simulation and reinforcement learning. The heterogeneous middleware module is used to distribute the optimal smoke generation strategy to the execution end based on the Data Distribution Service (DDS) protocol, thereby realizing distributed collaborative control of multiple equipment; it performs multimodal evaluation based on the execution effect, and performs online calibration and closed-loop optimization of the prediction model and control strategy based on the evaluation results.

[0015] Compared with the prior art, the present invention has the following advantages and technical effects: The technical features of this invention, namely "real-time acquisition of multi-source heterogeneous data of the battlefield environment through a multi-modal data sensing module" and "spatiotemporal synchronization processing of multi-source heterogeneous data", integrate multi-source heterogeneous information such as optical, electromagnetic and environmental data, and achieve precise spatiotemporal alignment between sensors. Thus, it can still acquire high-confidence environmental state data under complex electromagnetic interference and variable weather conditions, providing reliable input for subsequent decision-making.

[0016] The technical features of this invention, namely "local fusion and lightweight inference of spatiotemporally synchronized multimodal data based on edge computing modules" and "prediction of smoke spread paths", enable rapid data fusion and real-time inference based on lightweight CFD models at the edge, significantly reducing end-to-end decision latency and enabling the prediction of smoke spread paths to respond quickly to dynamic changes in the battlefield environment.

[0017] The technical feature of this invention, which "analyzes the commander's voice, gestures, or AR commands through an intent-based intelligent interaction module and transforms them into structured machine-executable commands," directly converts the commander's natural interactive intent into executable commands, reducing the cumbersome process and response delay of traditional manual input and achieving efficient and intuitive human-machine collaboration.

[0018] The technical feature of this invention is that it "combines the environmental state vector with the machine executable instructions to generate the optimal smoke generation strategy through a hybrid decision-making process of CFD simulation and reinforcement learning". It uses CFD simulation to provide a reliable initial strategy that conforms to the laws of fluid mechanics, and performs multi-objective online optimization through reinforcement learning. This enables the generated smoke generation strategy to adapt to environmental disturbances and significantly reduce resource consumption while ensuring the occlusion effect.

[0019] The technical feature of this invention, which is "distributing the smoke generation strategy to the execution end through a heterogeneous middleware module based on the Data Distribution Service (DDS) protocol", utilizes the customizable QoS policy of the DDS protocol to ensure low latency and high reliability of control command transmission between heterogeneous platforms, and integrates national cryptographic algorithms to ensure data transmission security, thus meeting the dual requirements of real-time performance and security for military applications.

[0020] The technical feature of this invention, which is "to perform multimodal evaluation based on the execution effect and to perform online calibration and closed-loop optimization of the prediction model and control strategy based on the evaluation results", can dynamically correct model parameters and optimize decision-making strategies based on feedback data of actual occlusion effect, thereby continuously improving prediction accuracy and control performance in multiple task executions and possessing the ability to continuously self-optimize. Attached Figure Description

[0021] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of a multimodal data perception and intentional intelligent interaction control method for occlusion tasks according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a multimodal data perception and intentional intelligent interaction control system for occlusion tasks according to an embodiment of the present invention. Detailed Implementation

[0022] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0024] Example 1 like Figure 1 As shown, this embodiment provides a multimodal data perception and intentional intelligent interaction control method for occlusion tasks, including: Step 1: Collect multi-source heterogeneous data of the battlefield environment in real time through the multi-modal data sensing module. The multi-source heterogeneous data includes optical data, electromagnetic data and environmental data. Perform spatiotemporal synchronization processing on the multi-source heterogeneous data. Furthermore, the spatiotemporal synchronization processing of multi-source heterogeneous data includes: The PTP precision clock protocol is used to perform hardware-level clock synchronization between the lidar and the optical camera, with a synchronization accuracy of ±1μs. Based on the UTM unified transverse Mercator projection, the data from each sensor are mapped to a unified geographic coordinate system; Timestamp alignment is achieved using Kalman filtering and the Network Time Protocol (NTP), with a time synchronization accuracy of ±1ms.

[0025] Specifically, multimodal data acquisition and spatiotemporal synchronization enable real-time acquisition of multi-dimensional battlefield environment data, eliminating sensor spatiotemporal biases. Multi-source data triggering: The lidar and 4K camera (Sony IMX585) are synchronously triggered via the PTP precise clock protocol (synchronization accuracy ±1μs) to generate spatiotemporally aligned RGB-D point cloud data. The millimeter-wave radar (AWR2944) outputs a target range-velocity matrix (RD spectrum), and a spectrum analyzer generates an electromagnetic interference heatmap (frequency band coverage 2.4-40GHz). Environmental sensors (temperature, humidity, wind speed / direction, sound) collect data at a 1kHz sampling rate to construct a physical environment constraint model. Spatiotemporal alignment: Based on the UTM unified transverse Mercator projection, the data from each sensor is mapped to a unified geographic coordinate system; through Kalman filtering and sensor clock calibration (NTP protocol), timestamps are aligned to ±1ms accuracy.

[0026] Step 2: Based on the edge computing module, perform localized fusion and lightweight inference on the spatiotemporally synchronized multimodal data to generate an environmental state vector and predict the smoke diffusion path; Furthermore, the localized fusion of multimodal data includes: The DS evidence theory is used to weight and fuse optical, electromagnetic, and environmental data to output a comprehensive threat level matrix; The order of the full-order model of computational fluid dynamics (CFD) is reduced by intrinsic orthogonal decomposition (POD) to construct a low-dimensional surrogate model. The prediction error of the full-order CFD model is dynamically corrected using a long short-term memory network. The input of the long short-term memory network includes historical airflow data and real-time environmental parameters.

[0027] Specifically, the dynamic diffusion model is based on a pre-trained smoke diffusion model using CFD (Computational Fluid Dynamics) simulation, combined with real-time environmental data (wind speed, temperature, and humidity) to predict the diffusion path. Construction of a CFD-based smoke diffusion prediction model: Governing equations: The unsteady Navier-Stokes equations were solved using Large Eddy Simulation (LES). ; in For the source of smoke, For subgrid stress tensor.

[0028] Boundary condition settings: Mathematical description of launch port-mass flow inlet: Ground-No-Slip Wall Mathematical Description: Far-field pressure outlet mathematical description: .

[0029] Turbulence model calibration involves retrieving turbulence model parameters from wind tunnel experimental data and establishing parameter mapping relationships. Real-time prediction model construction: Model reduction: Intrinsic orthogonal decomposition (POD) is used to compress the full-order CFD model into a low-dimensional surrogate model. ; in For POD mode, For time coefficients, dimensions .

[0030] Parameters updated online: Real-time collection of environmental data Dynamically corrected turbulence coefficient: ; LSTM (Long Short-Term Memory) network is used to dynamically correct prediction bias, ensuring the robustness of the model under complex airflow conditions.

[0031] LSTM Dynamic Bias Correction Model Design - Network Architecture: Input layer: Time series data Dimension 5 ; Hidden layer: Two-layer LSTM structure, 128 units per layer, Dropout rate 0.2; State update equation: ; ; ; ; ; Output layer: Fully connected layer output correction amount ; LSTM Dynamic Bias Correction Model Design - Training Strategy: Dataset Construction - Collect the deviation sequence between battlefield measured data and CFD predicted values ​​to construct time series samples: ; Sliding window length The sampling frequency is 10Hz.

[0032] Loss function: Composite loss function balances short-term and long-term errors. .

[0033] This embodiment utilizes edge computing and data fusion to locally process multimodal data and generate a high-confidence environmental state vector. Data preprocessing: LiDAR point cloud processing: Point cloud clustering and obstacle contour extraction are achieved using the PointPillars algorithm accelerated by the Ascend 310 NPU (latency ≤20ms). Optical data enhancement: HDR fusion and dynamic noise reduction (DVPP hardware acceleration) are performed on 4K video streams to improve target detection accuracy in low-light environments. Multimodal fusion: Feature-level fusion: Point cloud, image, and radar data are used to generate a 3D battlefield situation map using the ICP registration algorithm; Semantic-level fusion: Based on DS evidence theory, optical (confidence 0.8), electromagnetic (confidence 0.7), and acoustic (confidence 0.6) data are weighted and fused to output a comprehensive threat level matrix. Lightweight inference: An edge-compressed CFD model (70% volume reduction) is run to predict smoke diffusion paths. Combined with LSTM to dynamically correct airflow disturbance errors, the root mean square error (RMSE) is ≤0.5m.

[0034] Step 3: The intentional intelligent interaction module parses the commander's voice, gestures, or AR commands and converts them into structured machine-executable instructions; Furthermore, the parsing of the commander's voice, gestures, or AR commands includes: Semantic slots in voice commands are extracted using a BERT model finely tuned for military applications, and control commands in JSON format are generated. Based on the spatiotemporal convolutional network ST-GCN, key points of gesture skeletons are identified, and the intention of dividing and scaling operations is analyzed. The augmented reality (AR) sandbox maps touch operations in real time to geofence GeoJSON format and overlays them with optical video streams.

[0035] This embodiment also includes: when multiple interactive commands have overlapping areas, conflict arbitration is performed based on a preset priority strategy, wherein the priority marked by the commander is higher than the priority suggested by the AI.

[0036] Specifically, voice / gesture input: voice commands (such as "preferential masking on the southeast side") are collected through a microphone array (beamforming technology), and semantics are analyzed by combining a natural language processing model (such as a military-grade version of BERT); gestures (such as drawing an area or setting a duration) are captured using a TOF (Time-of-Flight) camera, and the intention of the operation is analyzed by a skeletal key point recognition algorithm.

[0037] For intelligent intent parsing and command conversion, natural interactive commands are transformed into structured control parameters. Voice command parsing: Speech is picked up directionally using a beamforming microphone array, and a military-grade fine-tuned BERT model is used to extract key semantics (e.g., "key obscured area A"); semantic slot filling generates JSON commands. { "action": "SMOKE_DEPLOY", "priority": "HIGH", "area": ​​{"x1": 320.5, "y1": 480.2, "x2": 325.8, "y2": 482.3}, "duration": 300 } Gesture and AR Sandbox Analysis: The TOF camera captures key points of the skeletal structure of gestures, and recognizes operations such as area division and zooming through a spatiotemporal convolutional network (ST-GCN); AR sandbox touch operations are mapped in real time to geofences (GeoJSON format) and overlaid with optical video streams for display.

[0038] Intent conflict arbitration: When a voice command overlaps with the AR operation area, it is automatically arbitrated based on a priority strategy (such as “Commander Mark > AI Suggestion”), with a response time of ≤200ms.

[0039] Specifically, the AR tactical sand table: based on augmented reality technology, it maps the physical battlefield into a virtual sand table; it supports touch operation (such as dragging to delineate the smoke screen coverage area), and the commands are synchronized to the control system in real time, realizing "what you see is what you control".

[0040] Step 4: Based on the intelligent control core module, and combining the environmental state vector with the machine executable instructions, the optimal smoke generation strategy is generated through a hybrid decision-making process combining CFD simulation and reinforcement learning. Furthermore, the method of generating the optimal smoke generation strategy through a hybrid decision-making process combining CFD simulation and reinforcement learning includes: Constructing an initial reinforcement learning policy based on CFD simulation results; Design a reward function with occlusion coverage, duration, and resource consumption as optimization objectives; The combination of smoke generation parameters, including emission elevation angle, dose, and frequency, is iteratively optimized using the Q-Learning algorithm.

[0041] This embodiment also includes: introducing a Generative Adversarial Network (GAN) to generate adversarial examples and robustly training the reinforcement learning strategy.

[0042] Specifically, the reward functions are the coverage rate (percentage of the target area), duration (smoke screen stabilization time), and resource consumption (amount of smoke-generating agent used); The Q-Learning algorithm iteratively optimizes the combination of smoke generation parameters (such as emission angle, dose, and frequency) to achieve multi-objective trade-off optimization.

[0043] The Q-Learning algorithm flow and core element definitions are shown in Table 1.

[0044] Table 1

[0045] Q-Learning Algorithm Flow - Algorithm Iteration Process: Initialize Q-table randomly for each episode: Reset environment (randomly initialize wind speed / terrain) while not done: 1. Observe the current state s_t 2. ε-greedy action selection a_t: if random < ε: Random exploration else: Select max Q(s_t,a) 3. Perform the action to obtain the new state s_{t+1} and reward R_t. 4. Update the Q table: Q(s_t, a_t) += α[R_t + γ max_aQ(s_{t+1},a)-Q(s_t,a_t)] 5.s_t s_{t+1} Decrease ε and α This embodiment employs dynamic decision-making and collaborative optimization to generate an optimal occlusion strategy and dynamically adjust its execution. It integrates physical simulation and data-driven decision-making: a CFD-reinforcement learning hybrid decision-making approach. The initial strategy is based on CFD simulation to generate diffusion paths and emission parameters, guiding the accelerated convergence of reinforcement learning (RL) through a physically reliable initial strategy, thus constructing a physical-data closed loop for hybrid decision-making. Dynamic optimization: using the Q-Learning algorithm, with occlusion coverage (weight 0.6), duration (weight 0.3), and resource consumption (weight 0.1) as reward functions, iteratively optimizes the parameter combination. Adversarial training: introducing GANs to generate interference samples for opponent target detection (such as infrared imager noise) improves strategy robustness. Multi-objective trade-offs: constructing a Pareto front solution set, allowing commanders to interactively select preferred solutions (such as "maximum coverage" or "longest duration") via an AR sandbox.

[0046] Step 5: Through the heterogeneous middleware module, the optimal smoke generation strategy is distributed to the execution end based on the Data Distribution Service (DDS) protocol to realize distributed collaborative control of multiple equipment; Furthermore, the optimal smoke generation strategy is distributed to the execution end based on the Data Distribution Service (DDS) protocol: Multiple QoS policy templates are predefined, including a RELIABLE reliability mode for control commands, a deadline of 10ms for maximum latency, and a LatencyBudget of 20ms. The instruction payload is encrypted using the national cryptographic algorithm SM4-CTR mode and verified using the SM3 hash algorithm.

[0047] Specifically, the end-to-end collaboration mechanism is implemented based on the Data Distribution Service (DDS) protocol. Through its publish-subscribe model and customizable QoS policies, it ensures efficient, reliable, and low-latency communication between the backend handheld control terminal (RK3588+OpenHarmony) and the frontend unmanned equipment (Ascend 310+openEuler). Its cross-platform communication adaptation layer shields the differences between heterogeneous hardware and operating systems, providing a unified communication interface; the DDS core technology stack implements the publish-subscribe model, ensuring low-latency and highly reliable data transmission; the data encoding / decoding and serialization layer solves the data format compatibility problem across heterogeneous platforms; the security and trust unit ensures the security of the communication link, preventing data tampering and eavesdropping; the resource scheduler and optimizer maximizes the utilization of heterogeneous hardware resources and reduces end-to-end latency; and the diagnostic and fault-tolerant module monitors the communication status in real time and quickly recovers from anomalies.

[0048] This embodiment demonstrates distributed collaborative control and execution, enabling efficient collaboration and real-time response among multiple devices. Low-latency communication is driven by the DDS protocol: control commands are published via the topic / control / command, with the QoS policy configured in RELIABLE mode (maximum latency 10ms); sensor data is compressed using ZSTD (70% compression rate) and transmitted back via the topic / sensor / fusion_data.

[0049] Dynamic resource scheduling: Computing power offloading: Complex CFD computing tasks are dynamically allocated to the backend RK3588 NPU (6 TOPS computing power) to reduce the load on the frontend Ascend 310; Bandwidth allocation: Data flow priority is dynamically adjusted according to network status (5G / LoRa) (control commands > sensor data).

[0050] Security Execution Guarantee: Command encryption using national cryptographic standards: The payload is encrypted using SM4-CTR mode, and SM3 hash verification is used to prevent tampering; Resume interrupted transmission: The 10 most recent commands are cached locally, and automatically resent after network interruption (recovery time ≤300ms).

[0051] Step 6: Perform multimodal evaluation based on the execution results, and perform online calibration and closed-loop optimization of the prediction model and control strategy based on the evaluation results.

[0052] Furthermore, the online calibration of the prediction model based on the evaluation results includes: The smoke screen concealment effect was monitored by infrared thermal imager and millimeter-wave radar, and attenuation rate and signal-to-noise ratio data were obtained. The turbulence coefficient in the full-order CFD model is dynamically corrected based on the gradient descent method, and the update formula is as follows: ; Where, ΔR (k) Let η be the error between the CFD prediction and the measured data, and let η be the adaptive learning rate.

[0053] In this embodiment, the lightweight inference is executed on the domestically produced embedded chip Ascend 310, and the full-order CFD model is compressed through channel pruning and quantization joint optimization.

[0054] Specifically, for real-time feedback and adaptive adjustment, the strategy is dynamically optimized based on the execution effect. Multimodal effect evaluation: Infrared thermal imager monitors the smoke screen temperature field to evaluate infrared shielding efficiency (attenuation rate ≥90%); millimeter-wave radar detects the opponent's radar echo intensity to verify the electromagnetic interference effect (signal-to-noise ratio ≤-20dB). Closed-loop optimization: The reinforcement learning engine updates the reward function weights based on the evaluation results, with an iteration cycle ≤5s; CFD model parameters are calibrated online (e.g., turbulence coefficient correction) to improve the prediction accuracy for the next task.

[0055] Example 2 This embodiment addresses urban warfare cover missions where the enemy has deployed infrared detection equipment and drone patrols in urban alleyways. It necessitates rapidly generating smoke screens to cover key areas (within 200m of towers) to facilitate the covert advance of troops. The alleyways contain building obstructions and sudden gusts of wind (0-10m / s), requiring dynamic adjustments to the smoke screen dispersion strategy.

[0056] Step 1: Multimodal data acquisition and spatiotemporal synchronization; 1. Sensor triggering and data generation: (1) Optical sensing: The 4K camera captures footage of the alleyway at 30fps to detect the heat source of the enemy sniper (area with temperature > 40°C).

[0057] The lidar is synchronously triggered (PTP clock error ±1μs) to generate RGB-D data with a point cloud density of 1.6 million points / second, and to construct a three-dimensional model of the tunnel (accuracy ±5cm).

[0058] (2) Electromagnetic sensing: 1) The millimeter-wave radar (AWR2944) scans the communication frequency band (2.4GHz) of the opposing UAV and outputs the RD matrix; Example of RD spectral data: [Distance (m), Speed ​​(m / s)] = [(120, 5.2), (85, -3.1)] / / Enemy drone and armored vehicle targets.

[0059] 2) The spectrum analyzer generates an interference heatmap for the 2.4-40GHz frequency band and marks high-threat areas (signal strength > -60dBm).

[0060] (3) Environmental perception: The temperature and humidity sensor measured the humidity at 82%, and the wind speed sensor (ultrasonic type) detected an instantaneous gust of 8 m / s.

[0061] 2. Spatiotemporal alignment: (1) All sensor data are mapped to a unified coordinate system via UTM projection (Zone 49N). Example coordinates: Tower coordinates: UTM 49N 325000mE, 4815000mN.

[0062] (2) Kalman filtering aligns the timestamp to ±1ms to eliminate asynchronous sensor sampling error.

[0063] Step 2: Edge computing and data fusion; 1. Data preprocessing: (1) Point cloud processing: The Ascend 310 NPU runs the PointPillars algorithm, takes point cloud data as input, and outputs obstacle clustering results (delay 18ms): pillars = voxelize(point_cloud, voxel_size=(0.2m, 0.2m, 4m)) features = extract_pillar_features(pillars) predictions = CNN(features) # Output bounding boxes and categories.

[0064] (2) Optical enhancement: DVPP hardware performs HDR fusion (dynamic range increased to 120dB) and 3D noise reduction (PSNR increased by 6dB) on 4K video streams.

[0065] 2. Multimodal fusion: (1) Feature-level fusion: The ICP algorithm registers point clouds with optical images to generate textured 3D situation maps (registration error ≤ 0.1m).

[0066] (2) Semantic-level fusion: Calculation of the overall threat value based on DS evidence theory: Optical confidence level 0.8 → Threat weight 0.5; Electromagnetic confidence level 0.7 → Threat weight 0.3; Acoustic confidence level 0.6 → Threat weight 0.2; Overall threat value = 0.8 × 0.5 + 0.7 × 0.3 + 0.6 × 0.2 = 0.73.

[0067] (3) Lightweight reasoning: A compressed CFD model is used to predict the smoke diffusion path (input wind speed 8m / s, humidity 82%), and the output is the diffusion radius. CFD prediction: After 10 seconds, the smoke screen coverage radius R = 35m (after LSTM correction, R = 33.5m, RMSE = 0.3m).

[0068] Step 3: Intelligent intent parsing and command conversion; 1. Voice command parsing: Commander's voice command: "Prioritize shielding the southeast side of the City Hall for 5 minutes."

[0069] Military BERT model extracts semantic slots and generates JSON instructions: { "action": "SMOKE_PRIORITY", "priority": "URGENT", "area": ​​{"x1": 324800, "y1": 4814500, "x2": 325200, "y2": 4814700}, "duration": 300 } 2. AR Sand Table Interaction: If the touch-defined area overlaps with the JSON command, a conflict arbitration is triggered (the commander's mark has higher priority), and the command coverage area is ultimately increased by 10%.

[0070] Step 4: Dynamic decision-making and collaborative optimization; 1. Hybrid decision-making model: (1) Initial CFD strategy: The emission elevation angle is 38°, the smoke agent type is nano-graphite (infrared shielding rate ≥95%), and the dose is 20kg / s.

[0071] (2) Q-Learning optimization: 1) Reward function: ; in: Indicates time step Immediate reward. , , Let be the weighting coefficient, satisfying ; This represents coverage, a normalized metric for coverage effectiveness. Duration gain can be defined as... ; Resource consumption can be standardized as Actual resource usage / Maximum allowed resource usage / Maximum allowed resource usage.

[0072] 2) After 10 iterations, the parameters were optimized: elevation angle → 40°, dose → 22kg / s, and coverage increased by 12%.

[0073] 2. Combat training: GAN generates infrared interference noise from the target (signal-to-noise ratio -10dB), and the reinforcement learning model maintains occlusion efficiency ≥90% in noisy environments.

[0074] Step 5: Distributed collaborative control and execution; 1.DDS communication: (1) Control commands are published under the topic / control / command, with QoS configuration: Reliability=RELIABLE (Reliability Policy): Sets the reliability level of data transmission to "reliable transmission" mode, ensuring that all subscriber nodes will receive the messages sent by the publisher, and guarantees data integrity through automatic retransmission and acknowledgment (ACK) protocol.

[0075] Deadline=10ms (Deadline Period): Defines the maximum allowed time interval between two consecutive valid data publications as 10 milliseconds. If the publisher fails to update the data within the time limit, a DeadlineMissed status notification will be triggered.

[0076] LatencyBudget=20ms (Latency Budget): Sets the acceptable upper limit for end-to-end communication latency to 20 milliseconds, including the time consumed throughout the entire link, such as serialization, network transmission, and deserialization.

[0077] (2) Sensor data is compressed by ZSTD before being transmitted back, reducing bandwidth usage by 70%.

[0078] 2. Resource scheduling: Complex CFD computations are offloaded to the RK3588 NPU (utilization rate 60% → 30%), with end-to-end latency below 20ms.

[0079] 3. Safe execution: The instruction uses SM4 encryption (key length 256 bits) and has a 100% hash verification pass rate.

[0080] Step 6: Real-time feedback and optimization.

[0081] 1. Effectiveness Evaluation: (1) Infrared thermal imaging shows that the temperature difference in the shielded area is ≤2℃ (attenuation rate 92%). (2) The millimeter-wave radar detected the radar echo intensity of the other party to be -25dB (signal-to-noise ratio meets the standard).

[0082] 2. Closed-loop optimization: (1) Enhanced learning update reward weight: Coverage weight increased to 0.65 (due to changes in task priority); (2) The turbulence coefficient of the CFD model was corrected from 0.12 to 0.15 to adapt to the airflow disturbance in the roadway.

[0083] Example 3 like Figure 2 As shown, this embodiment provides a multimodal data perception and intentional intelligent interaction control system for occlusion tasks, used to implement the steps of the method described in Embodiment 1. The system includes a multimodal data perception module and an edge computing module set at the execution end, an intentional intelligent interaction module and an intelligent control core module at the control end, and a heterogeneous middleware module connecting the control end and the execution end.

[0084] The multimodal data perception module is integrated into the vehicle controller of the execution end (unmanned smoke-generating equipment), forming an embedded perception-decision integrated platform with the Ascend 310 computing unit and the openEuler operating system. Its connected sensor array includes optical perception units (ultra-wide-angle camera, LiDAR), electromagnetic perception units (millimeter-wave radar, spectrum analyzer), and environmental perception units (temperature and humidity sensors, wind speed / direction sensors, and sound sensors). The LiDAR and 4K camera are synchronously triggered (PTP clock synchronization, error ±1μs) to generate spatiotemporally aligned RGB-D point cloud data; the millimeter-wave radar outputs a target range-velocity matrix (RD spectrum), and the spectrum analyzer generates an electromagnetic interference heatmap; the temperature and humidity sensors, wind speed / direction sensors, and sound sensors are used to monitor environmental parameters in real time and construct a physical constraint model for smoke screen diffusion. The collaborative deployment and data complementarity of multi-source heterogeneous sensors overcome the perception limitations of single sensors, enabling real-time acquisition of multi-dimensional battlefield environmental data and providing high-confidence input for dynamic decision-making.

[0085] The edge computing module enables localized fusion and real-time processing of multimodal data. It utilizes a domestically produced embedded Huawei Ascend 310 chip, equipped with the CANN edge computing framework, and supports model compression and hardware acceleration. Through Kalman filtering and adaptive weighting algorithms, it fuses multimodal data such as temperature, humidity, wind speed, and optical data to generate an environmental state vector. Based on the chip's built-in NPU (Neural Processing Unit), it achieves timestamp synchronization (accuracy ±1ms) and spatial coordinate mapping (UTM projection) of sensor data, realizing spatiotemporal alignment. A compressed CFD prediction model (optimized by knowledge distillation technology, reducing model size to 30% of the original) is deployed, outputting real-time smoke diffusion path prediction results, achieving lightweight inference.

[0086] The intent-based intelligent interaction module transforms the commander's tactical intentions into machine-executable instructions, enabling efficient human-machine collaboration.

[0087] The intelligent control core module generates the optimal smoke generation strategy based on environmental data and combat intentions, and dynamically adjusts its execution.

[0088] The heterogeneous middleware module's end-to-end collaboration mechanism is based on the Data Distribution Service (DDS) protocol. Through its publish-subscribe model and customizable QoS policy, it ensures efficient, reliable, and low-latency communication between the backend handheld control terminal (RK3588+OpenHarmony) and the frontend unmanned equipment (Ascend 310+openEuler).

[0089] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-modal data perception and intent intelligent interaction control method for a sheltering task, characterized in that, The method comprises the following steps: Real-time acquisition of multi-source heterogeneous data of a battlefield environment by a multi-modal data perception module, the multi-source heterogeneous data comprising optical data, electromagnetic data and environmental data, and spatio-temporal synchronization processing of the multi-source heterogeneous data; Local fusion and lightweight inference of the spatio-temporally synchronized multi-modal data based on an edge computing module, generation of an environment state vector, and prediction of a smoke diffusion path; Analysis of voice, gestures or AR instructions of a commander by an intent intelligent interaction module, and conversion of the instructions into structured machine executable instructions; Generation of an optimal smoke emission strategy by a CFD simulation and reinforcement learning hybrid decision based on an intelligent control core module, in combination with the environment state vector and the machine executable instructions; Distribution of the optimal smoke emission strategy to an execution end based on a data distribution service (DDS) protocol by a heterogeneous middleware module, to realize distributed collaborative control of multiple equipment; Multi-modal evaluation according to execution effects, and online calibration and closed-loop optimization of a prediction model and a control strategy based on evaluation results.

2. The method of claim 1, wherein, The spatio-temporal synchronization processing of the multi-source heterogeneous data comprises: Hardware-level clock synchronization of a laser radar and an optical camera by a PTP precise clock protocol, with a synchronization accuracy of ±1 μs; Mapping of sensor data to a unified geographic coordinate system based on a UTM unified transverse Mercator projection; Timestamp alignment by Kalman filtering and a network time protocol (NTP), with a time synchronization accuracy of ±1 ms.

3. The method of claim 1, wherein, The local fusion of the multi-modal data comprises: Weighted fusion of optical, electromagnetic and environmental data by a D-S evidence theory, to output a comprehensive threat level matrix; Order reduction of a computational fluid dynamics (CFD) full-order model by principal orthogonal decomposition (POD), to construct a low-dimensional proxy model; Dynamic correction of prediction errors of the CFD full-order model by a long short-term memory (LSTM) network, with inputs of the LSTM network comprising historical airflow data and real-time environmental parameters.

4. The method of claim 1, wherein, The analysis of voice, gestures or AR instructions of a commander comprises: Extraction of semantic slots in voice instructions by a BERT model fine-tuned in the military field, to generate control instructions in JSON format; Identification of gesture skeleton key points based on a spatio-temporal convolutional network (ST-GCN), to analyze zone drawing and scaling operation intentions; Real-time mapping of touch operations to a geographic fence (GeoJSON) format by an augmented reality (AR) sand table, and superimposed display with an optical video stream.

5. The method of claim 4, wherein, Further comprising: When multiple interaction instructions exist in an overlapping area, conflict arbitration is performed based on a preset priority strategy, wherein a priority of a commander's marking is higher than a priority of an AI suggestion.

6. The method of claim 1, wherein, The generation of the optimal smoke emission strategy by a CFD simulation and reinforcement learning hybrid decision comprises: Construction of a reinforcement learning initial strategy based on CFD simulation results; Design of a reward function with shielding coverage, duration and resource consumption as optimization objectives, the reward function being represented as: ; wherein, denotes the coverage, denotes the duration gain, denotes the resource consumption, and a, b, g are weight coefficients and satisfy a+b+g = 1. Iterative optimization of smoke emission parameter combinations by a Q-Learning algorithm, the smoke emission parameters comprising emission elevation angle, dosage and frequency.

7. The method of claim 6, wherein, Further comprising: Introduction of a generative adversarial network (GAN) to generate adversarial samples, for robust training of a reinforcement learning strategy.

8. The method of claim 1, wherein, The optimal smoke emission strategy is distributed to an execution end based on a data distribution service (DDS) protocol: A plurality of QoS strategy templates are predefined, including a RELIABLE reliability mode for control instructions, a Deadline deadline of 10 ms, and a LatencyBudget delay budget of 20 ms; The instruction payload is encrypted using a SM4-CTR mode of a national secret algorithm, and is verified through an SM3 hash algorithm.

9. The method of claim 1, wherein, The online calibration of the prediction model based on the evaluation result comprises: The smoke screen shielding effect is monitored through an infrared thermal imager and a millimeter wave radar to obtain attenuation rate and signal-to-noise ratio data; The turbulent flow coefficient in the CFD full-order model is dynamically corrected based on a gradient descent method, and the update formula is: ; where ΔR (k) is the error of CFD prediction and measured data, and η is the adaptive learning rate.

10. A multimodal data perception and intention-based intelligent interaction control system for occlusion tasks, characterized in that, The system is used to implement the method of any one of claims 1-9, and comprises: A multi-modal data perception module is configured to collect multi-source heterogeneous data of a battlefield environment in real time, and to perform spatio-temporal synchronization processing on the multi-source heterogeneous data, wherein the multi-source heterogeneous data includes optical data, electromagnetic data, and environmental data; An edge computing module is configured to perform localized fusion and lightweight inference on the multi-modal data after spatio-temporal synchronization, to generate an environmental state vector, and to predict a smoke screen diffusion path; An intent intelligent interaction module is configured to analyze voice, gestures, or AR instructions of a commander, and to convert the instructions into structured machine executable instructions; An intelligent control core module is configured to combine the environmental state vector and the machine executable instructions, to generate an optimal smoke emission strategy through a CFD simulation and reinforcement learning hybrid decision; A heterogeneous middleware module is configured to distribute the optimal smoke emission strategy to an execution end based on a data distribution service (DDS) protocol, to implement distributed collaborative control of multiple equipment, to perform multi-modal evaluation according to execution effects, and to perform online calibration and closed-loop optimization of a prediction model and a control strategy based on evaluation results.