Industrial fire emergency method based on visual edge calculation

By synchronously collecting and fusion of visual, thermal and gas data in industrial fire protection systems, and using edge computing technology to generate accurate fire prediction and emergency solutions, the space-time misalignment and model generalization problems of multimodal data processing in traditional systems are solved, and fast and accurate fire emergency response is achieved.

CN120564329APending Publication Date: 2025-08-29HAINAN IND RES INST
View PDF 0 Cites 7 Cited by

Patent Information

Application Number
CN202510734826.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-06-04
Publication Date
2025-08-29

AI Technical Summary

Technical Problem

Traditional industrial fire monitoring systems have problems such as space-time data dislocation, inconsistent data formats and insufficient model generalization capabilities in terms of multimodal coordination, resulting in a deviation in flame spread path prediction, making it difficult to adapt to the fire protection needs of complex industrial sites.

Method used

By deploying multimodal acquisition equipment with integrated edge algorithms at the edge end of the factory area, visual, thermal and gas data are synchronized, feature extraction and fusion is used for ResNet-101 and MobileNetV3 models, fire prediction is combined with YOLOv7 framework, and fire emergency task flow is generated through the timing analysis model to achieve real-time processing and prediction of multimodal data.

Benefits of technology

It significantly shortens the fire emergency response time, provides accurate fire prediction and emergency response plans, ensures the orderly implementation of fire protection work, and adapts to the specific environmental needs of different factories.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120564329A_ABST
    Figure CN120564329A_ABST
Patent Text Reader

Abstract

The invention provides an industrial fire-fighting emergency method based on visual edge calculation. The method comprises the following steps: synchronously acquiring visual data, thermal data and gas data at the edge end of a factory; processing and feature extraction are respectively carried out on the collected data to obtain visual features, thermal features and gas features, after the visual features, the thermal features and the gas features are fused to obtain multi-modal fusion features, whether a fire happens or not is predicted, and if yes, a fire prediction picture is generated; presetting a time sequence analysis model, generating a fire emergency task flow according to a result output by the time sequence analysis model, dividing the priority of the fire emergency task flow, and submitting the fire emergency task flow to a workflow engine to distribute the fire emergency task flow to a corresponding execution unit; and on the basis of the steps S11 to S14, collecting multi-modal data of different factories, and pre-training corresponding fire emergency models. According to the invention, by synchronously collecting and fusing the multi-modal data of the treatment plant area, the corresponding industrial fire-fighting emergency scheme of the plant area is generated, and the fire-fighting emergency response time is remarkably shortened.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of fire warning technology, and in particular to an industrial fire emergency method based on visual edge computing. Background Art

[0002] Traditional industrial fire monitoring system architectures suffer from significant shortcomings in multimodal collaboration. The independent operation of visible light, infrared thermal imaging, and gas sensors can easily lead to spatiotemporal misalignment of collected data. For example, visible light cameras are deployed on the roof of a factory building, infrared sensors and cameras are installed in high-temperature risk areas, and gas sensors are distributed in ventilation openings or confined spaces. Furthermore, independent data processing by each sensor makes data integration and processing difficult, resulting in slow and timely response times for generated alarm signals and data spatiotemporal misalignment. For example, the sampling frequency differences between visible light video streams and infrared thermal imaging prevent the alignment of temporal features for the same event. Furthermore, the data formats of visible light RGB pixels, infrared temperature matrices, and gas concentration scalars are inconsistent, lacking a unified feature representation framework. Traditional algorithm systems utilize rule engines, which are unable to construct nonlinear correlations between multimodal data (such as the coupling effect between gas concentration and thermal radiation) and are unable to suppress interference. For example, visible light is easily affected by changes in illumination (e.g., strong welding light is misinterpreted as flames), infrared systems may falsely alert high-temperature equipment (e.g., a normally operating boiler), and gas sensors experience significant fluctuations due to ventilation.

[0003] The aforementioned system architecture's shortcomings in multimodal collaboration lead to discrepancies in flame spread path prediction: differences in thermal imaging frame rates cause heat source trajectory discontinuities, and gas diffusion rates are not factored into combustion dynamics modeling. When deployed across different scenarios, model generalization capabilities decrease due to variations in lighting conditions and equipment layout. These issues highlight the difficulty of traditional fire monitoring systems adapting to the complex environments of modern industrial sites, particularly in specialized scenarios like chemical storage and underground mines, putting the reliability of existing technologies under severe scrutiny. Summary of the Invention

[0004] The purpose of this invention is to provide an industrial fire emergency response method based on visual edge computing, which generates a corresponding industrial fire emergency plan for the factory area by synchronously collecting and fusing multimodal data of the factory area, thereby significantly shortening the fire emergency response time.

[0005] To achieve the above objectives, the present invention provides an industrial fire emergency method based on visual edge computing, the method comprising: S11. Deploy multimodal acquisition equipment integrated with edge algorithms at the edge of the factory, wherein the multimodal acquisition equipment is used to synchronously acquire visual data, thermal data, and gas data; S12. Processing and feature extraction are performed on the visual data, thermal data, and gas data respectively to obtain visual features, thermal features, and gas features, and fusing them to obtain multimodal fusion features. Based on the multimodal fusion features, whether a fire will occur is predicted. If a fire occurs, a fire prediction image is generated. S13, inputting visual features, thermal features, gas features, and multimodal fusion features into a preset time series analysis model, and outputting a predicted path coordinate sequence of fire spread, the direction of fire spread speed at different time nodes, and a dynamic 3D thermal map based on the fire prediction screen; S14. Generate a fire emergency task flow based on the output of the time series analysis model, prioritize the fire emergency task flow, and submit it to the workflow engine to assign the fire emergency task flow to the corresponding execution unit; S15. Based on steps S11 to S14, multimodal data of different factory areas are collected, and fire emergency models of different factory areas are pre-trained to generate corresponding fire emergency task flows for different factory areas.

[0006] Furthermore, step S11 specifically includes: Clock modules are implanted in visible light cameras, infrared thermal imagers, and gas sensors through a cross-sensor spatiotemporal synchronization calibration method. For asynchronous data with a time difference exceeding a preset threshold, the cubic spline interpolation method is used to generate the feature vector of the intermediate time; A multi-sensor extrinsic parameter matrix is ​​established based on Zhang's calibration method to map the temperature data of infrared thermal imaging to the visible light image coordinate system.

[0007] Furthermore, in step S12, the visual data, thermal data, and gas data are processed and feature extracted respectively, specifically including: Compress the pre-trained ResNet-101 model into the MobileNetV3 architecture, input the visual data into the MobileNetV3 architecture to extract image features and obtain visual features; The thermal data is encoded by ResNet-18 and aligned with the visual feature resolution to obtain the thermal features; The gas data is encoded and expanded through the LSTM network to obtain gas characteristics.

[0008] Furthermore, the pre-trained ResNet-101 model is compressed into the MobileNetV3 architecture, which includes: Use the pre-trained ResNet-101 model as the teacher model and the pre-trained MobileNetV3 model as the student model; The loss between the student model output and the true label and the loss between the student model output and the teacher model output are set as the distillation loss function; Use the output of the teacher model as auxiliary supervision information to train the student model, calculate the gradient through backpropagation and update the parameters of the student model; In the early stage of a fire, by reducing the temperature parameter, fine-grained knowledge transfer is enhanced to help the student model learn fuzzy features; In the open flame stage, by increasing the temperature parameters and reducing the migration intensity, the student model is encouraged to learn obvious features independently.

[0009] Furthermore, in step S12, the visual features, thermal features, and gas features are fused, specifically including: The visual features, thermal features, and gas features are extracted and abstracted in depth through the backbone layer of the YOLOv7 framework; A cross-modal attention module is introduced into the Neck layer of the YOLOv7 framework to perform multimodal fusion on the features processed by the backbone layer through the Neck layer of the YOLOv7 framework; Using visual features as queries and thermal and gas features as keys, it dynamically assigns weights and fuses them to output multimodal fusion features. The multimodal fusion features are used through the head layer of the YOLOv7 framework to predict whether a fire has occurred and generate a fire prediction image.

[0010] Furthermore, in step S13, the time series analysis model processes the input data, specifically including: The time series analysis model consists of a spatiotemporal graph convolutional network, an LSTM neural network, and a 3D convolutional layer; The spatiotemporal graph convolutional network processes the spatiotemporal characteristics of the input data and outputs a coordinate sequence of the predicted path of fire spread; The LSTM neural network processes the time series information of the input data and outputs the direction of the fire spread speed at different time nodes; The 3D convolutional layer processes the input data in the spatial and temporal dimensions and outputs a dynamic 3D heat map based on the fire prediction image.

[0011] Furthermore, in step S14, a fire emergency task flow is generated according to the output of the time series analysis model, specifically including: According to the fire scene situation and fire emergency plan, the fire emergency task is decomposed into multiple subtasks, and the input, output and execution logic of each subtask are defined, and the dependencies between subtasks are determined; Using the DAG algorithm, the fire emergency task flow is constructed based on the dependency relationships between subtasks, with subtasks as nodes in the graph and dependencies as directed edges; Optimize the priority of fire emergency task flow in real time and submit it to the workflow engine, which will assign the task to the corresponding execution unit according to predefined rules and task scheduling algorithm; During the task execution process, the execution status of the fire emergency task flow is monitored in real time, and dynamic scheduling and adjustments are made according to actual conditions.

[0012] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides an industrial fire emergency method based on visual edge computing. By synchronously collecting multimodal data and performing feature fusion and analysis, it predicts the occurrence of fire and generates a fire prediction screen, allowing relevant personnel to visually see the area where the fire occurred and its possible severity at the first time. The predicted fire spread path coordinate sequence output by the preset time series analysis model helps to determine the possible scope of the fire and plan the areas that need to be evacuated and the fire fighting areas in advance; the speed and direction of the fire spread at different time nodes provide an accurate time reference for the deployment of fire forces and the formulation of fire extinguishing strategies; the dynamic 3D heat map intuitively displays the intensity distribution and changes of the fire, providing a data basis for subsequent fire emergency task flows. According to the generated fire emergency task flow, it can be ensured that when a fire occurs, the corresponding execution unit can carry out fire fighting work in an orderly manner. Pre-training fire emergency models for different factory areas can realize the customization of emergency plans. The present invention generates corresponding industrial fire emergency plans for the factory area by synchronously collecting and fusing multimodal data of the factory area, significantly shortening the fire emergency response time. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0014] Figure 1 A flowchart of an industrial fire emergency method based on visual edge computing is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0016] Reference Figure 1This embodiment provides an industrial fire emergency method based on visual edge computing, the method comprising: S11. Deploy multimodal acquisition equipment with integrated edge algorithms at the edge of the factory area, where the multimodal acquisition equipment is used to synchronously collect visual data, thermal data, and gas data.

[0017] S12. Process and extract features of the visual data, thermal data, and gas data respectively to obtain visual features, thermal features, and gas features, and fuse them to obtain multimodal fusion features. Based on the multimodal fusion features, predict whether a fire will occur. If a fire occurs, generate a fire prediction screen.

[0018] S13. Input visual features, thermal features, gas features, and multimodal fusion features into a preset time series analysis model, and output the predicted path coordinate sequence of the fire spread, the direction of the fire spread speed at different time nodes, and a dynamic 3D thermal map based on the fire prediction screen.

[0019] S14. Generate a fire emergency task flow based on the output of the timing analysis model, divide the priority of the fire emergency task flow, and submit it to the workflow engine to allocate the fire emergency task flow to the corresponding execution unit.

[0020] S15. Based on steps S11 to S14, multimodal data of different factory areas are collected, and fire emergency models of different factory areas are pre-trained to generate corresponding fire emergency task flows for different factory areas.

[0021] In this embodiment, multimodal data is collected by multimodal acquisition equipment deployed within the factory. This data is then processed and fused to predict whether a fire has occurred within the factory and generate a fire prediction screen. If no fire has occurred, continuous data collection is performed. When a fire occurs, multimodal data from the scene is continuously collected and a fire prediction screen is generated, which is updated in real time. Based on the multimodal features collected and processed in real time and the corresponding fused features, a time series analysis model is used to predict the occurrence and development of the fire. This generates a sequence of predicted fire spread path coordinates, the direction of fire spread speed at different time points, and a dynamic 3D heat map based on the fire prediction screen.

[0022] Based on the predictions of fire occurrence and development using a time series analysis model, a fire emergency task flow is generated and prioritized based on the actual plant conditions. Execution instructions are generated and sent to the corresponding execution units. Considering the varying fire conditions across different plant areas, multiple fire emergency models are pre-trained for scenarios such as chemical, energy storage, and warehousing. Edge nodes automatically load adapted models based on collected environmental parameters. Edge nodes collect new samples and upload them to the cloud. The fire emergency models are iterated weekly and distributed to nodes across the entire network.

[0023] Specifically, for chemical, energy storage, and warehousing scenarios, multiple fire emergency response models are constructed during the pre-training phase. A variational autoencoder is used to generate scene meta-feature vectors. A lightweight environmental perception module is deployed on the edge to collect environmental parameters in real time. The cloud meta-feature library is matched using cosine similarity, and the optimal sub-model weights are dynamically loaded. A three-level incremental learning pipeline is designed. At the edge layer, nodes locally cache new samples (with annotated confidence ≥ 95%) and generate compact pseudo-labeled datasets through knowledge distillation. At the regional layer, incremental data from edge nodes is aggregated and the FedAvg algorithm is used to update regional shared models. At the cloud layer, data uploaded from the entire network is integrated weekly. Training is performed in stages using a curriculum learning strategy, prioritizing high-confidence samples (with a 3x increase in loss weight).

[0024] Blockchain hash verification is used to ensure consistent synchronization across all nodes in the fire emergency response model version control. This primarily facilitates phased processing of incremental data, optimizing model iteration efficiency and stability, reducing the computational overhead of incremental learning across scenarios, and preventing catastrophic forgetting. The edge layer caches high-confidence new samples and generates pseudo-labeled datasets through knowledge distillation. The regional layer aggregates edge node data and updates the shared model using a federated averaging algorithm. The cloud regularly integrates data from the entire network, employing a curriculum learning strategy to prioritize high-confidence samples for training. Model version control, combined with blockchain hash verification, ensures consistent synchronization across nodes. This phased data processing optimizes iteration efficiency and stability, reduces computational overhead across scenarios, and prevents the fire emergency response model from forgetting old knowledge.

[0025] A cross-scenario knowledge graph is constructed, encoding structured knowledge such as corrosion protection rules for chemical scenarios, thermal runaway warning thresholds for energy storage scenarios, and cargo ignition point parameters for warehousing scenarios into transferable features. A Graph Attention Network (GAT) is used to extract common features across scenarios and generate a transfer weight matrix. Once the transfer matrix is ​​loaded at the edge, only 5%-15% of the scenario-specific parameters need to be updated to complete adaptation. This enables targeted cross-domain knowledge transfer and integration, accelerates cold start of new scenario models, and reduces reliance on annotated data. A dynamic neural network pruning controller is deployed to automatically adjust model sparsity based on real-time environmental parameters (such as the operating status of chemical equipment and warehouse smoke concentration). Low-risk scenarios use 85% sparsity, improving inference speed to 15ms / frame; high-risk scenarios switch to 30% sparsity, keeping accuracy fluctuation within ±0.5%. Hardware-aware quantization technology (HAQ) enables FP16 / INT8 mixed-precision inference on the NPU, reducing energy consumption by 40%. This dynamically optimizes model computing resource allocation, balancing accuracy and efficiency to meet the real-time requirements of scenarios at varying risk levels.

[0026] As a preferred embodiment, step S11 specifically includes: Clock modules are implanted in visible light cameras, infrared thermal imagers, and gas sensors through a cross-sensor spatiotemporal synchronization calibration method.

[0027] For asynchronous data whose time difference exceeds a preset threshold, the cubic spline interpolation method is used to generate the feature vector of the intermediate time.

[0028] A multi-sensor extrinsic parameter matrix is ​​established based on Zhang's calibration method to map the temperature data of infrared thermal imaging to the visible light image coordinate system.

[0029] In this embodiment, based on the Armv9 edge AI computing platform, visible light cameras, infrared thermal imagers, and gas sensors with integrated edge algorithms are deployed at the edge of the factory area. Computing resources are dynamically allocated according to the risk level of the factory area to achieve synchronous collection of three-modal data: vision (flame), heat (temperature), and gas (gas concentration). A cross-sensor spatiotemporal synchronization calibration method is used to solve the problem of asynchronous collection of multimodal data in industrial environments. A high-precision clock module (error ≤ 10μs) is implanted in the visible light camera, infrared thermal imager, and gas sensor to synchronously record the UTC timestamp when collecting data. For asynchronous data with a time difference of more than 50ms, the cubic spline interpolation method is used (formula: ) generates feature vectors at intermediate moments, ensuring temporal alignment of the multimodal data. Using Zhang's calibration method, a multi-sensor extrinsic parameter matrix is ​​established, mapping the infrared thermal imaging temperature data to the visible light image coordinate system with a registration accuracy of ≤0.5 pixels. This setup improves the spatiotemporal consistency of the multimodal data to 99.2%, providing reliable input for subsequent fusion algorithms.

[0030] Specifically, a model reconstruction algorithm adaptive to edge device computing power was designed to address heterogeneous hardware compatibility issues. Floating-point computing power (FLOPS) and memory bandwidth (GB / s) were obtained from the edge node's NPU / GPU performance counters, and a hardware feature vector H = [f, b, c] was dynamically generated. Based on this hardware feature vector, an LSTM network was used to predict the optimal model sparsity. For example, when NPU computing power is insufficient, the number of MobileNetV3 channels is automatically compressed to 75%. Taylor expansion importance evaluation is used to retain key weights, ensuring accuracy and minimizing loss. FP16 / INT8 mixed-precision inference is implemented on the ARMv9 architecture, automatically adjusting activation function accuracy based on hardware cache size to improve edge inference speed and reduce energy consumption.

[0031] As a preferred embodiment, in step S12, the visual data, thermal data and gas data are processed and feature extracted respectively, specifically including: The pre-trained ResNet-101 model is compressed into the MobileNetV3 architecture, and the visual data is input into the MobileNetV3 architecture to extract image features and obtain visual features.

[0032] In this implementation, the ResNet-101 model, consisting of convolutional layers, batch normalization layers, activation function layers, and residual blocks, is compressed into a MobileNetV3 architecture. Visual data is then processed using a lightweight MobileNetV3 architecture to extract visual features. The hardware ISP module performs HDR fusion and demosaicing, outputting a normalized RGB three-channel image. The h-swish activation function is used to enhance smoke texture features in dark areas and suppress overexposure noise in highlight areas. The image is then fed into the lightweight MobileNetV3 architecture for feature extraction. The initial convolutional layer captures basic image features such as edges and textures, generating a low-level feature map. The depthwise separable convolution module then splits traditional convolution into depthwise and pointwise convolutions, reducing computational effort while extracting visual features.

[0033] The thermal data is encoded by ResNet-18 and aligned with the visual feature resolution to obtain the thermal features.

[0034] In this implementation, the 14-bit RAW temperature data output by the infrared sensor is mapped into a 0-255 grayscale image after non-uniformity correction and radiation calibration, and then encoded by ResNet-18 and aligned with the visual feature resolution.

[0035] The gas data is encoded and expanded through the LSTM network to obtain gas characteristics.

[0036] In this implementation, gas data is captured using the gas sensor's time series signal through a sliding window. This signal is then encoded and expanded using an LSTM encoder (with a 64-dimensional hidden layer) to extract temporal features. The output 64-dimensional vector is then expanded into a 64×64 pseudo-image through a fully connected layer. This pseudo-image is then spatially aligned with the visual data. Channel-wise concatenation is then used to concatenate the gas pseudo-image with the downsampled visible light feature map, forming a 64×64×25 gas feature. An LSTM network is used to extract features from gas data, which is characterized by its temporal nature. LSTMs can effectively process sequential data and capture long-term dependencies within it, making them ideal for extracting features from time-varying sequences such as gas data.

[0037] As a preferred embodiment, the pre-trained ResNet-101 model is compressed into the MobileNetV3 architecture, specifically including: The pre-trained ResNet-101 model is used as the teacher model, and the pre-trained MobileNetV3 model is used as the student model.

[0038] In this implementation, the ResNet-101 model outputs a feature vector after deep feature extraction and fusion. The output data serves as the input source for the knowledge distillation technology, which is used for MobileNetV3 architecture knowledge migration to reduce the model size.

[0039] The loss between the student model output and the true label and the loss between the student model output and the teacher model output are set as the distillation loss function.

[0040] In this implementation, the detection loss (CIoU+FocalLoss) and distillation loss (KL divergence) are jointly trained through knowledge distillation to compress the MobileNetV3 model size and improve the accuracy of multimodal collaborative detection.

[0041] The output of the teacher model is used as auxiliary supervision information to train the student model, calculate the gradient through backpropagation and update the parameters of the student model.

[0042] In the early stages of a fire, we lowered the temperature parameter to enhance fine-grained knowledge transfer, helping the student model learn fuzzy features. In the open fire stage, we increased the temperature parameter to reduce the transfer intensity, encouraging the student model to autonomously learn explicit features.

[0043] In this embodiment, by dynamically adjusting the temperature parameter τ, fine-grained knowledge transfer is enhanced (τ approaches 0) when smoke features are blurred in the early stages of a fire, and the transfer intensity is reduced (τ approaches 1) during the open fire stage. This combines feature space constraints with classification boundary optimization to reduce the space occupied by the MobileNetV3 model and the training time.

[0044] As a preferred embodiment, in step S12, the visual features, thermal features and gas features are fused, specifically including: The visual features, thermal features and gas features are deeply extracted and abstracted through the backbone layer of the YOLOv7 framework.

[0045] A cross-modal attention module is introduced into the Neck layer of the YOLOv7 framework, and the features processed by the backbone layer are fused through the Neck layer of the YOLOv7 framework for multimodal fusion.

[0046] Using visual features as queries and thermal and gas features as keys, weights are dynamically assigned and fused to output multimodal fusion features.

[0047] The multimodal fusion features are used through the head layer of the YOLOv7 framework to predict whether a fire has occurred and generate a fire prediction image.

[0048] In this embodiment, YOLOv7 receives visual features extracted by MobileNetV3, thermal features encoded by ResNet-18, and gas features encoded and expanded by LSTM. The input layer receives preprocessed multimodal data, while the backbone structure performs deep feature extraction and abstraction on the input image data. The output layer generates the final detection result by combining the features provided by the backbone structure with the prediction image provided by the detection head. These feature vectors are fused by a cross-modal attention module embedded in YOLOv7's feature pyramid network (Neck layer). The Neck layer utilizes a cross-modal attention mechanism, using visual features as queries and thermal and gas features as key-value pairs. The fused features are then dynamically weighted and fused. The detection head then predicts the target, which includes fire-related information such as fire location, fire size, and flame shape, while also generating a fire prediction image.

[0049] Specifically, during the cross-modal attention module fusion process, a three-dimensional visual-thermal-gas feature space is constructed, and modal weights are dynamically assigned through gated attention. When the temperature anomaly ΔT>100°C, the thermal feature weight increases to 0.75, and when the CO concentration change rate δC>15ppm / s, the gas feature weight increases to 0.6, achieving multi-source information complementarity. Replacing standard convolution with depthwise separable convolution effectively reduces the computational load, thus enabling feature pyramid fusion with multi-scale detection capabilities. Replacing the traditional convolution of the YOLOv7 model with the Faster-C2f module reduces the computational load from 15.6GFLOPs to 9.3GFLOPs. Combined with TensorRT optimization, it achieves 192FPS real-time processing, effectively improving the processing speed of multimodal data fusion and reducing module energy consumption.

[0050] As a preferred embodiment, in step S13, the time series analysis model processes the input data, specifically including: The time series analysis model consists of a spatiotemporal graph convolutional network, an LSTM neural network, and a 3D convolutional layer.

[0051] The spatiotemporal graph convolutional network processes the spatiotemporal characteristics of the input data and outputs a coordinate sequence of the predicted path of fire spread.

[0052] The LSTM neural network processes the time series information of the input data and outputs the direction of the fire spread speed at different time nodes.

[0053] In this implementation, the LSTM neural network model captures the temporal dependency of the input data. LSTM captures the temporal dependency through a gating mechanism, which includes an input gate, a forget gate, and an output gate. When processing fire temporal data, LSTM is based on the multimodal features of each time step, combined with the hidden state and cell state at the previous moment, and updates the two through gating to memorize the development process of the fire, predict future trends, and achieve temporal dependency capture.

[0054] The 3D convolutional layer processes the input data in the spatial and temporal dimensions and outputs a dynamic 3D heat map based on the fire prediction image.

[0055] In this implementation, the 3D convolution layer extracts the three-dimensional spatial features of the fire spread of the input data. The 3D convolution layer uses a three-dimensional convolution kernel to perform sliding convolution on the data in the time and space dimensions. Through parallel processing of multiple 3D convolution kernels, it generates multiple feature maps to comprehensively characterize the three-dimensional features of the fire spread.

[0056] In this embodiment, a time series analysis model is used to predict the fire spread path. The input data for the time series analysis model is multimodal feature data fused by YOLOv7, including visual, thermal, and gas features. The time series analysis model utilizes a spatiotemporal graph convolutional network, an LSTM neural network, and a 3D convolutional layer. The spatiotemporal graph convolutional network is used to model multimodal data with spatiotemporal characteristics, providing strong support for fire spread prediction. The LSTM neural network primarily processes the time series information in the input data, and the 3D convolutional layer extracts features in both spatial and temporal dimensions, enhancing the ability to recognize complex patterns in the fire spread process. The time series analysis model outputs a sequence of predicted fire spread path coordinates, the direction of fire spread speed at different time points, and a dynamic 3D heat map generated based on the prediction results.

[0057] Specifically, within the time series analysis model used to predict fire spread paths, a spatiotemporal self-attention mechanism is implemented to reduce spatiotemporal modeling errors. A digital twin engine is integrated to map parameters such as terrain elevation and vegetation density in real time, enhancing confidence in predicted paths. This primarily improves fire spread prediction accuracy and supports global decision-making. NeRF (Neural Radiance Field) is used to reconstruct the three-dimensional light field of the fire scene. This is combined with multispectral data to generate interactive heat maps. Dynamic 3D heat maps visually display high-risk areas, assisting rescuers in quickly locating the core of the fire. Adversarial examples (such as simulated mirror reflections and smoke interference) are injected into model training to enhance the generalization capabilities of multi-source data fusion, reduce false alarms in complex environments, and improve robustness.

[0058] To ensure real-time detection and alerting at the edge layer, the output of the time series analysis model utilizes dynamic sparse training technology, sharing a subset of model gradients (sparseness ≥ 85%) between edge nodes. This allows for collaborative optimization of a 7.5M parameter model. Neural Architecture Search (NAS) is introduced to automatically generate lightweight models, increasing inference speed to 25ms / frame, reducing communication overhead, accelerating model inference, and ensuring rapid edge fire response. Keyframe extraction based on entropy value determination triggers data transmission when the entropy change rate ΔE > 1.5, reducing redundant transmission in static scenes, conserving bandwidth resources, and ensuring efficient fire video streaming. Segmented H.266 encoding is employed, with dynamic quantization enabled during the flame spread phase. A Trusted Execution Environment (TEE) is deployed for encrypted inference, ensuring the integrity and tamper resistance of alarm triggering commands and improving the interception rate of malicious attacks. Alarm commands are also equipped with a tamper-proof lock to ensure command security and reliability.

[0059] A quantum-inspired resource allocation strategy is used to allocate resources at edge nodes, resulting in a 41% improvement in scheduling efficiency compared to the traditional SWARM algorithm, optimizing resource utilization and shortening task response time. A knowledge graph of fire scenarios is constructed, converting video streams into semantic symbols (such as flame morphology codes and diffusion vectors), improving transmission efficiency and ensuring critical information is not lost in weak network environments. The DPU (data processing unit) offloads CPU / GPU / NPU tasks, reducing the time required to decompose complex tasks to 50ms and achieving 92% resource utilization, improving computing power coordination efficiency, and supporting high-concurrency tasks.

[0060] At the same time, programmable sensors deployed at the edge simultaneously capture 0.3-14μm wide-spectrum data, achieving a fire source size detection accuracy of ±0.02m². Dynamically modulated sensing automatically adjusts the operating wavelength based on ambient temperature, precisely identifying fire sources and improving detection accuracy in complex environments. A Bayesian causal graph model is constructed to quantify the conditional dependencies between multimodal data, further reducing false alarm rates. Probabilistic reasoning eliminates interference factors and reduces false alarms. A MEMS microphone array captures low-frequency soundprint characteristics (20-200Hz) of flames and fuses them with visual data to improve detection recall in dark environments. "Fire identification by sound" compensates for visual blind spots, adapts to nighttime scenarios, expands detection coverage, and enhances all-weather monitoring capabilities.

[0061] As a preferred embodiment, in step S14, generating a fire emergency task flow according to the output of the time series analysis model specifically includes: According to the situation at the fire scene and the fire emergency plan, the fire emergency task is decomposed into multiple subtasks, and the input, output and execution logic of each subtask are defined, and the dependencies between subtasks are determined.

[0062] Using the DAG algorithm, the fire emergency task flow is constructed according to the dependency relationships between subtasks, with subtasks as nodes in the graph and dependencies as directed edges.

[0063] In this implementation, a DAG (Directed Acyclic Graph) task flow (fire emergency task flow) is automatically generated based on real-time fire scene data. This automatic generation of the DAG task flow is driven by real-time fire scene data and is implemented by combining multi-source information with algorithmic logic. Dynamic 3D heat maps are used to visually display the distribution and changing trends of fire activity in spatial and temporal dimensions. They serve as a crucial data basis for generating the DAG task flow, supporting dynamic adjustment of task execution paths. A natural language processing module converts the fire emergency task flow into executable scheduling instructions, achieving automated conversion from textual strategy to code logic.

[0064] The priority of the fire emergency task flow is optimized in real time and submitted to the workflow engine, which then assigns the task to the corresponding execution unit based on predefined rules and task scheduling algorithms.

[0065] In this implementation, after generating a fire emergency task flow, the PPO algorithm prioritizes tasks based on flame spread speed, temperature gradient, and gas concentration change rate. Low-priority tasks are offloaded to the cloud for asynchronous processing, while high-priority tasks are forced to remain on the edge for execution. A reinforcement learning model is deployed on the cloud to analyze the state of edge nodes across the network and generate optimal scheduling instructions. When a fire breaks out, the cloud automatically calls on public cloud resources to expand the edge computing pool and increase the number of concurrent tasks for the corresponding execution unit. A sliding window algorithm updates weight coefficients in real time. This mechanism dynamically allocates computing resources based on fire severity, prioritizing core tasks, ensuring the real-time performance of critical tasks (such as fire source location), and avoiding overloading edge computing power.

[0066] During the task execution process, the execution status of the fire emergency task flow is monitored in real time, and dynamic scheduling and adjustments are made according to actual conditions.

[0067] In this implementation, a fire-driven resource tidal scheduling strategy is used to address edge node computing power fluctuations. A three-dimensional priority vector is constructed, encompassing flame spread rate, temperature gradient, and gas concentration change rate. Weights are determined using the Analytic Hierarchy Process (AHP). The scheduling strategy is trained using the PPO algorithm. The state space encompasses node computing power utilization, task queue length, and network latency, while the action space is the task offload ratio (0-100%). The reward function is designed to be R = 0.6 task completion rate - 0.3 end-to-end latency + 0.1 energy efficiency. When a flame spread rate Us > 1m² / s is detected, the edge node's GPU overclocking mode is automatically triggered, and temporary computing resources are requested from the cloud, ensuring that the processing latency of high-priority tasks remains stable at less than 50ms.

[0068] In this embodiment, when a fire breaks out, the cloud automatically calls upon AWS Fargate, a public cloud container instance, to expand the edge computing pool. This is achieved through Kubernetes cluster management: dynamic scaling of container instances to 500 nodes within 1 second, supporting an increase in concurrent tasks from 10,000 to 100,000 per second. The lightweight service mesh Istio is used to implement cross-cloud resource scheduling with a latency of ≤80ms. Computing pool expansion is automatically initiated upon a fire outbreak, rapidly increasing edge computing power to cope with the surge in computing demand during large-scale fires and ensuring system stability. A dual-delay deep deterministic policy gradient (TD3) model is deployed. Its input layer includes the computing power status of edge nodes across the entire network (GPU utilization, memory usage), task queue length, and network latency, and outputs optimal scheduling instructions (such as task offloading ratio and edge node coordination strategy).

[0069] Application examples based on the above methods 1. Main hardware components Network switch: responsible for connecting various devices to ensure stable transmission of data streams, including servers, cameras, PLCs, alarm devices, fire extinguishing devices, etc.

[0070] Server: used to process video image data, run deep learning models, generate alarm results, and connect to PLC to send corresponding alarm device and fire extinguishing device startup signal instructions.

[0071] Mining intrinsically safe cameras: installed in different areas of each mine, used to collect real-time image data streams related to underground safety operations.

[0072] PLC (Programmable Logic Controller): Receives control instructions from the server, drives the movement and operation of alarm devices and fire extinguishing devices, and performs corresponding start and stop operations on mining equipment in the alarm area based on the configuration and actual conditions.

[0073] Alarm device: The mine intrinsically safe sound and light alarm and the mine intrinsically safe camera have built-in speakers, which can flash and issue voice notifications according to instructions to warn of the current dangerous situation.

[0074] Fire extinguishing equipment: including equipment such as sprinklers and gas fire extinguishing agents, which can adjust their positions according to instructions, spray water or gas, and deal with dangerous situations such as fireworks and fires in mines.

[0075] Monitoring screen: used to display system operation status, monitoring video and alarm records in real time, and provide manual intervention and management interface.

[0076] 2. Collection of accidental and dangerous situations Equipment configuration: Install independently developed intrinsically safe mining cameras in areas with frequent entry and exit of personnel and work positions in various areas of the coal mine to ensure that unexpected dangers underground can be clearly captured.

[0077] Operation process: The intrinsically safe mining camera collects images of dangerous situations such as fire, explosion, gas leakage, etc. in real time and generates image sets.

[0078] 3. Dataset upload Device configuration: Run the Django framework, YOLOv7 model, and MobileNetV3 model on the server.

[0079] Operation process: The server receives the video stream from the camera and uploads the corresponding dangerous situation pictures and normal operation pictures to the corresponding folder of the server through the Django framework. The server automatically starts the YOLOv7 model.

[0080] 4. Dataset training and annotation Device configuration: Start labelimg annotation on the server and start labeling the dangerous situation dataset.

[0081] Operation process: Classify and label the data set according to the hazard type, randomly select 80% of the labeled training set image bibliography from the total training set, and use the remaining 20% ​​as the test data set for testing the model algorithm after training.

[0082] The data set is generated after data set recognition and processing, including identification of dangerous situations such as underground fire, smoke detection, open flame detection, and bunker collapse detection.

[0083] 5. Video surveillance platform launch and use of recognition data sets to monitor safety operations Equipment configuration: Run Spring Boot backend framework, Vue front-end framework, and MySQL database in the server.

[0084] Operation process: The server starts all the components required by the video surveillance platform, imports the trained YOLOv7 model and MobileNetV3 model files into the independently developed video surveillance platform, starts the monitoring service for underground dangerous situations, and automatically switches the recognition model used according to the current server load to improve recognition efficiency. When a dangerous situation occurs, it responds to the alarm in a timely manner and triggers the parsing engine to analyze the alarm event information.

[0085] 6. Parsing engine parses fire emergency plan text The parsing engine is activated. When an unexpected situation occurs, the fire emergency plan text, on-site voice instructions, and infrared thermal images are jointly parsed according to the DAG task flow diagram that generates time-series dependencies. Based on the instructions and task flow diagram, the event is automatically processed, the escape route for personnel is planned, and the alarm equipment and emergency equipment along the escape route are activated.

[0086] 7. Monitoring and feedback Equipment configuration: The monitoring screen displays the area status, camera alarm information and camera monitoring results in real time. It can display real-time images, area status, alarm status, processing status and other information for operators to supervise and manage.

[0087] Operation process: Through the monitoring screen, operators can view the real-time situation of the monitoring platform, equipment status feedback, and alarm information, and process and monitor danger alarms, and view the current danger and personnel and handling status.

[0088] Through this implementation process, dangerous underground situations can be automatically identified and corresponding alarm commands can be triggered, allowing for efficient monitoring and guidance tasks. Ultimately, real-time monitoring of the entire process can be achieved through a large monitoring screen, ensuring accurate and efficient monitoring and handling of dangerous situations.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An industrial fire emergency method based on visual edge computing, characterized in that: The method comprises: S11. Deploy multimodal acquisition equipment integrated with edge algorithms at the edge of the factory, wherein the multimodal acquisition equipment is used to synchronously acquire visual data, thermal data, and gas data; S12. Processing and feature extraction are performed on the visual data, thermal data, and gas data respectively to obtain visual features, thermal features, and gas features, and fusing them to obtain multimodal fusion features. Based on the multimodal fusion features, whether a fire will occur is predicted. If a fire occurs, a fire prediction image is generated. S13, inputting visual features, thermal features, gas features, and multimodal fusion features into a preset time series analysis model, and outputting a predicted path coordinate sequence of fire spread, the direction of fire spread speed at different time nodes, and a dynamic 3D thermal map based on the fire prediction screen; S14. Generate a fire emergency task flow based on the output of the time series analysis model, prioritize the fire emergency task flow, and submit it to the workflow engine to assign the fire emergency task flow to the corresponding execution unit; S15. Based on steps S11 to S14, multimodal data of different factory areas are collected, and fire emergency models of different factory areas are pre-trained to generate corresponding fire emergency task flows for different factory areas.

2. The industrial fire emergency method based on visual edge computing according to claim 1 is characterized in that: Step S11 specifically includes: Clock modules are implanted in visible light cameras, infrared thermal imagers, and gas sensors through a cross-sensor spatiotemporal synchronization calibration method. For asynchronous data with a time difference exceeding a preset threshold, the cubic spline interpolation method is used to generate the feature vector of the intermediate time; A multi-sensor extrinsic parameter matrix is ​​established based on Zhang's calibration method to map the temperature data of infrared thermal imaging to the visible light image coordinate system.

3. The industrial fire emergency method based on visual edge computing according to claim 2 is characterized in that: In step S12, the visual data, thermal data, and gas data are processed and feature extracted respectively, specifically including: Compress the pre-trained ResNet-101 model into the MobileNetV3 architecture, input the visual data into the MobileNetV3 architecture to extract image features and obtain visual features; The thermal data is encoded by ResNet-18 and aligned with the visual feature resolution to obtain the thermal features; The gas data is encoded and expanded through the LSTM network to obtain gas characteristics.

4. The industrial fire emergency method based on visual edge computing according to claim 3 is characterized in that: Compress the pre-trained ResNet-101 model to the MobileNetV3 architecture, including: Use the pre-trained ResNet-101 model as the teacher model and the pre-trained MobileNetV3 model as the student model; The loss between the student model output and the true label and the loss between the student model output and the teacher model output are set as the distillation loss function; Use the output of the teacher model as auxiliary supervision information to train the student model, calculate the gradient through backpropagation and update the parameters of the student model; In the early stage of a fire, by reducing the temperature parameter, fine-grained knowledge transfer is enhanced to help the student model learn fuzzy features; In the open flame stage, by increasing the temperature parameters and reducing the migration intensity, the student model is encouraged to learn obvious features independently.

5. The industrial fire emergency method based on visual edge computing according to claim 1 is characterized in that: In step S12, the visual features, thermal features, and gas features are fused, specifically including: The visual features, thermal features, and gas features are extracted and abstracted in depth through the backbone layer of the YOLOv7 framework; A cross-modal attention module is introduced into the Neck layer of the YOLOv7 framework to perform multimodal fusion on the features processed by the backbone layer through the Neck layer of the YOLOv7 framework; Using visual features as queries and thermal and gas features as keys, it dynamically assigns weights and fuses them to output multimodal fusion features. The multimodal fusion features are used through the head layer of the YOLOv7 framework to predict whether a fire has occurred and generate a fire prediction image.

6. The industrial fire emergency method based on visual edge computing according to claim 1 is characterized in that: In step S13, the time series analysis model processes the input data, specifically including: The time series analysis model consists of a spatiotemporal graph convolutional network, an LSTM neural network, and a 3D convolutional layer; The spatiotemporal graph convolutional network processes the spatiotemporal characteristics of the input data and outputs a coordinate sequence of the predicted path of fire spread; The LSTM neural network processes the time series information of the input data and outputs the direction of the fire spread speed at different time nodes; The 3D convolutional layer processes the input data in the spatial and temporal dimensions and outputs a dynamic 3D heat map based on the fire prediction image.

7. The industrial fire emergency method based on visual edge computing according to claim 1 is characterized in that: In step S14, a fire emergency task flow is generated based on the output of the time series analysis model, specifically including: According to the fire scene situation and fire emergency plan, the fire emergency task is decomposed into multiple subtasks, and the input, output and execution logic of each subtask are defined, and the dependencies between subtasks are determined; Using the DAG algorithm, the fire emergency task flow is constructed based on the dependency relationships between subtasks, with subtasks as nodes in the graph and dependencies as directed edges; Optimize the priority of fire emergency task flow in real time and submit it to the workflow engine, which will assign the task to the corresponding execution unit according to predefined rules and task scheduling algorithm; During the task execution process, the execution status of the fire emergency task flow is monitored in real time, and dynamic scheduling and adjustments are made according to actual conditions.

Citation Information

Cited By

  • Flue gas early warning method based on edge calculation

    CN120726790A

  • HPC curtain wall installation safety protection intelligent monitoring system

    CN121034055A

  • HPC curtain wall installation safety protection intelligent monitoring system

    CN121034055B

  • Fire prevention and control method and system using computer vision

    CN121147856A

  • Video cloud intelligent AI retrieval method and device and medium

    CN121579730A