A deep learning-based fire intelligent identification and prediction method and system

Abstract

The application provides a fire intelligent identification and prediction method and system based on deep learning, which comprises the following steps: acquiring multi-modal data of a target area, including a visible light image sequence, a thermal imaging image sequence and environmental sensor data; inputting the visible light and thermal imaging images into a multi-task fire identification network, extracting multi-scale fusion features through a shared backbone network, and performing flame detection, fire segmentation and intensity estimation tasks in parallel; inputting historical fire identification results, building combustible material distribution characteristics and wind direction and speed characteristics into a fire spread prediction network based on a time series model and introducing fire physical laws, and outputting future fire prediction distribution; and generating a risk cloud map according to the prediction distribution, and visually displaying the risk levels of different geographical locations. Through multi-modal fusion, multi-task learning, time series prediction and physical knowledge fusion, the application realizes comprehensive perception and accurate prediction of fire, and improves the intelligent level and decision support capability of fire rescue.

Description

Technical Field

[0001] This invention relates to the fields of image processing and fire safety technology, specifically to a method and system for intelligent fire identification and prediction based on deep learning. Background Technology

[0002] Fire is a major disaster threatening people's lives and property. Quickly and accurately identifying fires and predicting their development is crucial for timely rescue and minimizing losses. Traditional fire monitoring relies mainly on smoke and heat sensors, which can only provide location information and cannot provide a comprehensive understanding of the fire situation. With the widespread use of video surveillance and the development of computer vision technology, image-based fire identification has become a research hotspot.

[0003] In recent years, deep learning technology has made groundbreaking progress in the field of image recognition, providing new technical means for intelligent fire identification. However, the complexity of fire scenes (changes in lighting, smoke obstruction, diverse flame shapes, etc.) poses a huge challenge to identification.

[0004] Currently, fire detection technologies mainly include the following categories: (1) Flame detection based on traditional image processing Early methods primarily relied on the color, texture, and motion characteristics of flames for identification. For example: Color feature method: Utilize the red and yellow characteristics of flames to set thresholds in the RGB or HSV color space; Texture feature method: Extracting the texture features of the flame (such as LBP, Gabor filter); Motion characteristic method: Analyzing the flickering and jumping characteristics of flames; These methods are computationally simple, but they are not robust and are easily affected by lighting and background interference, resulting in a high false alarm rate.

[0005] (2) Fire identification based on shallow machine learning Classification is performed using machine learning algorithms such as SVM and Random Forest, combined with manually designed features (SIFT, HOG, etc.). Representative research includes using an SVM classifier combined with color and texture features to identify flames. AdaBoost algorithms are also used for flame detection; these methods outperform traditional methods, but feature engineering is complex and generalization ability is limited.

[0006] (3) Fire identification based on deep learning In recent years, researchers have begun to use CNNs for fire identification: using networks such as AlexNet and VGG for flame classification, using Faster R-CNN for flame target detection, and using FCN for flame region segmentation. These methods have significantly improved the identification accuracy, but existing research mostly focuses on a single task (classification or detection) and lacks a comprehensive analysis of the fire situation.

[0007] (4) Fire prediction methods Traditional fire prediction primarily relies on physical models: regional models divide the space into several regions and calculate the temperature and smoke concentration in each region; field models (CFD) are based on fluid dynamics equations to precisely simulate fire and smoke flow. These methods have clear physical meanings but are computationally complex and difficult to apply in real time. Recent research has attempted to use machine learning to predict fires, but these are mostly simple regression models with insufficient prediction accuracy and timeliness.

[0008] In summary, the shortcomings of existing technologies are summarized as follows: (1) Insufficient accuracy of single-modal recognition Existing methods are mostly based on visible light images, which lead to a significant drop in recognition rates in scenarios such as dense smoke or nighttime conditions. Although some studies have used thermal imaging, there is a lack of an effective fusion mechanism between visible light and thermal imaging, failing to fully leverage the advantages of multimodal data.

[0009] (2) Lack of comprehensive fire analysis capabilities Existing methods mostly focus on flame detection (whether there is a fire), lacking a comprehensive analysis of the fire, such as key information like combustion intensity, fire range, direction of spread, and smoke concentration distribution, which cannot meet the actual needs of fire rescue.

[0010] (3) Static identification cannot predict development trends. Existing methods are mostly static identification methods, which can only analyze the fire situation in the current frame and cannot use time-series information to predict the development trend of the fire, thus failing to provide forward-looking support for rescue decisions.

[0011] (4) Ignoring building structure and environmental factors Existing methods only analyze the images themselves and do not consider the influence of building structure (distribution of combustibles, ventilation conditions, etc.) and environmental factors (wind direction and speed, temperature and humidity, etc.) on the development of fire, resulting in limited prediction accuracy.

[0012] (5) It is difficult to balance real-time performance and accuracy. Physical models (such as CFD) offer high accuracy but are computationally complex, making them difficult to apply in real time; simple machine learning models offer good real-time performance but lack accuracy. Existing methods struggle to achieve a good balance between real-time performance and accuracy.

[0013] (6) Lack of small sample learning ability Deep learning models typically require large amounts of labeled data, but fire scene data is difficult to obtain and labeling is costly. Existing methods show significant performance degradation and insufficient generalization ability in small sample sizes. Summary of the Invention

[0014] The technical problem to be solved by the present invention is to provide a method and system for intelligent fire identification and prediction based on deep learning, which aims to solve the technical problems in the existing technology such as single fire identification modality, one-sided analysis dimension, lack of prediction ability, insufficient integration of physical knowledge, difficulty in balancing real-time performance and accuracy, and weak small sample learning ability.

[0015] To achieve the above objectives, the first aspect of this invention proposes a method for intelligent fire identification and prediction based on deep learning, comprising the following steps: Acquire multimodal data of the target area, including visible light image sequences, thermal imaging image sequences, environmental sensor data, and building structure data; The visible light image sequence and thermal imaging image sequence are input into a pre-trained multi-task fire recognition network. Multi-scale fusion features are extracted through a shared backbone network, and the flame detection task, fire area segmentation task, and combustion intensity estimation task are performed in parallel using the multi-scale fusion features. The corresponding outputs are the flame position, fire spread range, smoke distribution information, and combustion intensity level at the current moment. The flame location and fire spread range output at multiple moments within a historical time period are combined with the combustible material distribution characteristics extracted from the building structure data and the wind direction and speed characteristics extracted from the environmental sensor data, and input into the fire spread prediction network. The fire spread prediction network is constructed based on a temporal convolutional network and, by introducing the physical laws of fire as constraints, outputs the fire spread prediction distribution within a preset time period in the future. The fire spread prediction distribution includes the flame spread range, temperature distribution, and smoke concentration distribution. Based on the predicted distribution of fire intensity within the preset future time frame, the risk score for each location within the target area is calculated by combining multiple risk factors, and the risk level is divided. A dynamic risk cloud map is generated through color mapping. The dynamic risk cloud map is visualized, and decision support information including priority fire extinguishing areas, personnel evacuation routes, and fire resource allocation suggestions is generated based on the prediction results.

[0016] Furthermore, the multi-task fire identification network includes a shared backbone network and flame detection branch, fire segmentation branch, and intensity estimation branch connected to the backbone network respectively; the flame detection branch outputs the bounding box of the flame based on the target detection algorithm; the fire segmentation branch outputs pixel-level classification results of flame and smoke regions based on the semantic segmentation algorithm, and further estimates the smoke concentration distribution and identifies the smoke type; the intensity estimation branch outputs the combustion intensity level based on the regression network.

[0017] Furthermore, the visible light image sequence and the thermal imaging image sequence are fused in a multimodal manner through at least one of the following methods: channel-dimensional splicing is performed before input to the backbone network to achieve early fusion; or, features are extracted through two independent backbone networks and feature fusion is performed in the middle layer of the network through an attention mechanism to achieve mid-term fusion; or, the two modalities are fully identified separately and then fused in the decision layer to achieve late-term fusion.

[0018] Furthermore, the fire spread prediction network includes: The encoder is used to encode the spatiotemporal features of the flame location and fire spread range within the historical time period to obtain the hidden state vector; The physical constraint layer is used to use the combustible material distribution characteristics as a mask to constrain the predicted fire spread range to be within the combustible material area, and to add residual terms of the heat conduction equation, flue gas flow equation or energy conservation equation as physical constraints during the training process. The decoder is used to decode and generate a fire prediction distribution within a preset time period based on the hidden state vector and the output of the physical constraint layer.

[0019] Furthermore, the method also includes constructing the fire spread prediction network using a Bayesian neural network or an ensemble learning method, such that the output fire prediction distribution includes a confidence interval to characterize the prediction uncertainty.

[0020] Furthermore, the risk factors include the probability of fire spread, the predicted temperature hazard, the smoke concentration hazard, the building structure hazard, and the risk of secondary disasters; the risk score is obtained by combining the risk factors through a weighted summation model or a neural network model; the risk level is divided into four levels: low, medium, high, and extremely high, and is visualized in green, yellow, orange, and red, respectively.

[0021] Furthermore, the method also includes model training and optimization steps: constructing a training dataset using real fire data, simulated data generated by fire simulation software, and data augmentation techniques; initializing network parameters using transfer learning, and combining semi-supervised learning with unlabeled data for consistency regularization training; and compressing and accelerating the trained model using knowledge distillation, model pruning, or quantization techniques to adapt to the real-time deployment of edge computing devices.

[0022] Furthermore, the method employs a keyframe extraction strategy in real-time video stream processing: key frames are determined by inter-frame difference detection or motion detection, and only key frames are used for complete fire identification and prediction; for non-key frames, optical flow or motion compensation techniques are used for result interpolation or the identification result of the previous frame is reused.

[0023] Furthermore, the method also includes a step of performing spatiotemporal alignment preprocessing on the multimodal data: performing spatial registration based on feature points on visible light images and thermal imaging images from different sources to achieve pixel-level alignment; and synchronizing the data from different sensors through hardware triggering or software timestamps to ensure that data at the same moment correspond to the same fire scene state.

[0024] A second aspect of this invention also proposes a deep learning-based intelligent fire identification and prediction system for implementing the above-described method, comprising: A multimodal data acquisition module is used to acquire multimodal data of a target area, including visible light image sequences, thermal imaging image sequences, environmental sensor data, and building structure data. The data preprocessing module is used to perform spatiotemporal alignment and enhancement preprocessing on the multimodal data; The multi-task fire identification module is used to input the pre-processed visible light image sequence and thermal imaging image sequence into the pre-trained multi-task fire identification network. It extracts multi-scale fusion features through a shared backbone network and uses the multi-scale fusion features to perform flame detection, fire area segmentation and combustion intensity estimation tasks in parallel, and outputs the flame position, fire spread range, smoke distribution information and combustion intensity level at the current moment. The fire spread prediction module is used to input the flame location and fire spread range output at multiple times within a historical time period, combined with the combustible material distribution characteristics extracted from the building structure data and the wind direction and speed characteristics extracted from the environmental sensor data, into the fire spread prediction network; the fire spread prediction network is constructed based on a temporal convolutional network and outputs the fire spread prediction distribution within a preset time period by introducing the physical laws of fire as constraints. The risk assessment and cloud map generation module is used to calculate the risk score of each location in the target area based on the fire prediction distribution within the preset future time period, combined with multiple risk factors, to classify the risk level, and generate a dynamic risk cloud map through color mapping. The decision support output module is used to visualize the dynamic risk cloud map and generate decision support information based on the prediction results, including priority fire extinguishing areas, personnel evacuation routes, and fire resource allocation suggestions.

[0025] The beneficial effects of the above-described technical solution of the present invention are as follows: (1) Through multimodal fusion (visible light + thermal imaging) and multi-task learning, the present invention has higher fire identification accuracy in complex scenes than existing single-modal methods, especially in difficult scenes such as dense smoke obscuring the view and nighttime. Thermal imaging can penetrate smoke to capture temperature information, making up for the shortcomings of visible light. Multi-task learning enables each task to share features and promote each other, further improving the overall performance.

[0026] (2) This invention not only identifies the location of the flame, but also outputs multi-dimensional information such as combustion intensity, fire range, smoke concentration distribution, and smoke type. Smoke type identification can determine the type of burning material based on color (black smoke for plastic / oil, white smoke for wood / paper, and yellow smoke for sulfur-containing substances), providing rescuers with a reference for burning materials and guiding the selection of extinguishing agents and protective measures.

[0027] (3) This invention, through a ConvLSTM-based temporal prediction network, can predict future fire spread trends, temperature changes, and smoke diffusion directions, thus buying valuable time for fire rescue. This is something that existing static identification methods cannot achieve, enabling rescue command to shift from passive response to proactive prediction.

[0028] (4) This invention integrates the physical laws of fire (constraints of combustibles, heat conduction, smoke flow, and energy conservation) into a deep learning model, avoiding unreasonable predictions that may occur with purely data-driven methods (such as fire spreading across non-combustible areas). Physical constraints make the prediction results more consistent with the laws of fire development, improving the credibility and interpretability of the prediction.

[0029] (5) This invention provides confidence intervals for prediction results through Bayesian deep learning or ensemble learning, allowing decision-makers to understand the reliability of the predictions. Through transfer learning and semi-supervised learning, high-performance models can still be trained even when fire data is scarce, solving the main bottleneck of deep learning applications and improving the generalization ability and practicality of the models.

[0030] (6) This invention achieves near real-time processing while ensuring recognition accuracy by using model compression and acceleration (knowledge distillation, model pruning, quantization) and real-time processing optimization (keyframe extraction, incremental update, prediction-assisted search, distributed processing). It meets the timeliness requirements of fire rescue and can be deployed and run on edge computing devices.

[0031] (7) This invention generates a dynamic risk cloud map, presenting complex prediction results in intuitive colors and patterns, which facilitates fire commanders to quickly understand the fire situation, identify high-risk areas, and formulate scientific rescue plans. The risk cloud map can be overlaid on a 3D building model, supporting multi-angle viewing and interactive operation. Attached Figure Description

[0032] Figure 1 This is an overall flowchart of the deep learning-based intelligent fire identification and prediction method provided in the embodiments of the present invention.

[0033] Figure 2 This is a schematic diagram of the structure of the multi-task fire identification network provided in an embodiment of the present invention.

[0034] Figure 3 This is a schematic diagram of the visible light and thermal imaging dual-stream fusion network provided in an embodiment of the present invention.

[0035] Figure 4 This is a schematic diagram of the attention mechanism provided in an embodiment of the present invention.

[0036] Figure 5 This is a flowchart of smoke recognition and analysis provided in an embodiment of the present invention.

[0037] Figure 6 This is a schematic diagram of the fire spread prediction network provided in an embodiment of the present invention.

[0038] Figure 7 This is a schematic diagram of physical knowledge fusion provided in an embodiment of the present invention.

[0039] Figure 8 This is a flowchart of risk assessment and cloud map generation provided in an embodiment of the present invention.

[0040] Figure 9 This is a flowchart of model training and optimization provided in the embodiments of the present invention.

[0041] Figure 10 This is a schematic diagram of the real-time processing flow provided in an embodiment of the present invention.

[0042] Figure 11 This is a risk assessment cloud map of fire spread according to the present invention.

[0043] Figure 12 This is a risk assessment cloud map of thermal radiation according to the present invention.

[0044] Figure 13 This invention provides a risk assessment cloud map of building load-bearing capacity. Detailed Implementation

[0045] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0046] Example 1: Overall Architecture and Workflow Please see Figure 1 This embodiment provides a deep learning-based intelligent fire identification and prediction method, which includes the following steps: Step S101: Multimodal data acquisition. Deploy multiple sensor devices in the target area to acquire multi-source data in real time via an IoT gateway. Specifically, this includes: Visible light video acquisition: Acquire RGB video from devices such as drones, fixed surveillance cameras, and firefighter body cameras, with a preferred resolution of 1920×1080 or higher and a preferred frame rate of 25-30fps. Perform preprocessing such as noise reduction and enhancement on the video to improve image quality.

[0047] Thermal imaging video acquisition: Infrared video is acquired from devices such as thermal imaging drones and thermal imaging cameras. The preferred temperature measurement range is -20℃ to 1500℃, covering temperature changes throughout the entire fire process. The thermal imaging data is converted into pseudo-color images for easier subsequent processing and visualization.

[0048] Environmental sensor data acquisition: Deploy temperature sensors to collect temperature data at multiple points, with a sampling frequency preferably of 1Hz; smoke sensors to collect smoke concentration data; gas sensors to collect concentrations of toxic gases such as CO and CO2; and meteorological sensors to collect environmental data such as wind direction, wind speed, and humidity.

[0049] Building structure data acquisition: retrieve Building Information Model (BIM) or 3D model to obtain information such as building materials, distribution of combustibles, ventilation system layout, load-bearing structure, etc., and convert this data into feature representations that can be used by the model.

[0050] Step S102: Data preprocessing and spatiotemporal alignment. Since multi-source data have different sampling frequencies, spatial resolutions, and coordinate systems, spatiotemporal alignment preprocessing is required. Spatial registration: Perform feature-point-based spatial registration on visible light images and thermal imaging images from different perspectives. For example, use SIFT or ORB features to extract matching points, and map the thermal imaging image to the coordinate system of the visible light image through affine transformation or perspective transformation to achieve pixel-level alignment.

[0051] Time synchronization: Data from different sensors is synchronized via hardware triggering or software timestamps to ensure that data from the same moment corresponds to the same fire scene state. Sensor data is mapped to the corresponding pixel coordinates or three-dimensional spatial coordinates of the image based on its spatial location.

[0052] Data augmentation: Performing augmentation operations on images such as random rotation, scaling, flipping, cropping, color jittering, and Gaussian noise to increase sample diversity and provide more variations for subsequent model training.

[0053] Step S103: Multi-task fire identification. The preprocessed visible light image and thermal imaging image of the current frame are input into a pre-trained multi-task fire identification network. This network adopts a multi-task learning architecture, shares a feature extraction backbone, and outputs flame detection results, fire segmentation results, and combustion intensity level. Specific implementation details will be described in detail in Example 2.

[0054] Step S104: Fire Spread Prediction. The fire identification results (flame location, fire range, intensity) from multiple moments over a past period (e.g., the past 2 minutes) are constructed into a time series. Simultaneously, combustible material distribution features (e.g., combustible areas represented by masks) are extracted from building structure data, and features such as wind direction and speed are extracted from environmental sensor data. This data is input into a fire spread prediction network, which is based on temporal models such as ConvLSTM and incorporates physical constraints. It outputs the predicted fire spread distribution over preset time intervals of 5, 10, and 15 minutes, including the potential range of flame spread, temperature distribution, and smoke concentration distribution. Specific implementation details will be described in detail in Example 3.

[0055] Step S105: Risk Assessment and Cloud Map Generation. Based on the fire prediction results and combined with risk factors (fire spread probability, predicted temperature, smoke concentration, building structural hazard, secondary disaster risk, etc.), a risk assessment model is used to calculate the risk score for each location within the target area and classify the risk levels (e.g., low, medium, high, extremely high). Then, the risk levels are overlaid on the map or 3D model of the target area using color mapping (e.g., green-yellow-orange-red) to generate a dynamic risk cloud map, which can be animated to show the evolution of the risk over time. Specific implementation details will be described in detail in Example 4.

[0056] Step S106: Visualization Output and Decision Support. The generated risk cloud map is displayed on devices such as the command center's large screen and mobile terminals, and key information (such as fire location, evacuation routes, and hazardous areas) can be overlaid and annotated. The system can also automatically generate decision suggestions based on the prediction results, such as priority fire-fighting areas, personnel evacuation routes, and fire resource allocation suggestions, to assist commanders in formulating scientific rescue plans.

[0057] Example 2: Detailed Design of a Multi-Task Fire Identification Network Please see Figure 2 This embodiment details the structure and working principle of the multi-task fire identification network.

[0058] 2.1 Overall Network Structure The multi-task fire detection network comprises a shared backbone and three parallel task branches: a flame detection branch, a fire segmentation branch, and an intensity estimation branch. The backbone is responsible for extracting multi-level, multi-scale feature maps from the input visible light and thermal imaging images. The task branches then perform different prediction tasks based on these shared features. This multi-task learning architecture not only reduces the number of model parameters but also improves overall performance through inter-task promotion, enabling each branch to utilize useful information from other tasks.

[0059] 2.2 Backbone Network The backbone network can employ advanced deep convolutional neural network architectures such as ResNet-50, EfficientNet-B4, and Swin Transformer. Taking ResNet-50 as an example, the network contains multiple residual blocks, extracting features from low to high levels through progressive downsampling. The preferred input image size is 512×512 pixels. For multimodal inputs, fusion is required first (see Section 2.5), and the fused tensor serves as the input to the backbone network. The feature maps output by the backbone network contain rich spatial semantic information, providing shared feature representations for each task branch.

[0060] 2.3 Flame Detection Branch The flame detection branch is implemented based on object detection algorithms, preferably using YOLOv5 or Faster R-CNN architectures. This branch takes the feature map output from the backbone network as input, generates candidate bounding boxes through a Region Proposal Network (RPN) or anchoring mechanism, and then outputs the confidence score and precise bounding box coordinates (x, y, w, h) for each candidate box through classification and regression heads. To accommodate the irregularity and dynamic changes in flame shape, deformable convolution or dynamic anchoring strategies can be introduced to better fit the flame boundary. The loss function of the detection branch includes classification loss (such as cross-entropy) and regression loss (such as Smooth L1 loss), denoted as L_detection.

[0061] 2.4 Fire Branching The fire segmentation branch is implemented based on a semantic segmentation algorithm, preferably using the U-Net++ or DeepLabV3+ architecture. This branch upsamples and skips connections through the multi-layer features of the backbone network to gradually restore the spatial resolution, ultimately outputting a pixel-level classification map of the same size as the input image. Each pixel is classified as flame, smoke, or normal background. The segmentation results can accurately depict the boundaries of fire spread and the extent of smoke coverage. The classification loss uses cross-entropy loss or Dice loss, denoted as L_segmentation. Furthermore, this branch extends the smoke analysis functionality: Smoke concentration estimation: Within the smoke area, the smoke concentration value of each pixel is estimated by using features such as image transparency, contrast, and dark channel prior, combined with smoke sensor data for calibration, and outputting a smoke concentration distribution map (concentration value 0-1).

[0062] Smoke type identification: Determine the type of burning material based on smoke color. Black smoke indicates incomplete combustion of petroleum products, plastics, etc.; white smoke indicates combustion of water-containing materials such as wood and paper; yellow smoke indicates combustion of sulfur-containing materials. This information can provide rescue personnel with a reference for the type of burning material, guiding the selection of extinguishing agents and protective measures.

[0063] 2.5 Intensity Estimation Branch The intensity estimation branch is implemented based on a regression network, typically consisting of a global average pooling layer and fully connected layers. It maps the global features extracted by the backbone network to combustion intensity levels. Combustion intensity can be divided into 1-5 levels, for example: level 1 is a weak flame, level 2 is a small fire, level 3 is a medium fire, level 4 is a strong fire, and level 5 is an extremely strong fire. This branch outputs the probability of each level or a direct regression of the intensity value. The loss function uses mean squared error (MSE) or cross-entropy (if considered a classification problem), denoted as L_intensity.

[0064] 2.6 Joint Loss Function for Multiple Tasks The multi-task fire detection network uses a multi-task joint loss function for end-to-end training. L_total = α·L_detection + β·L_segmentation + γ·L_intensity α, β, and γ are weighting coefficients, whose optimal values ​​are determined experimentally, and are typically set to values ​​that make the magnitudes of each loss term comparable. Joint training allows each task to share underlying features, promoting mutual improvement and enhancing overall performance.

[0065] 2.7 Fusion of Visible Light and Thermal Imaging Please see Figure 3 For multimodal data, this invention designs three fusion schemes: Early fusion: The registered visible light image (3 channels) and thermal imaging image (1 channel, or pseudo-color 3 channels) are stitched together along the channel dimension to form a 4-channel or 6-channel input, which is then fed into the backbone network. This scheme is simple and efficient, suitable for situations where the images are already accurately registered, and can fuse information from two modalities from the bottom layer.

[0066] Intermediate-level fusion: Two independent backbone networks are used to extract features from visible light and thermal imaging, respectively. These features are then fused in intermediate layers (such as after certain convolutional layers). Fusion methods can include channel concatenation, element-wise addition, or adaptive fusion of the two features through attention mechanisms (such as spatial attention or channel attention). Subsequent layers share parameters and perform joint optimization. This approach better preserves the unique features of each modality, exhibits stronger robustness, and is the preferred solution in this invention.

[0067] Post-processing fusion: Visible light and thermal imaging images are separately identified (i.e., each uses an independent multi-task network), and then fused at the decision layer. This includes weighted averaging of bounding boxes, voting fusion of segmentation results, and weighted averaging of intensity estimates. This scheme is suitable for situations where the registration accuracy between the two images is not high, but it involves significant computational cost.

[0068] 2.8 Enhanced Attention Mechanisms Please see Figure 4 This invention introduces multiple attention mechanisms into the backbone network and task branches, enabling the model to focus on key regions and important features: Spatial attention: Calculates the importance weight of each spatial location in the feature map. A spatial attention map is generated by performing global pooling and convolution operations on the feature map, and then multiplied with the original feature map to highlight flame and smoke areas and suppress background interference. Spatial attention makes the model pay more attention to local areas where flames may exist.

[0069] Channel attention: Calculates the importance weight of each channel in the feature map. Channel weight vectors are generated through global average pooling and fully connected layers, then the original feature maps are reweighted, adaptively selecting the most discriminative feature channels. For example, some channels may be sensitive to flame color, while others may be sensitive to smoke texture; channel attention can dynamically adjust the contribution of each channel.

[0070] Temporal attention: When processing video sequences, it calculates the importance weights of features at different times. By analyzing changes in temporal features, it highlights critical moments of dramatic fire changes, providing more valuable features for subsequent temporal predictions. Temporal attention helps models capture the dynamic evolution patterns of fires.

[0071] Example 3: Detailed Design of Fire Spread Prediction Network Please see Figure 5 and Figure 6 This embodiment details the structure and working principle of the fire spread prediction network.

[0072] 3.1 Input Feature Construction The input to the fire spread prediction network consists of three parts: Historical fire sequence: Fire identification results from the past T time points (e.g., T=12, one time point every 10 seconds), including flame detection bounding boxes, fire segmentation masks (binarized flame regions), and intensity levels. These results can be encoded into multi-channel feature maps; for example, generating a feature map of the same size as the original image for each time point, containing information such as flame regions, smoke regions, and intensity levels. The historical fire sequence contains the spatiotemporal trajectory of fire evolution.

[0073] Building structural features: Combustible material distribution map extracted from Building Information Modeling (BIM), represented as a binary mask (1 for combustible, 0 for non-combustible). It may also include prior information such as the thermal conductivity of building materials and the location of ventilation openings, encoded as a multi-channel feature map. Building structural features determine the possible paths of fire spread.

[0074] Environmental characteristics: Wind direction, wind speed, temperature, humidity, etc., extracted from sensor data can be encoded as global vectors or pixel-by-pixel fields. For example, wind direction can be represented as a vector field, and wind speed can be used as a global scalar. Environmental characteristics are key factors affecting the direction and speed of fire spread.

[0075] 3.2 Encoder-Decoder Architecture The prediction network employs an encoder-decoder architecture and incorporates a physical constraint layer: Encoder: Convolutional LSTM (ConvLSTM) or Convolutional Gated Recurrent Unit (ConvGRU) is used to encode the spatiotemporal features of historical fire sequences. ConvLSTM can simultaneously capture spatial structure and temporal dynamics. Its internal state is propagated in the time dimension, and the output hidden state vector H_t contains key information about the evolution of past fires. Unlike ordinary LSTM, ConvLSTM uses convolution operations instead of fully connected layers, which can preserve spatial structure.

[0076] Decoder: Based on the hidden state vector H_t, the spatial resolution is gradually restored through multiple deconvolutional layers or upsampling layers to decode and generate the fire distribution for the next τ time steps. The decoder outputs the fire probability map (representing the probability of each location being covered by fire), temperature distribution map (representing the predicted temperature value at each location), and smoke concentration distribution map (representing the smoke concentration at each location) for each future time step. The decoder can adopt a structure symmetrical to the encoder, progressively upsampling and fusing the encoder's features.

[0077] 3.3 Physical Constraint Layer Design Please see Figure 7 The physical constraint layer is used to incorporate the physical laws of fire into the deep learning model, including both hard and soft constraints: Combustible material mask constraint: Using the building's combustible material distribution map as a mask, a constraint is enforced on the decoder output: the predicted flame area can only appear at locations where the combustible material mask is 1. This can be achieved by multiplying the decoder output by the mask, ensuring that the prediction result will not appear in non-combustible areas (such as concrete walls or open areas), which conforms to the basic physical law that fire can only spread on combustible materials.

[0078] Physical equation residual constraints: Residual terms from the physical equations are added to the training loss function, prompting the model to learn predictions that conform to physical laws. Specifically, this includes: Thermal conduction constraint: It is assumed that the predicted temperature field T(x,y,t) should follow the thermal conduction equation. T / t = α ²T, add L_heat = || to the loss T / t - α ²T||², where α is the thermal diffusivity. This constraint ensures that temperature changes conform to the laws of heat conduction.

[0079] Smoke flow constraints: Smoke movement is affected by buoyancy and wind force, which can be addressed by introducing a simplified smoke diffusion model. For example, it can be assumed that the smoke diffusion direction should be consistent with the wind direction, and the concentration gradient should follow the convection-diffusion equation, with the addition of a corresponding loss term L_smoke.

[0080] Energy conservation constraint: The heat released by combustion should match the temperature rise and fire spread. An energy balance equation can be established to ensure that the predicted temperature rise is consistent with the energy consumed by fire spread, by adding a loss term L_energy.

[0081] The comprehensive physical constraint loss is L_physics = λ1·L_heat + λ2·L_smoke + λ3·L_energy, and the total loss function is L_total_pred = L_pred + L_physics, where L_pred is the error loss between the predicted and the true values ​​(e.g., MSE). Through this fusion of physical knowledge, the prediction results not only fit the data but also conform to physical laws, significantly improving the rationality and reliability of the predictions.

[0082] 3.4 Quantification of Uncertainty Fire prediction inherently involves uncertainty. This invention employs two methods to quantify this uncertainty: Bayesian deep learning introduces Dropout layers into the network and performs multiple forward propagations (Monte Carlo Dropout) during inference to obtain the distribution of predictions, thereby estimating the mean and variance. Variance reflects the model's uncertainty about the prediction results. For example, performing N forward propagations on the same input yields N predictions, and the mean and standard deviation are calculated as confidence intervals.

[0083] Ensemble learning involves training multiple independent prediction models (e.g., with different initializations and different subsets of data), and then combining the outputs of these models during inference to calculate the mean and variance. The variance reflects the inconsistency between the models, i.e., cognitive uncertainty.

[0084] Ultimately, the prediction results are presented in the form of mean ± confidence interval. For example, the probability that the fire will spread to coordinate A in 5 minutes is 80% ± 10%, which allows decision-makers to understand the credibility of the prediction and avoid blindly relying on unreliable predictions.

[0085] Example 4: Risk Assessment and Cloud Map Generation Please see Figure 8 This embodiment details the specific implementation of risk assessment and cloud map generation.

[0086] 4.1 Risk Factor Calculation For each spatial location (pixel or grid) within the target area, the following risk factors should be considered comprehensively: Fire spread probability P_spread: Obtained from the future flame probability map output by the fire prediction network, representing the probability that this location will be directly affected by the fire at a future time. Value range [0,1]. For example... Figure 11 The image shown is a cloud map for assessing the risk of fire spread according to the present invention.

[0087] Temperature danger level T_danger: Obtained from the predicted temperature map, the danger level is calculated based on the temperature value. For example, a temperature exceeding 60℃ (the limit of human tolerance) is dangerous. The Sigmoid function can be used to map the temperature to the range of 0-1: T_danger = 1 / (1 + e^(-k(T-T0))), where T0 is the threshold temperature and k is the steepness coefficient. Figure 12 The image shown is a cloud map for assessing the risk of thermal radiation according to the present invention.

[0088] Smoke hazard level S_danger: Obtained from the predicted smoke concentration map. The higher the smoke concentration, the greater the toxicity, and the higher the hazard level. It can be comprehensively assessed in conjunction with the concentration of toxic gases (CO, HCN). S_danger = min(1, w_c·C_smoke + w_g·C_gas), where C_smoke is the smoke concentration and C_gas is the normalized value of the toxic gas concentration.

[0089] Structural Danger Level (St_danger): This is an assessment based on the predicted impact of temperature on the building structure. For example, the strength of concrete structures begins to decline above 300°C and may crack above 500°C; the yield strength of steel structures decreases above 400°C and may become unstable above 600°C. The probability of structural instability can be calculated based on material properties and temperature thresholds. Figure 13 The diagram shown is a risk assessment cloud map for building weighing according to the present invention.

[0090] Secondary disaster risk (Sec_danger): This risk factor assesses the likelihood of secondary disasters based on the presence of flammable and explosive materials (such as gas stations, chemical plants, and flour mills), hazardous chemicals, etc., near the fire site, combined with the direction of fire spread and the distance of sparks. This risk factor is typically a discrete value (0 or 1), or a probability calculated based on distance and wind direction.

[0091] 4.2 Risk Scoring Model By combining multiple risk factors using a weighted summation or neural network model, a risk score for each location is obtained: Risk_score = w1·P_spread + w2·T_danger + w3·S_danger + w4·St_danger + w5·Sec_danger The weighting coefficients w1-w5 can be determined based on the actual application scenario through expert experience or machine learning. For example, in densely populated areas, the weight of smoke hazard can be increased; in important building areas, the weight of structural hazard can be increased. Alternatively, a neural network model can be used, taking multiple risk factors as input and outputting a risk score, with the optimal weights obtained through training on historical case data.

[0092] 4.3 Risk Level Classification Risk levels are assigned based on risk scores, and corresponding color mappings are used: Low risk: Risk_score < 0.3, corresponding to green, indicating a relatively safe zone. Medium risk: 0.3 ≤ Risk_score < 0.6, corresponding to yellow, indicating an area requiring caution. High risk: 0.6 ≤ Risk_score < 0.8, corresponding to orange, indicating a danger zone. Extremely high risk: Risk_score ≥ 0.8, corresponding to red, indicating an extremely dangerous area where immediate evacuation or emergency measures should be taken. 4.4 Risk Cloud Map Generation The risk levels of each location are overlaid onto the map or 3D model of the target area using color mapping to generate a risk cloud map: Spatial interpolation: For areas with sparse sensors, spatial interpolation methods are used to estimate the risk values ​​of unsampled points. Kriging interpolation is preferred, as it considers spatial autocorrelation and generates a continuous risk distribution map. Inverse distance weighted interpolation or spline interpolation can also be used.

[0093] Time-based animation: Based on predictions at multiple future points in time (e.g., 5 minutes, 10 minutes, 15 minutes), a series of risk cloud maps are generated and animated to dynamically display the evolution of risks over time. Commanders can view the risk distribution at any given moment by sliding the timeline.

[0094] 3D visualization: Risk cloud maps are overlaid onto 3D building models, supporting multi-angle viewing and interactive operations. Commanders can rotate and zoom the model to view the risk distribution inside and outside the fire scene from any perspective, which is particularly suitable for fires in high-rise buildings or complex structures.

[0095] Legends and annotations: Add legends to the visualization interface to explain the risk levels corresponding to the colors. Annotate key risk points (such as explosion risk areas, structures about to collapse) and early warning information to remind commanders to pay close attention.

[0096] Example 5: Model Training and Optimization Please see Figure 9 This embodiment details the training and optimization methods for the model.

[0097] 5.1 Training Data Construction To address the scarcity of fire data, this invention employs multiple methods to construct the training dataset: Real-world fire data collection: Collecting video, image, and sensor data from historical fire incidents and manually annotating them. Annotations include flame bounding boxes, flame / smoke region segmentation masks, and combustion intensity levels. Establishing a large-scale real-world fire dataset covering fire samples of different types (buildings, forests, industry) and scenarios (daytime, nighttime, indoors, outdoor).

[0098] Simulation Data Generation: Fire simulation software (such as FDS - Fire Dynamics Simulator) is used to generate simulated fire data under various scenarios. By setting different building structures, combustible material types, ventilation conditions, and meteorological parameters, the fire development process is simulated, and corresponding images and sensor data are automatically generated. The simulation data can automatically acquire accurate annotation information, greatly expanding the number of training samples, especially for the coverage of rare scenarios.

[0099] Data augmentation: Enhancing existing data online or offline, including: Geometric transformations: random rotation (±30°), scaling (0.8-1.2x), flipping (horizontal / vertical), cropping; Color transformation: brightness adjustment (±20%), contrast adjustment, saturation adjustment, hue perturbation; Noise injection: Gaussian noise, salt-and-pepper noise, blurring; Hybrid enhancement: CutMix, MixUp, and other similar methods combine multiple samples to generate new samples.

[0100] 5.2 Transfer Learning Accelerate training and improve performance with small sample sizes by using pre-trained models: ImageNet Pre-training: The backbone network parameters of the multi-task fire detection network are initialized using model weights pre-trained on the ImageNet large-scale image classification dataset. The ImageNet pre-trained model has learned rich general visual features (edges, textures, shapes, etc.) that can be transferred to fire detection tasks.

[0101] Pre-training for relevant tasks: For the flame detection and segmentation branches, weights pre-trained on the COCO object detection dataset or the PASCAL VOC semantic segmentation dataset can be used for initialization. These pre-trained models have already learned the ability to locate and segment objects and only need fine-tuning on fire data.

[0102] Fine-tuning strategy: Perform end-to-end fine-tuning of the entire network on the fire dataset. A hierarchical learning rate can be adopted, using a smaller learning rate for the backbone network (to preserve general features) and a larger learning rate for task branches (to learn task-specific features).

[0103] 5.3 Semi-supervised learning By leveraging a large amount of unlabeled fire video data, semi-supervised learning is used to further improve model performance: Pseudo-labeling method: First, train an initial model using labeled data, then use it to predict unlabeled data. Add high-confidence predictions (e.g., confidence > 0.9) to the training set as pseudo-labels, and iterate the training. The pseudo-labeling method effectively utilizes unlabeled data to expand the training sample.

[0104] Consistency regularization: When applying different data augmentations (such as rotation or color jitter) to the same image, the model is required to maintain consistency in the output of the two augmented versions. A consistency constraint is added to the loss function to make the model insensitive to input perturbations, improving robustness. Specifically, for unlabeled data, the difference in prediction results between different augmented versions (such as KL divergence or MSE) is calculated and added to the total loss as a consistency loss.

[0105] 5.4 Model Compression and Acceleration To meet real-time processing requirements, the trained model is compressed and accelerated: Knowledge distillation involves training a large deep network as a teacher model (such as ResNet-152) and then using it to guide the training of a small student network (such as MobileNetV3 or ShuffleNetV2). The student network learns to mimic the output distribution (soft labels) of the teacher network, significantly reducing the number of parameters and computational cost while maintaining high accuracy. Distillation loss typically includes the cross-entropy loss between the student's predictions and the ground truth labels, as well as the KL divergence loss between the student's and teacher's outputs.

[0106] Model pruning involves removing connections or channels in a network whose importance is below a preset threshold based on weight magnitude or gradient information. For example, the importance of channels in convolutional layers can be evaluated, and channels with smaller contributions can be removed. The pruned model can then be fine-tuned to restore accuracy. Pruning reduces the number of model parameters and computational cost, accelerating inference.

[0107] Quantization: Converts the model's weights and activation values ​​from 32-bit floating-point numbers (FP32) to 8-bit integers (INT8) or lower fixed-point numbers. Quantization can significantly reduce model storage space (by a factor of 4) and inference latency, especially on hardware that supports INT8 acceleration (such as NVIDIA TensorRT and Qualcomm Hexagon). Post-training quantization or quantization-aware training can be used to minimize accuracy loss.

[0108] Hardware acceleration: Inference is accelerated using hardware such as GPUs and TPUs, and optimized using inference engines such as TensorRT and ONNXRuntime, including operator fusion, memory optimization, and parallel scheduling, to further improve inference speed.

[0109] Example 6: Real-time Processing Optimization Please see Figure 10 This embodiment details the optimization strategy for real-time video stream processing.

[0110] 6.1 Keyframe Extraction For continuous video streams, not every frame needs to be fully recognized. When the scene changes little, intermediate frames can be skipped, and only keyframes can be extracted for processing. Inter-frame difference detection: Calculate the pixel change rate or histogram difference between adjacent frames. When the change exceeds a preset threshold, it is determined to be a keyframe. For example, when a rapid expansion of a flame area or the appearance of a new fire spot is detected, keyframe processing is triggered.

[0111] Motion detection: Based on optical flow or background subtraction, motion regions in the scene are detected. When the area of ​​the motion region exceeds a threshold, it is determined to be a keyframe.

[0112] Keyframe processing: Complete fire identification and prediction are performed on keyframes. For frames between keyframes, interpolation is used using optical flow or motion compensation techniques to obtain identification results, or the results from the keyframes can be directly used (suitable for slowly changing scenarios). Keyframe extraction significantly reduces computational load without losing important information.

[0113] 6.2 Multi-scale processing Different resolutions are used to process different regions of the image to balance accuracy and speed: Region importance classification: Based on historical recognition results and prediction results, the image is divided into key areas of interest (such as the flame area detected in the previous frame and the predicted fire spread area) and general background areas.

[0114] Adaptive resolution: High resolution (such as the original resolution) is used for fine identification of key areas of interest, while low resolution (downsampling by 2 times) is used for coarse processing of background areas. The original size is then restored through upsampling. Multi-scale processing can significantly reduce the computational load while ensuring the accuracy of flame detection.

[0115] 6.3 Incremental Update Leveraging the temporal continuity of video, the recognition result of the previous frame is used as prior information for the current frame: Detection box propagation: The flame detection box from the previous frame is used as the initial candidate box for the current frame. The position of the box in the current frame is predicted using optical flow or Kalman filtering. Then, local search and fine-tuning are performed only in a small area near the predicted box, avoiding global scanning and significantly improving detection speed.

[0116] Segmentation mask update: The segmentation mask of the previous frame is used as the initial mask of the current frame. The mask is transformed to the current frame using optical flow. Then, local corrections are only made near the boundaries, without the need for global segmentation to be performed again.

[0117] Intensity estimation reuse: If the current frame is not significantly different from the previous frame, the intensity estimation result of the previous frame can be directly used to reduce computation.

[0118] 6.4 Prediction-Assisted Search Using a fire spread prediction network to predict the location of flames in the next moment, the detection process is guided: Predicted region generation: Based on the output of the fire spread prediction network, generate the regions where the flames may appear in the future (such as regions obtained by thresholding the probability map).

[0119] Priority Search: In flame detection within the current frame, priority is given to searching and detecting within the predicted regions. These regions have a higher probability of containing flames, which improves detection recall while reducing searches in irrelevant regions.

[0120] Dynamic adjustment: Based on the difference between the actual detection results and the prediction, the weights of the prediction network are dynamically adjusted or the search strategy is replanned to form a feedback loop.

[0121] 6.5 Distributed Processing For multi-video stream or multi-view monitoring, a distributed processing architecture is adopted: Multi-device collaboration: Video from different perspectives is distributed to different computing nodes (edge ​​servers) for parallel processing, and the results are then aggregated at the central node. Each node processes one or more video streams, and the nodes communicate with each other via a high-speed network.

[0122] Pipeline parallelism: For a single video stream, the processing flow is divided into multiple stages, such as preprocessing, feature extraction, flame detection, segmentation, and prediction. Different stages are executed in parallel pipelines on different devices to improve throughput. For example, the GPU handles feature extraction and model inference, while the CPU handles preprocessing and post-processing.

[0123] Load balancing: Dynamically allocate tasks based on the computing power and load of each node to avoid some nodes being overloaded while others are idle.

[0124] Example 7: Specific Application Scenario (Factory Dormitory Building Fire) The following section uses a specific fire scenario to fully illustrate the implementation process and effects of the method of the present invention.

[0125] Scene Description: A fire breaks out on the fourth floor of a factory dormitory building. A large quantity of polyurethane insulation material is present at the scene, and the fire spreads rapidly. Firefighters arrive at the scene and deploy drones for aerial reconnaissance. Meanwhile, the building's fixed surveillance cameras remain operational.

[0126] Step 1: Data Acquisition (T=0 time) At 12:11, the drone arrived over the fire and began collecting data. Visible light camera: Captures RGB video at a resolution of 1920×1080, at a frame rate of 30fps; Thermal imaging camera: Captures infrared video at a resolution of 640×512, with a temperature range of -20℃ to 1500℃; Simultaneously acquire 3D building model (BIM data) and weather station data (wind direction north-northwest, wind speed 3m / s, temperature 25℃, humidity 40%).

[0127] Step 2: Data Preprocessing and Spatiotemporal Alignment The system performs spatial registration of visible light and thermal imaging images, mapping the thermal imaging images to the coordinate system of the visible light images. It also performs time synchronization of sensor data, aligning meteorological data with video frames. Finally, it performs denoising and enhancement processing on the images to improve image quality.

[0128] Step 3: Fire Identification (at time T=0) The preprocessed visible light and thermal imaging images of the current frame are input into the multi-task fire identification network: (1) Visible light analysis: Flame detection branch: A flame was detected in the east window on the fourth floor, with a confidence level of 0.95 and bounding box coordinates of (x1, y1, x2, y2). Fire division branches: The output flame area is about 15 square meters, the smoke area is about 50 square meters, and the smoke concentration distribution map shows that the smoke concentration is highest near the window; Intensity estimation branch: Output combustion intensity level 4 (out of 5), which belongs to intense combustion.

[0129] (2) Thermal imaging analysis: The temperature is highest on the east side of the fourth floor, reaching 650℃, forming a clear heat source center. The temperature distribution shows a gradient that decreases from the center outwards, with the floor temperature on the fifth floor starting to rise and reaching 80℃; Thermal imaging clearly shows the outline of the flames, unaffected by smoke obscuring the view.

[0130] (3) Fusion results: Fire location: East side dormitory room on the fourth floor, specific coordinates (X, Y, Z); Flame range: 4 meters long × 3.5 meters wide, with an area of ​​approximately 14 square meters; Combustion intensity: Level 4, indicating intense combustion; Smoke analysis: Thick black smoke, identified as burning plastic or polyurethane materials based on smoke type, with the smoke spreading northward.

[0131] Step 4: Fire Prediction (Predicting the future at time T=0) The fire spread prediction network takes the fire location, extent, and intensity sequence from the past two minutes (T=-2min to T=0), combines it with building structure (the fourth-floor dormitory has polyurethane insulation, and the fifth floor is also a dormitory) and environmental data (wind direction is north-northwest, wind speed is 3m / s), and inputs it to obtain future predictions: (1) T+5 minute prediction: The fire spread to the north corridor on the fourth floor, expanding to an area of ​​30 square meters (85% confidence level). The temperature on the fifth floor has risen to 150°C, posing a risk of the fire spreading upwards (60% probability). Smoke filled the fourth floor and spread to the fifth floor, reducing visibility on the fourth floor to less than 1 meter.

[0132] (2) T+10 minute prediction: If no measures are taken, the fire will spread to the entire fourth floor, covering an area of ​​approximately 100 square meters (70% confidence level). A fire may start on the east side of the fifth floor (45% probability), potentially leading to a three-dimensional fire. Smoke filled most of the fourth and fifth floors, and the concentration of toxic gases increased.

[0133] (3) T+15 minute prediction: The fire may spread to the fifth floor, creating a three-dimensional fire (probability 55%). The building structure was affected by the high temperature, with the load-bearing wall on the fourth floor reaching a temperature of 400°C, which reduced its load-bearing capacity and posed a risk of partial collapse (probability 20%). The sparks may drift to a flour mill 30 meters to the east, posing a risk of dust explosion.

[0134] Step 5: Risk Assessment and Cloud Map Generation Based on the identification and prediction results, risk cloud maps for the current and future moments are generated: (1) Current risk (T=0): East side of the fourth floor: Extremely high risk (red), direct threat of flames, temperature exceeds 600℃; Fourth floor north corridor: High risk (orange), high smoke concentration, rising temperature, fire may spread; Fifth floor, east side: Medium risk (yellow), temperature rises to 80℃, there is a risk of fire spreading upwards; Other areas: Low risk (green), relatively safe.

[0135] (2) Future risks (T+10 minutes): Fourth floor: Extremely high risk (red), fire covers the entire floor; Fifth floor, east side: High risk (orange), potential fire, temperature rising; Other areas on the fifth floor: Medium risk (yellow), smoke is spreading, caution is required.

[0136] (3) Special risk labeling: The flour mill located 30 meters to the east is marked as a secondary disaster risk point and is warned with a flashing orange icon. Fourth floor load-bearing wall: Mark structural danger points and indicate areas that may collapse with orange dashed boxes.

[0137] Step 6: Decision Support Output The system automatically generates decision recommendations, which are displayed on the command terminal. Priority firefighting areas: It is recommended to concentrate efforts on controlling the fire source on the east side of the fourth floor to prevent it from spreading to the fifth floor and the flour mill. Fire extinguishing agent recommendation: If polyurethane material combustion is detected, it is recommended to use Class B foam fire extinguishing agent, with a mixing ratio of 6% and a foam coverage thickness of ≥15cm. Evacuation: Prioritize evacuating people trapped on the fifth floor, especially those in the east-facing rooms. It is recommended to use the east-facing staircase (where there is currently less smoke). External protection: Firebreaks were set up around the flour mill to prevent sparks from igniting dust explosions; Estimated time: If the above measures are taken, the fire is expected to be under control in 20-30 minutes.

[0138] Step 7: Dynamic Update (T=5 minutes) Five minutes later, the drone continued to collect new data, and the system updated the identification and prediction results: Actual fire situation: Due to the timely measures taken by firefighters, the fire was initially brought under control, the flames did not expand, and the fire was still on the east side of the fourth floor. The temperature dropped from 650℃ to 500℃. Updated forecast: Based on the latest data, the probability of the fire spreading to the fifth floor has been reduced from 60% to 20%, and the fire is expected to be completely under control within 10 minutes. Risk Update: The risk level on the fifth floor has been downgraded from medium to low, and the risk warning for the flour mill has been lifted.

[0139] As can be seen from the above embodiments, the method of the present invention can identify fire and predict its development trend in real time and accurately, providing scientific decision support for fire rescue and significantly improving the efficiency and safety of fire emergency response. Multimodal fusion ensures identification capability in dense smoke environments, temporal prediction provides valuable lead time, physical constraints guarantee the rationality of prediction, and risk cloud map intuitively displays the fire situation, together constituting a complete intelligent fire identification and prediction solution.

[0140] Example 8: Alternatives and Variations The various modules and steps in this invention can be implemented in multiple alternative ways, and those skilled in the art can make adjustments according to actual needs, all of which fall within the protection scope of this invention.

[0141] 8.1 Replacement of Backbone Networks In addition to ResNet, EfficientNet, and Swing Transformer, VGG, MobileNet, DenseNet, RegNet, and ConvNeXt can also be used as backbone networks. The choice depends on accuracy and speed requirements; for mobile deployments, MobileNet or ShuffleNet can be selected, while for server deployments, ResNet or Swing Transformer can be chosen.

[0142] 8.2 Alternatives to Object Detection Algorithms In addition to YOLO and Faster R-CNN, other object detection algorithms such as SSD, RetinaNet, CenterNet, DETR, and FCOS can also be used. DETR is based on the Transformer architecture and does not require anchor point design; FCOS is a fully convolutional single-stage detector with a simple structure.

[0143] 8.3 Alternatives to Semantic Segmentation Algorithms Besides U-Net and DeepLab, other network segmentation tools such as FCN, PSPNet, SegNet, Mask R-CNN (for instance segmentation), and SETR (Transformer-based) can also be used. For scenarios requiring differentiation of different flame instances, Mask R-CNN can be used for instance segmentation.

[0144] 8.4 Alternatives to Time Series Models In addition to ConvLSTM and ConvGRU, PredRNN, E3D-LSTM, Video Transformer, TimeSformer, and TSM (Time Shift Module) can also be used. PredRNN adds a spatiotemporal memory unit to ConvLSTM to capture more complex spatiotemporal dependencies.

[0145] 8.5 Alternatives to Integration Methods In addition to early, mid, and late-stage fusion, cross-attention fusion, graph neural network fusion, and dynamic fusion can also be used. Cross-attention fusion achieves fine-grained feature interaction by calculating the mutual attention matrix of features from two modalities; dynamic fusion can adaptively select fusion weights based on the input content (such as weather and lighting conditions).

[0146] 8.6 Substitution of Physical Constraints Besides incorporating physical residuals into the loss function, other methods such as Physical Information Neural Networks (PINN), Neural ODEs, and physics-guided data augmentation can also be used. PINN directly embeds physical equations into the network structure and calculates physical residuals through automatic differentiation; Neural ODE uses neural networks to parameterize differential equations, integrating physical laws into dynamic system modeling.

[0147] 8.7 Expansion of Application Scenarios This invention is not only applicable to building fires, but can also be applied to the following scenarios by adjusting training data and model parameters: Forest fires: Factors such as terrain slope, vegetation type, wind direction and speed need to be considered; the distribution of combustibles is obtained from satellite remote sensing data. Industrial fires (chemical plants, oil refineries): Factors such as chemical properties, explosion risk, and toxic gas diffusion must be considered. Vehicle fires: including fires involving vehicles in tunnels, vehicles in parking lots, and electric vehicle batteries. Electrical fires: Factors such as the characteristics of electrical equipment and the timing of power outages need to be considered.

[0148] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for intelligent fire identification and prediction based on deep learning, characterized in that, Includes the following steps: Acquire multimodal data of the target area, including visible light image sequences, thermal imaging image sequences, environmental sensor data, and building structure data; The visible light image sequence and thermal imaging image sequence are input into a pre-trained multi-task fire recognition network. Multi-scale fusion features are extracted through a shared backbone network, and the flame detection task, fire area segmentation task, and combustion intensity estimation task are performed in parallel using the multi-scale fusion features. The corresponding outputs are the flame position, fire spread range, smoke distribution information, and combustion intensity level at the current moment. The flame location and fire spread range output at multiple moments within a historical time period are combined with the combustible material distribution characteristics extracted from the building structure data and the wind direction and speed characteristics extracted from the environmental sensor data, and input into the fire spread prediction network. The fire spread prediction network is constructed based on a temporal convolutional network and, by introducing the physical laws of fire as constraints, outputs the fire spread prediction distribution within a preset time period in the future. The fire spread prediction distribution includes the flame spread range, temperature distribution, and smoke concentration distribution. Based on the predicted distribution of fire intensity within the preset future time frame, the risk score for each location within the target area is calculated by combining multiple risk factors, and the risk level is divided. A dynamic risk cloud map is generated through color mapping. The dynamic risk cloud map is visualized, and decision support information including priority fire extinguishing areas, personnel evacuation routes, and fire resource allocation suggestions is generated based on the prediction results.

2. The deep learning-based intelligent fire identification and prediction method according to claim 1, characterized in that, The multi-task fire identification network includes a shared backbone network and flame detection branch, fire segmentation branch and intensity estimation branch respectively connected to the backbone network; The flame detection branch outputs the bounding box of the flame based on the target detection algorithm; the fire segmentation branch outputs pixel-level classification results of the flame region and smoke region based on the semantic segmentation algorithm, and further estimates the smoke concentration distribution and identifies the smoke type. The intensity estimation branch outputs the combustion intensity level based on a regression network.

3. The deep learning-based intelligent fire identification and prediction method according to claim 2, characterized in that, The visible light image sequence and the thermal imaging image sequence are fused in a multimodal manner through at least one of the following methods: channel-dimensional splicing is performed before input to the backbone network to achieve early fusion; or features are extracted through two independent backbone networks and feature fusion is performed in the middle layer of the network through an attention mechanism to achieve mid-term fusion; or the two modalities are fully identified separately and then fused in the decision layer to achieve late-term fusion.

4. The method for intelligent fire identification and prediction based on deep learning according to claim 1, characterized in that, The fire spread prediction network includes: The encoder is used to encode the spatiotemporal features of the flame location and fire spread range within the historical time period to obtain the hidden state vector; The physical constraint layer is used to use the combustible material distribution characteristics as a mask to constrain the predicted fire spread range to be within the combustible material area, and to add residual terms of the heat conduction equation, flue gas flow equation or energy conservation equation as physical constraints during the training process. The decoder is used to decode and generate a fire prediction distribution within a preset time period based on the hidden state vector and the output of the physical constraint layer.

5. The deep learning-based intelligent fire identification and prediction method according to claim 4, characterized in that, The method further includes constructing the fire spread prediction network using a Bayesian neural network or an ensemble learning method, such that the output fire prediction distribution includes a confidence interval to characterize the prediction uncertainty.

6. The method for intelligent fire identification and prediction based on deep learning according to claim 1, characterized in that, The risk factors include the probability of fire spread, the predicted temperature hazard, the smoke concentration hazard, the building structure hazard, and the risk of secondary disasters; the risk score is obtained by combining the risk factors through a weighted summation model or a neural network model; the risk level is divided into four levels: low, medium, high, and extremely high, and is visualized in green, yellow, orange, and red, respectively.

7. The method for intelligent fire identification and prediction based on deep learning according to claim 1, characterized in that, The method also includes model training and optimization steps: constructing a training dataset using real fire data, simulated data generated by fire simulation software, and data augmentation techniques; Transfer learning was used to initialize network parameters, and semi-supervised learning was combined with unlabeled data for consistency regularization training. The trained model can be compressed and accelerated using knowledge distillation, model pruning, or quantization techniques to adapt to the real-time deployment of edge computing devices.

8. The method for intelligent fire identification and prediction based on deep learning according to claim 1, characterized in that, The method employs a keyframe extraction strategy in real-time video stream processing: key frames are determined by inter-frame difference detection or motion detection, and only key frames are used for complete fire identification and prediction; for non-key frames, optical flow or motion compensation techniques are used for result interpolation or the identification result of the previous frame is reused.

9. The method for intelligent fire identification and prediction based on deep learning according to claim 1, characterized in that, The method further includes a step of performing spatiotemporal alignment preprocessing on the multimodal data: performing spatial registration based on feature points on visible light images and thermal imaging images from different sources to achieve pixel-level alignment; and synchronizing data from different sensors through hardware triggering or software timestamps to ensure that data at the same moment correspond to the same fire scene state.

10. A deep learning-based intelligent fire identification and prediction system, used to implement the method as described in any one of claims 1 to 9, characterized in that, include: A multimodal data acquisition module is used to acquire multimodal data of a target area, including visible light image sequences, thermal imaging image sequences, environmental sensor data, and building structure data. The data preprocessing module is used to perform spatiotemporal alignment and enhancement preprocessing on the multimodal data; The multi-task fire identification module is used to input the pre-processed visible light image sequence and thermal imaging image sequence into the pre-trained multi-task fire identification network. It extracts multi-scale fusion features through a shared backbone network and uses the multi-scale fusion features to perform flame detection, fire area segmentation and combustion intensity estimation tasks in parallel, and outputs the flame position, fire spread range, smoke distribution information and combustion intensity level at the current moment. The fire spread prediction module is used to input the flame location and fire spread range output at multiple times within a historical time period, combined with the combustible material distribution characteristics extracted from the building structure data and the wind direction and speed characteristics extracted from the environmental sensor data, into the fire spread prediction network; the fire spread prediction network is constructed based on a temporal convolutional network and outputs the fire spread prediction distribution within a preset time period by introducing the physical laws of fire as constraints. The risk assessment and cloud map generation module is used to calculate the risk score of each location in the target area based on the fire prediction distribution within the preset future time period, combined with multiple risk factors, to classify the risk level, and generate a dynamic risk cloud map through color mapping. The decision support output module is used to visualize the dynamic risk cloud map and generate decision support information based on the prediction results, including priority fire extinguishing areas, personnel evacuation routes, and fire resource allocation suggestions.

Images