A Fire Situation Decision-Making Method and System Based on AI Dynamic Temperature Prediction and Multimodal Fusion

Abstract

This invention relates to the field of fire monitoring and emergency command technology, and discloses a method and system for fire situation decision-making based on AI dynamic temperature prediction and multimodal fusion. The method includes: collecting multimodal data from the fire scene, including visual image data and non-visual parameter data; dividing the fire scene area into grids, assigning a unique code to each grid cell, and associating the multimodal data with the corresponding grid cell code; inputting the multimodal data into a pre-trained fire situation analysis model; the fire situation analysis model dynamically fusing the extracted visual and non-visual features through an attention mechanism, and making predictions based on the fused features to output the situation analysis results; and generating visualized fire situation information and structured decision instructions based on the situation analysis results. This invention achieves accurate perception, phased prediction, and intelligent command of the fire situation.

Description

Technical Field

[0001] This invention relates to the field of fire monitoring and emergency command technology, specifically to a fire situation decision-making method and system based on AI dynamic temperature prediction and multimodal fusion. Background Technology

[0002] Fire, as a serious disaster threatening life, property, and the ecological environment, presents a complex and multifaceted situation, requiring differentiated firefighting tactics at different stages of its development. Currently, fire monitoring and decision-making primarily rely on video footage transmitted from fire command vehicle cameras and drones, allowing for visual assessment and command. However, single-modal monitoring has significant limitations: RGB images are easily affected by lighting and smoke, making it difficult to accurately identify flame areas; thermal imaging images lack an absolute thermal reference point, easily mistaking high-temperature objects for flames, and exhibit poor edge detail; furthermore, existing technologies lack effective dynamic sensing and prediction methods for temperature—a core parameter determining fire development—failing to accurately determine the fire's stage and trend, potentially leading to inappropriate decisions, impacting firefighting effectiveness, and posing safety risks.

[0003] Existing fire detection models mostly focus on single-task optimization or are designed only for specific scenarios, lacking an effective mechanism for fusing multimodal data and making it difficult to adapt to complex and ever-changing fire environments. Although some models attempt to combine multimodal data, they have shortcomings in feature fusion weight allocation, loss function optimization, and cross-scenario adaptability, resulting in low temperature prediction accuracy and limited accuracy in fire stage classification, failing to provide timely and reliable decision support for fire command.

[0004] Therefore, developing an intelligent decision-making system that can integrate multimodal data and achieve dynamic temperature prediction and accurate situation assessment has become a key issue that urgently needs to be addressed in the field of fire rescue. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing fire monitoring and decision-making technologies and provide a fire situation decision-making system based on AI dynamic temperature prediction and multimodal fusion. By integrating infrared thermal imaging and visible light multimodal data acquisition, AI dynamic temperature prediction, deep metric learning (DML) fusion algorithm and visualization decision support, it can achieve accurate perception, stage prediction and intelligent command of fire situation, and can be widely used in fire rescue, forest fire prevention, industrial fire safety and other fields.

[0006] To achieve the above objectives, the following technical solution is adopted:

[0007] In a first aspect, embodiments of the present invention provide a fire situation decision-making method based on AI dynamic temperature prediction and multimodal fusion, comprising: collecting multimodal data of the fire scene, wherein the multimodal data includes visual image data and non-visual parameter data; wherein the visual image data includes RGB images and thermal imaging images with absolute temperature calibration; the non-visual parameter data includes environmental parameter time-series data and combustible material type data; dividing the fire scene area into grids, assigning a unique code to each grid cell, and associating the multimodal data with the corresponding grid cell code; and inputting the multimodal data into a pre-trained fire situation analysis system. The fire situation analysis model is used to obtain the situation analysis results output by the fire situation analysis model. The fire situation analysis model uses an attention mechanism to dynamically fuse visual features extracted from the visual image data and non-visual features extracted from the non-visual parameter data, and makes predictions based on the fused features to output the situation analysis results. The situation analysis results include at least future temperature predictions, fire stage classification results, and fire risk levels. The situation analysis results are organized and output in units of grid cells. Based on the situation analysis results, fire situation visualization information and structured decision instructions are generated.

[0008] Furthermore, the method also includes: constructing a training dataset for training the fire situation analysis model, including data augmentation processing of RGB images and thermal imaging image pairs in the training dataset, including: photometric enhancement of the RGB images and nonlinear enhancement of the thermal imaging images based on relative temperature distribution; the nonlinear enhancement is: identifying the flame core region and background region in the thermal imaging image, keeping the pixel temperature value of the flame core region unchanged, and adding random noise or performing local contrast calibration on the pixel temperature value of the background region.

[0009] Furthermore, the fire situation analysis model is an end-to-end deep learning model, comprising: an input layer for receiving the multimodal data; a feature extraction layer for extracting visual features from the RGB images and thermal imaging images, extracting temporal features from the environmental parameter time-series data, and converting the combustible material type encoding into a feature vector; wherein the visual features are extracted from the RGB images and thermal imaging images by a U-shaped network with a shared encoder; the temporal features are extracted from the environmental parameter time-series data by a one-dimensional convolutional and long short-term memory network; a dynamic fusion layer for dynamically weighting and fusing the visual features, temporal features, and combustible material feature vectors using an attention mechanism to generate a unified multimodal fusion feature; a prediction layer including a temperature time-series prediction branch and a fire stage classification branch, wherein the temperature time-series prediction branch is used to output the temperature prediction value for at least one future time step, and the fire stage classification branch is used to output the fire stage classification result; and an output layer for outputting a situation analysis result that includes at least the temperature prediction value, the fire stage classification result, and the fire risk level.

[0010] Furthermore, the process by which the dynamic fusion layer assigns dynamic weights to various non-visual features through an attention mechanism includes: calculating the fusion similarity between each type of non-visual feature and the visual feature, wherein the fusion similarity is based on the cosine similarity between the non-visual feature and the visual feature, and is calculated in conjunction with a temperature importance function related to the region temperature value; assigning attention weights to each type of non-visual feature according to the fusion similarity using a Softmax function; weighting various types of non-visual features according to the attention weights, and fusing the weighted non-visual features with the visual features to obtain the multimodal fusion feature.

[0011] Furthermore, the total loss function for training the fire situation analysis model is a weighted sum of the classification task loss, temperature prediction task loss, and feature reconstruction task loss. During training, a gradient normalization strategy is used to dynamically adjust the adaptive weights corresponding to the losses of each task. The gradient normalization strategy includes: normalizing the real-time gradient of each task based on the norm of the initial gradient, and dynamically calculating and constraining the adaptive weights of each task based on the norm of the normalized gradient.

[0012] Furthermore, the predicted future temperature is output by the temperature time-series prediction branch of the fire situation analysis model; the temperature time-series prediction branch adopts a hybrid model combining a Long Short-Term Memory (LSTM) network and a cellular automaton. The cellular automaton is pre-trained using fire spread data generated by fire dynamics simulation software and fine-tuned using real fire spread data; the fire stage classification branch specifically includes: a fully connected layer, whose input is the multimodal fusion feature output by the dynamic fusion layer; and a classifier, used to generate fire stage classification results including at least four stages: initial growth, sudden burst, full development, and decay, based on the output of the fully connected layer.

[0013] Furthermore, the classification task loss adopts an improved Lovász-Softmax loss function, which introduces a fire context weighting mechanism and a sample ambiguity coefficient to dynamically adjust the contribution of different samples to the total loss based on the combustible material type and environmental parameters. The fire context weight is calculated jointly based on the combustible material combustion coefficient, temperature, humidity, and wind speed parameters, and the sample ambiguity coefficient is used to assign higher learning weights to boundary samples of the fire stage.

[0014] Furthermore, the method also includes: dividing the fire area into grids, assigning a unique code to each grid cell, and associating the multimodal data with the corresponding grid cell code; generating fire situation visualization information includes: dynamically coloring the grid cells with different colors based on the fire stage classification results and risk levels; generating structured decision instructions includes: integrating the grid cell code, predicted temperature, fire stage, and combustible material type information to generate command instruction text containing specific locations and operational suggestions.

[0015] Furthermore, the method also includes a fire anomaly identification step: adaptively calculating a dynamic temperature gradient threshold based on the type of combustible material and ambient humidity; comparing the actual temperature gradient of the grid cell in the fire scene with the dynamic temperature gradient threshold; if the actual temperature gradient continuously exceeds the threshold, it is determined that there is a hidden fire source; if the temperature continuously exceeds a preset value but the temperature gradient is below the lower limit, it is determined that there is a potential area for reignition.

[0016] Secondly, embodiments of the present invention also provide a fire situation decision-making system based on AI dynamic temperature prediction and multimodal fusion, comprising:

[0017] A multimodal data acquisition module is used to acquire multimodal data from a fire scene. The multimodal data includes visual image data and non-visual parameter data. The visual image data includes RGB images and thermal imaging images with absolute temperature calibration. The non-visual parameter data includes environmental parameter time series data and combustible material type data.

[0018] The spatial grid division module is used to divide the fire area into grids, assign a unique code to each grid cell, and associate the multimodal data with the corresponding grid cell code.

[0019] The fire situation analysis module is used to input the multimodal data into a pre-trained fire situation analysis model to obtain the situation analysis results output by the fire situation analysis model. The fire situation analysis model uses an attention mechanism to dynamically fuse visual features extracted from the visual image data and non-visual features extracted from the non-visual parameter data, and makes predictions based on the fused features to output the situation analysis results. The situation analysis results include at least future temperature predictions, fire stage classification results, and fire risk levels.

[0020] The decision output module generates fire situation visualization information and structured decision instructions based on the situation analysis results and the grid cell encoding.

[0021] Compared with the prior art, the present invention achieves the following beneficial effects:

[0022] 1. Multimodal fusion improves perception accuracy: This invention integrates RGB-thermal imaging visual modal data with environmental parameters and non-visual modal data of combustible material type. Through nonlinear enhancement strategies and dynamic fusion mechanisms, it effectively avoids the limitations of a single modality, significantly improving the accuracy of flame area recognition and the reliability of temperature perception. It can still accurately capture fire characteristics even in smoke-covered scenarios.

[0023] 2. Outstanding dynamic prediction capability: It adopts a hybrid model of "LSTM + cellular automata" and combines FDS simulated data pre-training with real data fine-tuning to achieve accurate temperature prediction for the next 5 time steps, with MAE≤3℃; it achieves accurate matching of fire stages based on a multi-type temperature-time curve library, with a stage classification accuracy of ≥95%, and can predict dangerous states such as the sudden explosion stage in advance, thus gaining valuable time for rescue.

[0024] 3. Precise and efficient anomaly identification: A dynamic temperature gradient threshold based on the type of combustible material and ambient humidity is proposed to effectively identify hidden fire sources and potential reignition hazards. The false negative rate (FNR) of fire detection is ≤11.6%, reducing missed and misjudgments and lowering safety risks during rescue operations.

[0025] 4. Intuitive and practical decision support: The dual-grid coding system enables precise positioning of the fire scene, and the dynamic coloring of the grid and the marking of anomalies make the situation clear at a glance; the structured command instructions are clear and specific, which can directly guide firefighting operations and greatly improve command efficiency and the scientific nature of operations.

[0026] 5. Stable and reliable model optimization: Through deep metric learning and multi-task loss optimization strategies, combined with gradient normalization weight adaptive adjustment, the stability and convergence speed of model training are guaranteed. The accuracy of adaptation to different fire types is ≥92%, and it has good generalization ability.

[0027] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of the present invention, nor is it intended to restrict the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0028] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the invention. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0029] Figure 1 This is a flowchart illustrating the fire situation decision-making method based on AI dynamic temperature prediction and multimodal fusion provided in an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of the fire situation analysis model architecture in an embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the fire temperature-time curve and stage matching in an embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of a forest fire situation provided in an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of the modules of the fire situation decision-making system based on AI dynamic temperature prediction and multimodal fusion provided in an embodiment of the present invention;

[0034] Figure 6 A schematic diagram of the edge-cloud layered processing architecture provided in this embodiment of the invention;

[0035] Figure 7 These are the performance comparison results of the models in the embodiments of the present invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0038] Figure 1 This is a flowchart illustrating a fire situation decision-making method based on AI dynamic temperature prediction and multimodal fusion, provided in an embodiment of the present invention. A fire situation decision-making method based on AI dynamic temperature prediction and multimodal fusion includes:

[0039] S1: Collect multimodal data of the fire scene, including visual image data and non-visual parameter data; wherein, the visual image data includes RGB images and thermal imaging images with absolute temperature calibration; the non-visual parameter data includes environmental parameter time series data and combustible material type data; divide the fire scene area into grids, assign a unique code to each grid cell, and associate the multimodal data with the corresponding grid cell code;

[0040] Step S1 is used to construct and preprocess the multimodal dataset. Specifically, it includes the following steps:

[0041] S11: Dataset Design

[0042] (1) Data scale and sources: A basic dataset containing 3,352 original multimodal data pairs was constructed and expanded to 10,698 pairs after data augmentation. Data sources include: RGB-thermal imaging image pairs collected from real fire scenes (thermal imaging images contain temperature calibration information), measured data of smoke-covered scenes, records of planned combustion experiments, and simulated smoke scenes without fire sources (such as industrial smoke and natural fog); corresponding non-visual modal data (temperature, humidity, wind speed, precipitation, and combustible material type) were collected simultaneously. All data came from sensors carried by fire-fighting drones, fixed monitoring stations, and publicly available data from meteorological departments.

[0043] (2) Labeling mechanism: At least three experts in the field of fire protection conduct double-blind labeling based on the "Forest Fire Weather Rating" and "Building Fire Emergency Response Code". The labeling content includes flame area, fire stage, risk level and time-series temperature characteristics, and the labeling consistency coefficient is 0.892. The method of automatic labeling and manual correction is adopted. The initial labeling is generated based on the RGB image unobstructed flame labeling algorithm (FlameSeg), and then verified and optimized by experts to improve the labeling efficiency and accuracy.

[0044] (3) Risk label definition: High-risk samples are real fire events and planned combustion experiment data, which meet the characteristic of continuous temperature rise of "temperature rise rate ≥ 2℃ / min for three consecutive time steps" and include flame spread trajectory marking; Low-risk samples are non-fire source smoke scenarios and fire smoke observation data under extreme weather conditions, with temperature change rate ≤ 0.5℃ / min; Critical risk samples are smoldering stage and reignition hazard area data, with abnormal temperature gradient characteristics (temperature fluctuation ≥ 3℃ for two consecutive time steps).

[0045] S12: Data Augmentation Strategy

[0046] (1) Geometric enhancement: Perform horizontal flip (probability: 0.5), ±15° restricted rotation, and 256×256 pixel random cropping on RGB and thermal imaging images to maintain the rationality of image spatial structure and adapt to different monitoring angle scenarios.

[0047] (2) Luminous enhancement: The RGB image is adjusted by ±20% brightness, Gaussian noise (variance: 0.01-0.03) is added, and motion blur (kernel size: 3×3~5×5) is processed; the thermal image adopts a nonlinear enhancement strategy based on relative temperature distribution: by threshold segmentation (T≥300℃ is determined as the core area of ​​the flame), the core area of ​​the flame and the background area in the thermal image are identified, the pixel temperature value of the core area of ​​the flame remains unchanged, and only the background area (T<300℃) is added with ±2℃ temperature noise, or local contrast calibration (adaptive adjustment based on temperature distribution) is used to avoid physical calibration distortion of high temperature areas caused by global scaling.

[0048] (3) Cross-layer combination enhancement: Randomly combine geometric enhancement and thermal imaging nonlinear enhancement operations (such as rotation + background temperature noise addition, cropping + local temperature contrast calibration) to generate complex scene data and improve the model's generalization ability.

[0049] (4) Non-visual data enhancement: Gaussian perturbation (fluctuation range ±5%) and time series interpolation are used to expand numerical data such as temperature and humidity to simulate the dynamic changes of environmental parameters; unique thermal coding is used to expand combustible material type data to cover 10 typical combustible materials such as trees, shrubs and building materials.

[0050] S13: Mesh-based spatial modeling

[0051] Step S13 is used to divide the fire area into grids, assign a unique code to each grid cell, and associate the multimodal data with the corresponding grid cell code.

[0052] (1) Dual grid coding system: Metric grid (default) divides the fire area into standard grids with a resolution of 5m×5m, and adopts "XY coordinate coding" (format: X+3 digits+Y+3 digits, such as X001Y001, X123Y456), which supports unlimited range expansion and is suitable for scenarios that require precise positioning, such as buildings and cities; Latitude and longitude grid (compatible) is used for open areas in the field, and adopts latitude and longitude grid coding (such as N30.1234°E120.5678°), which is divided with an accuracy of 0.00005° (about 5.5m×5.5m near the equator, which is similar to the accuracy of the metric grid).

[0053] (2) Grid transformation relationship: It is clear that 1° latitude ≈ 111km and 1° longitude ≈ 111×cos (latitude) km. The two grids are transformed in real time through the Universal Transverse Mercator (UTM) coordinate system to ensure the consistency of spatial resolution.

[0054] (3) Grid attribute association: Associate multimodal data (visual image blocks, temperature time series data, environmental parameters, combustible material types) to support refined area management and command.

[0055] S2: Input the multimodal data into a pre-trained fire situation analysis model to obtain the situation analysis results output by the fire situation analysis model; wherein, the fire situation analysis model dynamically fuses visual features extracted from the visual image data and non-visual features extracted from the non-visual parameter data through an attention mechanism, and makes predictions based on the fused features to output the situation analysis results; the situation analysis results include at least future temperature predictions, fire stage classification results, and fire risk levels; and the situation analysis results are organized and output in units of the grid cells;

[0056] Step S2 is used to construct and train an AI dynamic temperature prediction and fire detection model, and to perform fire situation analysis based on the trained fire situation analysis model.

[0057] S21: Overall structural design of the model

[0058] like Figure 2The diagram shown is a schematic of the fire situation analysis model architecture in an embodiment of the present invention. The model adopts an end-to-end deep learning model with an architecture of "multimodal feature extraction - dynamic fusion encoding - temperature prediction - stage classification - decision output", as detailed below:

[0059] (1) Input layer: Receives multimodal data, including: RGB image (3×256×256), temperature calibration thermal imaging image (1×256×256, including absolute temperature data), environmental parameter time series (temperature, humidity, wind speed, precipitation, dimension 4×T, T is the time step), and combustible material type code (dimension 10).

[0060] (2) Feature extraction layer: Visual features (number of channels) are extracted from RGB and thermal images by a U-Net encoder with shared parameters. =64); environmental parameters are extracted as temporal features (64 dimensions) using a 1D convolutional + Long Short-Term Memory (LSTM) network; combustible material types are converted into feature vectors (32 dimensions) through an embedding layer.

[0061] (3) Dynamic fusion layer: The weights of environmental parameters and combustible features are dynamically allocated through the attention mechanism and fused with visual features to generate a unified multimodal fusion feature vector (64 dimensions).

[0062] (4) Prediction layer: includes temperature time series prediction branch (integrated cellular automaton fire spread model, outputting temperature values ​​for the next 5 time steps) and fire stage classification branch (outputting four classification results: initial growth, sudden burst, full development, and decay).

[0063] (5) Output layer: output grid temperature prediction, fire stage classification results, stage transition remaining time, risk level, and combustible combustion risk coefficient.

[0064] It should be noted that the situation analysis results output by the output layer include at least the temperature prediction value, fire stage classification results and fire risk level. It can also output auxiliary information such as the remaining time of stage transition, combustible combustion risk coefficient and anomaly identification results as needed.

[0065] S22: Core Algorithm Design

[0066] S221: Multimodal Dynamic Fusion Mechanism

[0067] Furthermore, the dynamic fusion layer assigns dynamic weights to various non-visual features through an attention mechanism, including: calculating the fusion similarity between each type of non-visual feature and the visual feature, whereby the fusion similarity is based on the cosine similarity between the non-visual and visual features and is calculated in conjunction with a temperature importance function related to the region's temperature value; assigning attention weights to each type of non-visual feature based on the fusion similarity using a Softmax function; weighting each type of non-visual feature according to the attention weights; and fusing the weighted non-visual features with the visual features to obtain a multimodal fusion feature. The specific process is as follows:

[0068] (1) Visual modal processing: RGB images are used for flame annotation and feature calibration in smoke-free scenes, and thermal imaging images are used for temperature distribution and flame contour extraction under smoke cover. Through the Local Feature Extraction (LFE) mechanism, the thermal imaging image and the flame mask are subjected to Hadamard product operation to focus on the core region features of the flame.

[0069] (2) Non-visual modality weight allocation: Environmental parameters, temporal features, and combustible material type features are dynamically weighted by the attention module. The similarity calculation adopts the "cosine similarity + temperature-guided mechanism", and the formula is as follows:

[0070]

[0071]

[0072] The similarity between non-visual and visual features is used to dynamically assign weights to non-visual features. Non-visual characteristics, specifically including environmental parameters (temperature, humidity, wind speed, precipitation) and characteristics of combustible material type; Visual features are extracted from RGB images and thermal images using a U-Net encoder. : Dot product operation between non-visual feature vectors and visual feature vectors; L2 norm of non-visual feature vectors; L2 norm of visual feature vectors; Temperature guiding coefficient, preferably set to a fixed value of 0.3, is used to adjust the weight of the influence of temperature factors on feature similarity; Temperature importance function, dynamically adjusting weights based on temperature values. Take 1.2 for temperatures ≥300℃. The value is 1.0. Take 0.8 when the temperature is <100℃; Temperature value of the area corresponding to non-visual features (unit: °C); : Attention weights for the u-th class of non-visual features, used to measure the importance of this class of non-visual features in multimodal fusion; The natural exponential function is used to map similarity values ​​to an exponential form of weight proportions. The total number of non-visual feature categories is fixed at 5 (temperature, humidity, wind speed, precipitation, and combustible material type). : The kth class of non-visual features (k ranges from 1 to K).

[0073] It should be noted that the temperature guiding coefficient ρ is preferably 0.3, but those skilled in the art can adjust it within the range of 0.2 to 0.5 according to factors such as the scale of the fire and the characteristics of the sensor, and still achieve a similar feature similarity adjustment effect.

[0074] (3) Feature fusion formula: Let the visual feature be... Non-visual fusion features are fusion features The calculation formula is as follows:

[0075]

[0076]

[0077] Multimodal fusion feature vectors integrate effective information from visual and non-visual features; Element-wise multiplication is an operation that multiplies features and weights element by element. Global attention weights for visual features; Global attention weights for non-visual fusion features, and satisfy ; Visual features The dimension specifications are as follows: 64 is the number of feature channels, H is the feature map height, and W is the feature map width. Non-visual fusion features The dimension specification, where 64 represents the number of feature channels. The size of the feature map.

[0078] Non-visual fusion features And the non-visual features mentioned above The difference is: It is a single type of non-visual primitive feature, referring to a single feature among temperature, humidity, wind speed, precipitation, and combustible material type. Indexes corresponding to 5 feature categories, Each type of feature maintains its own independent dimension (e.g., temperature is a time-series feature vector, and combustible type is an embedding vector). It is a fusion and aggregation of all non-visual features, and is applied to each class through an attention mechanism. Assign dynamic weights After further integration operations such as weighted summation, the five categories of scattered data are... Transform into a unified dimension The feature vectors eliminate the dimensional differences between different non-visual features.

[0079] S222: Deep Metric Learning and Multi-Task Loss Optimization

[0080] This invention proposes a deep metric learning loss that integrates attention weights with a multi-task gradient normalization weight adaptive strategy to achieve synergistic optimization of feature discriminativeness, classification accuracy, and temperature prediction precision. Furthermore, the total loss function for training the fire situation analysis model is a weighted sum of the classification task loss, temperature prediction task loss, and feature reconstruction task loss. During training, a gradient normalization strategy is used to dynamically adjust the adaptive weights corresponding to the losses of each task. The gradient normalization strategy includes: normalizing the real-time gradients based on the norm of the initial gradients of each task, and dynamically calculating and constraining the adaptive weights of each task based on the norm of the normalized gradients. The specific design is as follows:

[0081] S2221: Deep Metric Learning Integrated Loss definition

[0082] This invention designs three types of core loss functions with attention weights, which are then fused into a comprehensive loss for deep metric learning using fixed weights. This is used to optimize the sample distribution in the high-dimensional feature space, and the formula is as follows:

[0083]

[0084] Deep metric learning comprehensive loss integrates three types of core loss functions with attention weights to optimize the sample distribution in a high-dimensional feature space; : Fixed fusion weights for the distance constraint loss of sample triples, with a value of 0.01; : Fixed fusion weights for feature vector angle similarity loss, with a value of 0.0001; : Fixed fusion weights for class center distance constraint loss, with a value of 0.01; Attention weights with triples The sample triplet distance constraint loss is used to bring positive samples closer and move negative samples further away. : Attention weights with cosine The feature vector angle similarity loss is adapted for category discrimination in high-dimensional feature spaces; : Attention weights with class center The class center distance constraint loss is used to reduce intra-class variance and increase inter-class variance. The set of triplet attention weights, containing the triplet attention weights for all samples. ; : The set of cosine attention weights, containing the cosine attention weights for all samples. ; : The set of class center attention weights, containing the class center attention weights for all samples. ; The triplet attention weights for the j-th feature of the i-th sample are adaptively assigned based on the similarity of the multimodal fusion features. The cosine attention weight of the j-th feature of the i-th sample is adaptively assigned based on the similarity of the multimodal fusion features. : The class center attention weight of the j-th feature of the i-th sample is adaptively assigned by the multimodal fusion feature similarity.

[0085] , , To optimize the values ​​based on experimental data, those skilled in the art can adjust them within a reasonable range according to factors such as task difficulty and sample distribution (e.g., (Values ​​can be taken between 0.005 and 0.05), and the overall optimization effect of deep metric learning loss can still be achieved.

[0086] S2222: Improved Lovász-Softmax Loss

[0087] The classification task loss adopts an improved Lovász-Softmax loss function, which introduces a fire context weighting mechanism and a sample ambiguity coefficient to dynamically adjust the contribution of different samples to the total loss based on the combustible material type and environmental parameters. The fire context weight is calculated jointly based on the combustible material combustion coefficient, temperature, humidity and wind speed parameters, while the sample ambiguity coefficient is used to assign higher learning weights to samples at the fire stage boundary.

[0088] This invention introduces a fire scene context weighting mechanism, assigning differentiated weights to samples with varying degrees of ambiguity at different fire stages to improve the learning performance of classification boundary samples. The formula is as follows:

[0089]

[0090] An improved Lovász-Softmax loss method is introduced, which incorporates a fire scene context weighting mechanism to enhance the learning performance of fire stage classification boundary samples. : Summation operation of fire stage category indices from 1 to C; Fire stage category index, corresponding to four stages: initial growth, sudden burst, full development, and decay; The total number of fire stage categories is fixed at 4. The category weight of the c-th fire stage is obtained by normalizing the inverse of the proportion of samples of that class. : The summation operation on the training samples from 1 to n; Sample index; The total number of samples participating in the training; : The ambiguity coefficient for the i-th sample belonging to the c-th fire stage. The coefficient is 1.2 for stage boundary samples and 0.8 for samples in clearly defined stages, or can be adjusted within the ranges of 1.1~1.5 and 0.6~0.9 respectively, depending on the difficulty of the classification task; Context The fire context weight of the i-th sample belonging to the c-th fire stage is calculated jointly by the combustible material type and environmental parameters. The natural logarithm function is used to calculate the log loss of the predicted probability. : The predicted probability that the i-th sample belongs to the c-th fire stage; The fire context weight is calculated jointly by the type of combustible material and environmental parameters. The specific formula is as follows:

[0091]

[0092]

[0093] The normalization factor for the fire context weights is obtained by summing four types of fixed weights; : Fixed weight for combustible material type characteristics, with a value of 0.4; : Fixed weight for temperature characteristics, with a value of 0.3; The fixed weight of the humidity feature is 0.2. : Fixed weight for wind speed characteristics, with a value of 0.1; The combustible coefficient of the i-th sample is 1.0 for trees, 0.8 for shrubs, 0.6 for building materials, and 0.4 for other types. : Combustible material type information for the i-th sample; Temperature value (in °C) of the region corresponding to the i-th sample; The maximum temperature that the infrared thermal imaging sensor can measure is fixed at 1500℃. : The relative humidity of the area corresponding to the i-th sample (unit: %); : The ambient wind speed (unit: m / s) in the area corresponding to the i-th sample. The maximum wind speed is set to 20 m / s.

[0094] S2223: Definition of Multi-Task Loss Components

[0095] The model training task is divided into three tasks: fire stage classification, temperature prediction, and feature reconstruction. The loss components for each task are defined as follows:

[0096]

[0097]

[0098] The total loss for the classification task integrates the binary cross-entropy loss, the deep metric learning comprehensive loss, and the improved Lovász-Softmax loss to optimize the classification performance during the fire phase. Binary cross-entropy loss is used to measure the difference between the predicted result and the true label in a fire stage classification task. : Fixed hyperparameter weights for the deep metric learning comprehensive loss, with a value of 0.01; Deep metric learning comprehensive loss is used to optimize the sample distribution in a high-dimensional feature space. The fixed hyperparameter weights of the improved Lovász-Softmax loss are set to 0.001. Improved Lovász-Softmax loss enhances the learning performance of fire stage classification boundary samples; Temperature prediction task loss, including only mean absolute error loss, is used to optimize temperature prediction accuracy. Mean absolute error loss (MARS) measures the absolute deviation between the predicted and actual temperature values.

[0099] S2224: Gradient Normalization Weight Adaptive Strategy

[0100] This invention employs a gradient normalization strategy to dynamically adjust the adaptive weights of each task, avoiding the dominance of a single task's gradient in the training process, and simultaneously addressing the weight imbalance problem caused by excessively large initial gradient differences during training. The specific formula is as follows:

[0101]

[0102]

[0103] These are the adaptive weights for classification, prediction, and reconstruction tasks, respectively. It is generated synchronously and dynamically from the above formula; The normalized gradient of the k-th task is obtained by dividing the real-time gradient by the sum of the initial gradient L2 norm and the coefficients that prevent the removal of zero. The real-time gradient of the k-th task, dynamically calculated and updated during training; The initial gradient of the k-th task is the gradient value calculated in the first round of training. : The L2 norm of the initial gradient of the k-th task; : Zero-prevention coefficient, with a fixed value To avoid calculation errors where the denominator is zero; The original adaptive weights for the k-th task are obtained by taking the reciprocal of the L2 norm of the normalized gradient. : The L2 norm of the normalized gradient of the k-th task; The final adaptive weights of the k-th task, after numerical truncation, are used for total loss calculation; : Numerical truncation function, which truncates the original adaptive weights Limited to Within the specified range, ensure training stability; Task index, with a value range of: These correspond to classification tasks, temperature prediction tasks, and feature reconstruction tasks, respectively.

[0104] S2225: Total loss function of the model

[0105] Based on gradient-normalized adaptive weights, the total loss function of the model is obtained by integrating the losses from all tasks, as shown in the following formula:

[0106]

[0107] The dynamic adaptive weights, after gradient normalization and numerical truncation, are updated in real time by the formula in step S2224. The dynamic adaptive weights for the classification task are obtained through gradient normalization and numerical truncation and are updated in real time. The dynamic adaptive weights for the temperature prediction task are obtained through gradient normalization and numerical truncation and are updated in real time. Dynamic adaptive weights for feature reconstruction tasks, and , Synchronously and dynamically generated, obtained through gradient normalization and numerical truncation; The overall loss function of the model integrates the losses from the classification task, temperature prediction task, and feature reconstruction task, and is used to optimize the overall parameters of the model. The autoencoder feature reconstruction loss is used to optimize the effectiveness of multimodal feature extraction and ensure that the features retain the core information of the input data. Multiplication operation is used to perform weighted calculation of adaptive weights and corresponding task losses.

[0108] S223: Dynamic Temperature Prediction and Stage Matching

[0109] Specifically, the temperature time-series prediction branch of the fire situation analysis model outputs predicted future temperatures. This branch employs a hybrid model combining a Long Short-Term Memory (LSTM) network and a cellular automaton. The cellular automaton is pre-trained using fire spread data generated by fire dynamics simulation software and fine-tuned using real fire spread data. The fire stage classification branch includes: a fully connected layer whose input is the multimodal fusion features output from the dynamic fusion layer; and a classifier, which generates fire stage classification results based on the output of the fully connected layer, including at least four stages: initial growth, rapid spread, full development, and decay. The specific process is as follows:

[0110] (1) Temperature prediction model: This invention proposes a hybrid model of "LSTM + cellular automata". The input is a historical temperature sequence (nearly 10 time steps), wind speed direction, and combustible type. The cellular automata are pre-trained using synthetic spread data generated by the fire dynamics simulation software (Fire DynamicsSimulator, FDS). The model is then fine-tuned on a small amount of real fire spread data (simulation error ≤15%), and the predicted temperature values ​​for the next 5 time steps are output. The MAE loss target is ≤3℃. The formula is as follows:

[0111]

[0112] : The predicted temperature value of the grid at time step t+1 (unit: °C), which is the output of the hybrid model; A hybrid prediction model combining LSTM and cellular automata is used to predict temperature by integrating temporal features and fire spread dynamics. The grid represents the historical temperature values ​​(in °C) at time step t, which is a temperature sequence over the past 10 time steps. : Ambient wind speed (unit: m / s) in the grid area; D: Wind direction in the grid area; S: Type of combustible material in the grid area.

[0113] (2) Fire stage matching: This invention establishes a multi-type fire temperature-time curve library, including three types of standard curves: building indoor fires, forest fires, and industrial fires. By calculating the similarity between the predicted temperature sequence and the curve library (cosine similarity ≥ 0.85), and combining the combustible material type, the current fire stage is determined; the stage transition time prediction is based on the temperature change rate. Curve fitting, warning conditions for the sudden fire stage: building fire And T≥600℃, forest fire And T≥450℃.

[0114] Temperature change rate (unit: °C / min), used for predicting fire phase transition time and early warning of sudden fire phase; Temperature change between adjacent time steps (unit: °C). : The time interval between adjacent time steps (unit: min).

[0115] In some alternative embodiments, the fully connected layer of the fire stage classification branch is one or more layers, and the classifier is a Softmax classifier to enhance support.

[0116] Specifically, the method for constructing the temperature-time curve library is as follows: For three types of fire scenarios—building, forest, and industrial—real fire experimental data and standard fire source data generated by fire dynamics simulation software are collected respectively. Temperature-time series for four stages—initial growth, burst, full development, and decay—are extracted. After smoothing and fitting, representative curves for each stage are generated, such as… Figure 3 The diagram shows a fire temperature-time curve and stage matching schematic diagram according to an embodiment of the present invention. During matching, the temperature sequence of the next 5 time steps output by the temperature prediction branch is slidably aligned with the stage curves in the curve library, and the cosine similarity is calculated. The stage corresponding to the maximum value is taken as the current fire stage. If the maximum similarity is lower than the preset threshold of 0.85, the temperature change rate and the type of combustible material are further combined for comprehensive judgment.

[0117] It should be noted that the similarity threshold of 0.85 in this embodiment of the invention is a fixed value selected based on experimental data. Those skilled in the art can adjust it within the range of 0.75 to 0.95 according to factors such as the actual fire type, sensor accuracy, and prediction time step, and accurate fire stage matching can still be achieved.

[0118] (3) Fire anomaly identification:

[0119] Furthermore, the method of the present invention also includes a fire anomaly identification step: adaptively calculating a dynamic temperature gradient threshold based on the type of combustible material and the ambient humidity; comparing the actual temperature gradient of the grid cell in the fire scene with the dynamic temperature gradient threshold; if the actual temperature gradient continuously exceeds the threshold, it is determined that there is a hidden fire source; if the temperature continuously exceeds a preset value but the temperature gradient is below the lower limit, it is determined that there is a potential area for reignition.

[0120] Specifically, the dynamic temperature gradient threshold is adaptively adjusted based on the type of combustible material and ambient humidity, as shown in the following formula:

[0121]

[0122] When the actual temperature gradient And it lasted for 3 time steps, which was determined to be a hidden fire source; when But the temperature remains higher than The area was identified as a potential area for reignition.

[0123] Dynamic temperature gradient threshold (unit: ℃ / m), adaptively adjusted based on combustible material type and ambient humidity, used for fire anomaly identification; The adjustment coefficient for the temperature gradient threshold is preferably set to a fixed value of 0.02 in this invention. : The highest temperature value (unit: °C) in the area corresponding to the grid. The spacing of the metric grid (unit: meters) is preferably a fixed value of 5m in this invention; The humidity adjustment coefficient for the temperature gradient threshold is preferably set to a fixed value of 2.0 in this invention. : Relative humidity of the area corresponding to the grid (unit: %); : Actual temperature gradient of the grid region (unit: °C / m); Humidity correction factor: The higher the humidity, the greater the correction to the temperature gradient threshold.

[0124] The fixed parameter values ​​given in the embodiments of the present invention are optimized values ​​based on experimental data. Those skilled in the art can adjust them within a reasonable range according to factors such as the actual fire type and sensor accuracy. For example, α can be taken between 0.01 and 0.05, and β can also be varied within a reasonable range without specific limitations, and similar feature similarity adjustment effects can still be achieved.

[0125] S23: Model Training and Performance Validation

[0126] S231: Training Data and Environment Configuration

[0127] (1) Data partitioning: The training set consists of 8558 sets of data (accounting for 80% of the augmented dataset), including three types of fire scenarios: building, forest, and industry. The fire spread data is generated by FDS simulation (accounting for 70%) + real data (accounting for 30%). The validation set consists of 1070 sets (10%), and the test set consists of 1070 sets (10%).

[0128] (2) Hardware environment: NVIDIA 3080 GPU, PyTorch framework, batch size 64, training rounds 20.

[0129] (3) Initialization parameters: The encoder and decoder are initialized using He, and the class center parameters are randomly initialized (range). The cellular automata parameters are pre-calibrated based on FDS simulation data, and the initial gradients for each task are... Statistics were completed and fixed in the first round of training.

[0130] S232: Training Steps Execution

[0131] (1) Pre-trained autoencoder, only optimizing reconstruction loss Learn multimodal feature representations with an initial learning rate of 0.01.

[0132] (2) The pre-trained cellular automata fire spread model was optimized based on 100,000 sets of composite spread data generated by FDS to optimize the core parameters (burning rate and spread direction), and then fine-tuned with 3,000 sets of real fire spread data to ensure simulation error. .

[0133] (3) Jointly train the DML module, the temperature time series prediction branch and the classification branch, and dynamically update the task weights using the gradient normalization strategy proposed in this invention. , , And all weights follow in real time Numerical constraints apply, with the learning rate decaying to 0.001 and 0.0001 in rounds 10 and 18, respectively.

[0134] (4) Test set validation, requiring temperature prediction Stage classification accuracy False Negative Rate (FNR) for Fire Detection Accuracy of adaptation to different fire types .

[0135] S233: Input / Output Specification Definition

[0136] (1) Input: RGB image (256×256×3), temperature calibration thermal imaging image (256×256×1), environmental parameters (temperature / ℃, humidity / %RH, wind speed / m・s) -1 Rainfall (mm), combustible material type code, and grid code.

[0137] (2) Output: grid temperature prediction (5 future time steps), fire type, fire stage, remaining time for stage transition, risk level, anomaly identification results, coloring instructions, and decision recommendations.

[0138] S3: Based on the situation analysis results, generate fire situation visualization information and structured decision instructions.

[0139] Step S3 is used to achieve situational visualization and intelligent decision generation. Generating fire situational visualization information includes: dynamically coloring the gridded areas of the fire scene using different colors based on fire stage classification results and risk levels; generating structured decision instructions includes: integrating grid coding, predicted temperature, fire stage, and combustible material type information to generate command instruction text containing specific locations and operational suggestions. For example... Figure 4 The diagram shown is a schematic representation of a forest fire situation in an embodiment of the present invention. Step S3 specifically includes the following steps:

[0140] S31: Dynamic Mesh Coloring and Situation Map Generation

[0141] This invention uses real-time coloring of the grid based on fire type, prediction stage, and risk level:

[0142] (1) Green grid: Initial growth stage (building fire T<300℃, forest fire T<250℃, <2℃ / min), prompting "rapid extinguishing tactic";

[0143] (2) Red grid: Fully developed stage (building fires 300-800℃, forest fires 250-600℃, (℃ / min), prompting "break through the wall to extinguish the fire + cover with foam (building)" or "establish a firebreak + sprinkle water to cool down (forest)";

[0144] (3) Yellow grid: Decay stage (temperature continues to drop, The message reads, "Focus on monitoring for reignition and extinguish any remaining embers."

[0145] (4) Purple-black flashing grid: predicts that the fire will enter the explosive stage within 3 minutes (meeting the temperature and warming rate thresholds of the corresponding fire type), triggers a high-level evacuation warning, and pushes evacuation route suggestions at the same time.

[0146] S32: Generation of Decision Support Suggestions

[0147] Based on grid coding, temperature prediction results, stage determination, and combustible material type, structured command instructions are automatically generated, as shown in the following example:

[0148] (1) "Grid X012Y013 (building fire scenario) is in the full development stage. The current predicted temperature is 620℃. The combustible material is concrete structure. It is recommended to deploy a high-expansion foam generator to cover this grid and the adjacent grids X011Y013 and X012Y014 first."

[0149] (2) Temperature gradient anomaly in grid X056Y034 (forest fire scenario) (G=6.2℃ / m, threshold) =4.8℃ / m), the combustible material is coniferous forest, there is a risk of hidden fire sources, it is recommended to send more personnel to conduct a thorough search.

[0150] (3) "Grid X089Y078 (industrial fire scenario) is expected to enter the explosive phase in 2 minutes and 30 seconds. Immediately evacuate all combat personnel in this area. The evacuation route is along X089Y078 → X089Y080 ​​→ safe assembly point.

[0151] Figure 5This is a schematic diagram of a fire situation decision-making system based on AI dynamic temperature prediction and multimodal fusion provided in an embodiment of the present invention, as shown below. Figure 5 As shown, a fire situation decision-making system based on AI dynamic temperature prediction and multimodal fusion includes:

[0152] The multimodal data acquisition module 210 is used to acquire multimodal data of the fire scene. The multimodal data includes visual image data and non-visual parameter data. The visual image data includes RGB images and thermal imaging images with absolute temperature calibration. The non-visual parameter data includes environmental parameter time series data and combustible material type data.

[0153] The spatial grid division module 220 is used to divide the fire area into grids, assign a unique code to each grid cell, and associate the multimodal data with the corresponding grid cell code.

[0154] The fire situation analysis module 230 is used to input the multimodal data into a pre-trained fire situation analysis model to obtain the situation analysis results output by the fire situation analysis model. The fire situation analysis model uses an attention mechanism to dynamically fuse visual features extracted from the visual image data and non-visual features extracted from the non-visual parameter data, and makes predictions based on the fused features to output the situation analysis results. The situation analysis results include at least future temperature predictions, fire stage classification results, and fire risk levels.

[0155] The decision output module 240 generates fire situation visualization information and structured decision instructions based on the situation analysis results and the grid cell encoding.

[0156] In some specific embodiments of the present invention, the system adopts an edge-cloud layered processing architecture: the edge computing nodes are equipped with a lightweight preliminary fire identification model for real-time fire identification and coarse temperature measurement; the cloud server is equipped with the fire situation analysis model for receiving key data from the edge computing nodes, performing accurate temperature prediction, stage classification and situation analysis, and asynchronously updating decision instructions.

[0157] System Deployment and Integration:

[0158] 1. Hardware Requirements and Compatibility

[0159] (1) Sensor requirements: Relying on an infrared thermal imaging sensor with temperature calibration function, the temperature measurement range is -20℃~1500℃, and the accuracy requirements are: ±2℃ when ≤800℃, ±5% when >800℃ (acceptable accuracy relaxation in high temperature range), compatible with fire-fighting drones, fixed monitoring stations and other mounting scenarios.

[0160] (2) Protocol support: Compatible with Real Time Streaming Protocol (RTSP) and Open Network Video Interface Forum (ONVIF) standard protocols, and can be seamlessly integrated into existing video management systems.

[0161] 2. Edge-Cloud Layered Processing Design

[0162] This invention proposes an edge-cloud layered processing architecture diagram, such as... Figure 6 As shown:

[0163] (1) Local edge computing: Deploy a lightweight detection model (INT8 quantization) to achieve preliminary fire identification and coarse temperature measurement, with local inference latency ≤100ms.

[0164] (2) Cloud-based fine computing: Key data is uploaded through H.265 video stream compression (bitrate ≤ 1Mbps), and the complete model is run in the cloud to perform accurate temperature prediction and stage classification, and decision suggestions are updated asynchronously (update cycle ≤ 1s).

[0165] (3) Latency index: In a low bandwidth environment (≥2Mbps), the total end-to-end latency (including encoding, transmission and inference) is ≤500ms.

[0166] 3. Alarm integration function

[0167] It supports integration with Security Information and Event Management (SIEM) systems, outputting standardized alarm information (including grid codes, risk levels, decision recommendations, and location coordinates) to achieve automated event response.

[0168] like Figure 7The figure shows the performance comparison results between the algorithm of this invention and existing algorithms. This simulation uses the number of iterations as the training progress indicator and the mean absolute error (MAE) as the evaluation standard for fire temperature prediction accuracy. The performance of the algorithm of this invention is compared with that of four mainstream baseline algorithms: VGG19 (Visual Geometric Group Network 19), ResNet18 (Residual Network 18), Faster-RCNN (Fast Region Convolutional Neural Network), and YOLOv8. The results show that the MAE of fire temperature prediction for all five algorithms shows a continuous decreasing trend with the increase of the number of iterations, which conforms to the convergence law of machine learning model training and shows no obvious overfitting phenomenon. Among them, the algorithm of this invention maintains the lowest MAE value throughout the entire iteration cycle, reaching 2.8℃ after 10 iterations and decreasing to 1.8℃ after 50 iterations. It is the only algorithm with an MAE below 2℃ after 50 iterations. Meanwhile, the algorithm of this invention also exhibits a faster convergence rate and stronger error optimization capability. The MAE decreases by 0.3℃ after 10 to 20 iterations, which is higher than the four baseline algorithms. This advantage stems from the fact that the algorithm of this invention is designed for the needs of fire temperature prediction. It integrates a multi-task loss optimization strategy that combines multimodal feature extraction, dynamic attention weighting, and gradient normalization. It is specifically adapted to the temperature numerical regression task in fire scenarios and can more efficiently mine temperature-related features in multimodal data. In contrast, VGG19 and ResNet18 are not designed for regression tasks, and Faster-RCNN and YOLOv8 focus on target classification and localization, which are not suitable for continuous temperature numerical prediction. In summary, the algorithm of this invention is superior to the four baseline algorithms in terms of prediction accuracy, convergence rate, and scenario adaptability, providing a more efficient algorithm solution for accurate temperature prediction in fire situation awareness.

[0169] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0170] It should also be noted that, in the embodiments of this application, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0171] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined in the embodiments of this application may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown in this application, but is to be accorded the widest scope consistent with the principles and novel features disclosed in the embodiments of this application.

Claims

1. A fire scene situation decision-making method based on AI dynamic temperature prediction and multi-modal fusion, characterized in that, include: Multimodal data of the fire scene is collected, including visual image data and non-visual parameter data; wherein, the visual image data includes RGB images and thermal imaging images with absolute temperature calibration; the non-visual parameter data includes environmental parameter time series data and combustible material type data; the fire area is divided into grids, each grid cell is assigned a unique code, and the multimodal data is associated with the corresponding grid cell code; The multimodal data is input into a pre-trained fire situation analysis model to obtain the situation analysis results output by the fire situation analysis model. The fire situation analysis model uses an attention mechanism to dynamically fuse visual features extracted from the visual image data and non-visual features extracted from the non-visual parameter data, and makes predictions based on the fused features to output the situation analysis results. The situation analysis results include at least future temperature predictions, fire stage classification results, and fire risk levels. Furthermore, the situation analysis results are organized and output in units of the grid cells. The fire situation analysis model uses an attention mechanism to dynamically fuse visual features extracted from the visual image data and non-visual features extracted from the non-visual parameter data. This includes calculating the fusion similarity between each type of non-visual feature and visual feature based on the cosine similarity between the non-visual and visual features, combined with a temperature importance function related to the regional temperature value. The specific formula is as follows: The similarity between non-visual and visual features is used to dynamically assign weights to non-visual features. Non-visual features; Visual characteristics; Temperature guiding coefficient, used to adjust the weight of the influence of temperature factors on feature similarity; Temperature importance function, dynamically adjusts weights based on temperature values; Temperature values ​​of areas corresponding to non-visual features; Based on the situation analysis results, visualized fire situation information and structured decision instructions are generated.

2. The method according to claim 1, characterized in that, The method further includes: Constructing a training dataset for training the fire situation analysis model includes data augmentation processing of RGB image and thermal imaging image pairs in the training dataset, including: The RGB image is enhanced with luminance, and the thermal image is enhanced with nonlinear enhancement based on relative temperature distribution. The nonlinear enhancement is as follows: the flame core region and the background region in the thermal image are identified, the pixel temperature value of the flame core region is kept unchanged, and random noise or local contrast calibration is added to the pixel temperature value of the background region.

3. The method according to claim 1, characterized in that, The fire situation analysis model is an end-to-end deep learning model, which includes: The input layer is used to receive the multimodal data; A feature extraction layer is used to extract visual features from the RGB images and thermal imaging images, extract temporal features from the environmental parameter time-series data, and encode the combustible material type into a feature vector; wherein, the visual features are extracted from the RGB images and thermal imaging images by a U-shaped network of a shared encoder; the temporal features are extracted from the environmental parameter time-series data by a one-dimensional convolutional and long short-term memory network; The dynamic fusion layer employs an attention mechanism to dynamically weight and fuse the visual features, temporal features, and combustible material feature vectors to generate a unified multimodal fusion feature. The prediction layer includes a temperature time-series prediction branch and a fire stage classification branch. The temperature time-series prediction branch is used to output the temperature prediction value at least one time step in the future, and the fire stage classification branch is used to output the classification result of the fire stage. The output layer is used to output situation analysis results that include at least the predicted temperature value, fire stage classification results, and fire risk level.

4. The method according to claim 3, characterized in that, The dynamic fusion layer assigns dynamic weights to various non-visual features through an attention mechanism, including the following process: Based on the fusion similarity, attention weights are assigned to each class of non-visual features using the Softmax function; The non-visual features are weighted according to the attention weights, and the weighted non-visual features are fused with the visual features to obtain the multimodal fusion features.

5. The method according to claim 3, characterized in that, The total loss function for training the fire situation analysis model is a weighted sum of the classification task loss, temperature prediction task loss, and feature reconstruction task loss. During training, a gradient normalization strategy is used to dynamically adjust the adaptive weights corresponding to the loss of each task. The gradient normalization strategy includes: normalizing the real-time gradient of each task according to the norm of the initial gradient, and dynamically calculating and constraining the adaptive weights of each task based on the norm of the normalized gradient.

6. The method according to claim 3, characterized in that, The predicted future temperature is output by the temperature time-series prediction branch of the fire situation analysis model. The temperature time series prediction branch adopts a hybrid model combining a long short-term memory network (LSTM) and a cellular automaton. The cellular automaton is pre-trained using fire spread data generated by fire dynamics simulation software and fine-tuned using real fire spread data. The fire stage classification branch specifically includes: a fully connected layer, whose input is the multimodal fusion feature output by the dynamic fusion layer; A classifier is used to generate fire stage classification results based on the output of the fully connected layer, including at least four stages: initial growth, rapid spread, full development, and decay.

7. The method according to claim 5, characterized in that, The classification task loss adopts an improved Lovász-Softmax loss function, which introduces a fire context weighting mechanism and a sample ambiguity coefficient to dynamically adjust the contribution of different samples to the total loss based on the combustible material type and environmental parameters. The fire context weight is calculated jointly based on the combustible material combustion coefficient, temperature, humidity and wind speed parameters, and the sample ambiguity coefficient is used to assign higher learning weights to boundary samples of the fire stage.

8. The method according to claim 1, characterized in that, in: The generated fire situation visualization information includes: dynamically coloring the grid cells with different colors based on the fire stage classification results and risk levels; The generation of structured decision instructions includes: integrating the grid cell encoding, predicted temperature, fire stage, and combustible material type information to generate command instruction text containing specific locations and operational suggestions.

9. The method according to claim 8, characterized in that, The method also includes a fire anomaly identification step: Based on the type of combustible material and ambient humidity, the dynamic temperature gradient threshold is adaptively calculated. The actual temperature gradient of the grid cell in the fire scene is compared with the dynamic temperature gradient threshold. If the actual temperature gradient continues to exceed the threshold, it is determined that there is a hidden fire source. If the temperature continues to be higher than the preset value but the temperature gradient is lower than the lower limit, it is determined that there is a potential area for reignition.

10. A fire situation decision-making system based on AI dynamic temperature prediction and multimodal fusion, characterized in that, include: A multimodal data acquisition module is used to acquire multimodal data from a fire scene. The multimodal data includes visual image data and non-visual parameter data. The visual image data includes RGB images and thermal imaging images with absolute temperature calibration. The non-visual parameter data includes environmental parameter time series data and combustible material type data. The spatial grid division module is used to divide the fire area into grids, assign a unique code to each grid cell, and associate the multimodal data with the corresponding grid cell code. The fire situation analysis module is used to input the multimodal data into a pre-trained fire situation analysis model to obtain the situation analysis results output by the fire situation analysis model. The fire situation analysis model uses an attention mechanism to dynamically fuse visual features extracted from the visual image data and non-visual features extracted from the non-visual parameter data, and makes predictions based on the fused features to output the situation analysis results. The situation analysis results include at least future temperature predictions, fire stage classification results, and fire risk levels. The fire situation analysis model uses an attention mechanism to dynamically fuse visual features extracted from the visual image data and non-visual features extracted from the non-visual parameter data. This includes calculating the fusion similarity between each type of non-visual feature and visual feature based on the cosine similarity between the non-visual and visual features, combined with a temperature importance function related to the regional temperature value. The specific formula is as follows: The similarity between non-visual and visual features is used to dynamically assign weights to non-visual features. Non-visual features; Visual characteristics; Temperature guiding coefficient, used to adjust the weight of the influence of temperature factors on feature similarity; Temperature importance function, dynamically adjusts weights based on temperature values; Temperature values ​​of areas corresponding to non-visual features; The decision output module generates fire situation visualization information and structured decision instructions based on the situation analysis results and the grid cell encoding.

Images