Fire hazard identification method and system based on multi-modal large model

Abstract

The application relates to the technical field of intelligent safety monitoring, and discloses a fire hazard identification method and system based on a multi-modal large model. The method standardizes and preprocesses field multi-modal data through a unified timestamp system and a quality evaluation mechanism, extracts visual, text and sensor features by using a pre-trained deep learning model, aligns the features, realizes adaptive feature fusion by using a scene perception attention mechanism, completes hazard identification and confidence evaluation by using a hierarchical classification network and an uncertainty quantification method, generates a structured rectification suggestion based on template matching and multi-target optimization, and supports progressive model updating of experience playback and collaborative management of RESTful API. The application can effectively improve the accuracy and efficiency of fire hazard identification and provide intelligent decision support for safety management.

Description

Technical Field

[0001] This invention relates to the field of intelligent security monitoring technology, and more specifically, to a method and system for identifying fire hazards based on a multimodal large model. Background Technology

[0002] With rapid urbanization and increasing building complexity, fire hazard identification has become a significant challenge for public safety management. Traditional fire hazard inspections rely primarily on manual patrols, which are inefficient, subjective, and prone to missed detections. Especially during special safety campaigns, comprehensive inspections of numerous areas and businesses are required, and traditional manual methods are ill-suited to the refined and real-time requirements of modern safety management.

[0003] Existing fire detection technologies are primarily based on single-modal sensors or visual recognition systems, which face numerous limitations in practical applications. While sensor-based detection systems offer good stability, they lack the ability to identify fire hazards in their early stages, often only issuing alarms after a fire has occurred, failing to provide preventative fire hazard identification. Visual recognition-based systems are susceptible to environmental factors such as changes in lighting, obstructions, and camera angles, resulting in high false alarm and false negative rates. Furthermore, most existing technologies remain at the detection and alarm stage, lacking in-depth intelligent analysis capabilities and unable to provide targeted rectification suggestions and solutions.

[0004] Current technologies suffer from poor scenario adaptability. Most systems are designed for specific scenarios and lack cross-scenario adaptability, failing to differentiate between different building types and usage areas. Furthermore, existing systems are mostly independent detection or monitoring systems, lacking a unified integration platform. Inconsistent data formats between different systems hinder effective information sharing and collaborative management. Therefore, there is an urgent need for a fire hazard identification technology solution that can integrate multimodal information, provide intelligent decision support, and adapt to various scenarios. Summary of the Invention

[0005] This invention provides a fire hazard identification method and system based on a multimodal large model, which solves the technical problems of poor adaptability to early warning scenarios and insufficient identification ability in related technologies.

[0006] This invention provides a fire hazard identification method based on a multimodal large model, comprising the following steps: The raw data stream of fire hazards is acquired, and the raw data stream is processed to obtain a standardized set of fire hazard monitoring data. A standardized fire hazard monitoring dataset is processed using feature engineering to obtain a multimodal feature representation; The multimodal features are weighted and fused to obtain the fused multimodal features and scene context information; Based on the fused multimodal features, a fire hazard classification network is constructed; a fire hazard identification model and hazard identification results are obtained. Based on the hazard identification results and scene context information, a fire prevention plan is obtained by using template matching and multi-objective optimization methods. Based on the fire prevention plan, the existing fire hazard identification model is updated and optimized using experience playback and parameter regularization methods to obtain the updated fire hazard identification model. Based on the updated fire hazard identification model and hazard identification results, a multi-dimensional statistical analysis report and collaborative management instructions are obtained using RESTful API and real-time statistical methods.

[0007] In a preferred embodiment, the step of processing the raw data stream to obtain a standardized fire hazard monitoring dataset includes: A unified timestamp is obtained based on GPS clock synchronization technology and network time protocols; A sharpness assessment algorithm based on the Laplacian operator is used to obtain video frame data for fire hazard monitoring through image gradient calculation. Based on the statistical method of the 3σ criterion, fire hazard environmental sensor data are obtained through sliding window anomaly detection. Based on the BERT pre-trained model, word segmentation and entity recognition technologies are used, and a fire safety terminology dictionary matching method is adopted to obtain standardized fire hazard inspection text data. A data quality score is obtained by weighting the scores based on three dimensions: completeness, accuracy, and timeliness.

[0008] In a preferred embodiment, the step of using feature engineering to process the standardized fire hazard monitoring data set to obtain multimodal feature representations includes: A deep learning network architecture is adopted, and a transfer learning strategy is used to obtain visual feature vectors of fire hazards; A pre-trained language model is used to obtain the semantic vector of fire hazard text through a context-aware semantic encoding method; A convolutional network is used to extract feature vectors from fire hazard environmental sensors using a sliding window feature extraction method. By employing a multilayer perceptron network and using a nonlinear mapping method, a unified-dimensional multimodal feature representation is obtained.

[0009] In a preferred embodiment, the step of performing weighted fusion processing on the multimodal features to obtain fused multimodal features and scene context information includes: A lightweight classification network is used to obtain scene context information; Attention weights are calculated using a single-layer attention network based on scene context information to obtain dynamic weights for each modality. Weighted feature fusion is performed by using a weighted average method based on the calculated attention weights to obtain the fused multimodal features.

[0010] In a preferred embodiment, the steps of constructing and training a two-layer fire hazard classification network based on the fused multimodal features, and obtaining the fire hazard identification model and hazard identification results by combining uncertainty quantification methods, include: A multi-class neural network is used to obtain the probability distribution of fire hazards; a classifier is then used to obtain the specific types of fire hazards. Based on the stochastic forward propagation method, the mean and variance of the prediction results are calculated to obtain the uncertainty estimate; based on the prediction variance, a normalization method is used to obtain the confidence score.

[0011] In a preferred embodiment, the step of obtaining a fire prevention plan based on hazard identification results and scene context information, using template matching and multi-objective optimization methods, includes: The rectification template matching is based on a pre-built rectification suggestion template library and uses a multi-dimensional matching algorithm to obtain a set of candidate rectification solutions. Cost-benefit assessment, based on a predefined cost model and benefit assessment function, employs a multi-objective optimization algorithm to obtain a comprehensive score for the rectification plan; The time prediction model, based on historical rectification data, uses regression analysis to obtain a predicted estimate of the rectification time. Multiple schemes are generated and sorted. Based on the scoring results, a selection algorithm is used to obtain the sorted fire prevention schemes.

[0012] In a preferred embodiment, the step of updating and optimizing the constructed fire hazard identification model using experience replay and parameter regularization methods to obtain the updated fire hazard identification model includes: Feedback data collection and preprocessing: Based on the fire hazard identification results and rectification effect evaluation confirmed by users, data cleaning and labeling methods are used to obtain processed feedback data and use it as training samples. The experience replay mechanism, based on the historical sample buffer, uses an importance sampling method to obtain a mixed training dataset; The parameter regularization update is based on the elastic weight consolidation method and adopts a parameter importance weighted regularization strategy to obtain the updated fire hazard identification model. A / B testing was used to validate the new model by employing a controlled experiment method based on a small-scale test set.

[0013] In a preferred embodiment, the step of obtaining a multi-dimensional statistical analysis report and collaborative management instructions based on the updated fire hazard identification model and hazard identification results, using RESTful API and real-time statistical methods, includes: A standardized data exchange interface, based on a RESTful architecture, adopts a structured data transmission protocol in JSON format, resulting in a unified data exchange standard. Role-based access control employs a hierarchical access control strategy, resulting in a differentiated access control mechanism. Real-time statistical analysis, based on a sliding time window, uses an incremental calculation method to obtain real-time statistical indicators; Regional ranking and collaborative instruction generation are based on multi-dimensional evaluation indicators and a comprehensive scoring algorithm to obtain regional ranking and collaborative management suggestions.

[0014] In a preferred embodiment, the fire hazard identification method based on a multimodal large model further includes: Ensemble learning for fire hazard identification is based on multiple classifiers with different architectures, and Bayesian model averaging is used to obtain ensemble prediction results. The consistency assessment of predictions among models is based on the standard deviation and mean of the predicted probabilities of all models in the integrated prediction results, resulting in an integrated confidence score.

[0015] In a preferred embodiment, the fire hazard identification system based on a multimodal large model is used to execute the above-described fire hazard identification method based on a multimodal large model, including: The data acquisition module is used to acquire the raw data stream of fire hazards, process the raw data stream, and obtain a standardized set of fire hazard monitoring data. The feature extraction module is used to process the standardized fire hazard monitoring data set using feature engineering to obtain multimodal feature representations; The feature fusion module is used to perform weighted fusion processing on multimodal features to obtain fused multimodal features and scene context information; The fire hazard identification module constructs a fire hazard classification network based on fused multimodal features, thereby obtaining a fire hazard identification model and hazard identification results. It is recommended that the generation module, based on the hazard identification results and scenario context information, adopt template matching and multi-objective optimization methods to obtain fire prevention solutions; The model update module, based on the fire prevention plan, uses experience playback and parameter regularization methods to update and optimize the existing fire hazard identification model, resulting in an updated fire hazard identification model. The collaborative management module, based on the updated fire hazard identification model and hazard identification results, uses RESTful API and real-time statistical methods to obtain multi-dimensional statistical analysis reports and collaborative management instructions.

[0016] The beneficial effects of this invention are as follows: The accuracy and robustness of hazard identification are improved through multimodal deep fusion technology. This invention integrates and processes information from multiple modalities, including visual, text, and sensor data, and dynamically adjusts the weights of different modalities using a scene-aware adaptive attention mechanism, fully leveraging the complementary advantages of multi-source information. Through uncertainty quantification, the system can provide a reliable confidence assessment for each identification result, effectively identify low-confidence predictions and suggest manual review, further ensuring the reliability of the identification results.

[0017] This system establishes a complete intelligent decision-making chain from hazard identification to rectification recommendations, improving the efficiency and quality of safety management. It not only accurately identifies fire hazards but also automatically generates personalized rectification recommendations based on rule engines and case-based reasoning technology, including specific measures, timelines, and cost estimates. The system uses a multi-objective optimization algorithm to comprehensively consider effectiveness, feasibility, and cost-effectiveness, providing users with multiple ranked rectification options. Attached Figure Description

[0018] Figure 1 This is a flowchart of the fire hazard identification method based on a multimodal large model according to the present invention; Figure 2 This is a block diagram of the fire hazard identification system based on a multimodal large model according to the present invention. Detailed Implementation

[0019] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0020] At least one embodiment of the present invention discloses a fire hazard identification method based on a multimodal large model, such as... Figure 1 As shown, it includes the following steps: S100: Acquire the raw data stream of fire hazards, process the raw data stream, and obtain a standardized set of fire hazard monitoring data. The system acquires raw data streams of fire hazards, including video surveillance data, infrared thermal imaging data, environmental sensor data, and inspection text records. The environmental sensor data includes temperature, humidity, and smoke concentration. A standardized set of fire hazard monitoring data is obtained by adopting a unified timestamp system and quality assessment mechanism.

[0021] The specific steps are as follows: S101, Establish a unified timestamp system; Based on GPS clock synchronization technology, use Network Time Protocol (NTP) for time calibration to obtain a unified timestamp with millisecond-level accuracy; S102, Quality assessment and screening of fire hazard monitoring video data; Based on the Laplacian operator-based sharpness assessment algorithm, the image gradient calculation method is used to obtain effective fire hazard monitoring video frame data; By calculating the second-order gradient value of the image, a sharpness score is obtained. When the score is greater than 0.8, the video frame is retained; when the score is less than 0.8, the video frame is not retained.

[0022] S103, Anomaly detection of environmental sensor data for fire hazards; Based on the statistical method of the 3σ criterion, a sliding window anomaly detection algorithm is used to obtain reliable environmental sensor data for fire hazards; Anomaly detection is as follows: the mean and standard deviation of the data sequence are calculated, and when the absolute difference between a data point and the mean is greater than three times the standard deviation, it is judged as an outlier, and an interpolation method is used to repair it. The interpolation method can be linear interpolation or spline interpolation, etc., and the reasonable value of the outlier is estimated by using the values ​​of adjacent normal data points; S104, Standardization of Fire Hazard Inspection Text Data: Based on BERT pre-trained model word segmentation and entity recognition technology, and using a fire safety terminology dictionary matching method, standardized fire hazard inspection text data is obtained. Key entities such as fire-fighting facilities, electrical equipment, and evacuation routes are extracted through named entity recognition, and terminology is standardized. The fire safety terminology dictionary matching method includes the following steps: constructing a fire safety terminology dictionary, which includes terms for fire-fighting facilities, such as fire extinguishers, fire hydrants, sprinkler systems, and smoke detectors; terms for electrical equipment, such as distribution boxes, cables, switches, and sockets; and terms for evacuation routes, such as safety exits, evacuation staircases, etc. Emergency lighting, evacuation signs, etc.; fire hazard terms, including blockage, damage, failure, overload, etc.; each term corresponds to a unique standardized code; term matching is performed on the entity recognition results output by the BERT model, and the similarity between the identified entities and the standard terms in the dictionary is calculated using cosine similarity. A successful match is considered when the similarity is greater than 0.85; the successfully matched terms are standardized and replaced, replacing non-standard expressions in the original text with standard terms in the dictionary, and the correspondence before and after the replacement is recorded; the unmatched terms are manually reviewed and marked, a new term candidate list is established, and the fire safety terminology dictionary is updated regularly to ensure the completeness and timeliness of the dictionary.

[0023] S105, Comprehensive Data Quality Score; Based on three dimensions—completeness, accuracy, and timeliness—a weighted average is used to obtain the data quality score. When the comprehensive score is below 0.7, data re-collection is triggered. S200 uses feature engineering to process standardized fire hazard monitoring data sets to obtain multimodal feature representations; The specific steps are as follows: S201, Visual feature extraction of fire hazards; Based on the pre-trained ResNet-50 network architecture, a fine-tuning strategy of transfer learning is adopted to obtain the visual feature vector of fire hazards; The fine-tuning strategy of transfer learning includes: freezing the parameters of the first three layers of the network, only fine-tuning the last two layers of the network, and using a combination loss function of cross-entropy loss and L2 regularization loss (coefficient 0.001). S202, Semantic Encoding of Fire Hazard Text; Based on a pre-trained RoBERTa model, a context-aware semantic encoding method is used to obtain semantic vectors for fire hazard text. Semantic understanding is achieved through a multi-layer Transformer structure, and average pooling is performed on the output hidden state sequence to obtain a fixed-length text feature vector.

[0024] S203, Temporal Feature Extraction of Fire Hazard Environmental Sensors: Based on a one-dimensional convolutional neural network, a sliding window feature extraction method is used to obtain the feature vector of the fire hazard environmental sensors. Local temporal patterns are extracted using a convolutional kernel of size 10 and stride 1, and a nonlinear transformation is introduced through the ReLU activation function.

[0025] S204, Multimodal Feature Alignment and Fusion of Fire Hazards; Based on a two-layer multilayer perceptron (MLP) network, a nonlinear mapping method is used to obtain a unified feature space for fire hazards; Visual, text, and sensor modal features are mapped to the unified feature space through an intermediate representation space; Specifically, feature alignment and fusion include: inputting the original feature vectors of each modality into the first layer of the multilayer perceptron, performing a linear transformation through a weight matrix, and mapping the features of different dimensions to the first-dimensional intermediate representation space through the first weight matrix; An activation function is applied to the linear transformation result, setting negative values ​​to zero and retaining positive values, thus introducing nonlinear transformation capability to obtain intermediate feature representations. These intermediate feature representations are then input into the second layer of the multilayer perceptron, where a linear transformation is performed through the second weight matrix to uniformly map the intermediate features to the second-dimensional feature space.

[0026] S300 performs weighted fusion processing on the multimodal features to obtain the fused multimodal features and scene context information; Specifically, the following steps are included: S301, Fire Hazard Scene Type Recognition; Employing a lightweight classification network, it identifies six fire hazard scene categories: indoor industrial, indoor commercial, indoor residential, outdoor industrial, outdoor commercial, and outdoor public. Feature vectors from visual, textual, and sensor modalities are concatenated to form a comprehensive feature. This feature is then transformed linearly using a third weight matrix and normalized using softmax to obtain the scene context vector.

[0027] S302, Attention Weight Calculation: Based on scene context information, a single-layer attention network is used to calculate the dynamic weights of each modality; the features and scene vectors of each modality are mapped to the intermediate space respectively, and after fusion, the attention energy value is obtained by tanh activation and output matrix, and the weights of each modality are obtained by softmax normalization. S303, weighted feature fusion: The alignment features of each modality are multiplied by their corresponding attention weights and then summed element by element to obtain a fused multimodal feature vector, which integrates information from the visual, text, and sensor modalities. In some embodiments, a multimodal fusion method based on graph neural networks can also be employed, representing features of different modalities as nodes in a graph, and achieving adaptive fusion by learning the relational weights between nodes. This method constructs a heterogeneous graph containing visual, text, and sensor nodes, and learns the message passing mechanism between nodes through a graph convolutional network. The update process of the graph convolution includes the following steps:

[0028] For each node in the graph, collect the feature representations of all its neighboring nodes to form a neighbor feature set; apply a weight matrix transformation to the feature representation of each neighboring node to map the features to a new representation space; calculate the normalization coefficient, which is based on the square root of the product of the degree of the current node and the degree of its neighboring nodes, to balance the influence of nodes with different degrees; multiply the transformed neighbor features by the corresponding normalization coefficients, and then sum the weighted features of all neighbors to obtain the aggregated feature representation; An activation function is applied to the aggregated feature representation, introducing a nonlinear transformation to obtain the final feature representation of the node in the current layer.

[0029] S400, based on the fused multimodal features, constructs a fire hazard classification network; obtains a fire hazard identification model and hazard identification results; S401, coarse-grained fire hazard classification; employs a multi-classification neural network to identify five major categories of fire hazards: fire protection facilities, electrical safety, evacuation routes, hazardous materials storage, and building structure. The fused features are linearly transformed using the fourth weight matrix and then normalized using softmax to obtain the coarse-grained classification probability distribution.

[0030] S402, Fine-grained fire hazard identification; based on coarse-grained classification results, a category-specific fine-grained classifier is used to identify specific hazard types. The corresponding fine-grained classifier is selected according to the coarse-grained category, and the fused features are input to obtain the fine-grained hazard identification probability distribution.

[0031] In one embodiment of the present invention, the fine-grained classifier includes: A fine-grained classifier for fire protection facilities; it identifies 8 specific hazard types: damaged fire hydrants, expired fire extinguishers, missing fire hoses, malfunctioning fire pumps, malfunctioning automatic sprinkler systems, malfunctioning fire alarms, damaged fire doors, and unclear fire exit markings; the classifier uses an 8×2-dimensional weight matrix and outputs the probability distribution of each subclass through softmax normalization; A fine-grained classifier for electrical safety; it identifies 10 specific hazard types: aging electrical wiring, overheating electrical equipment, electrical short circuit, leakage current device failure, electrical overload, loose electrical connections, damaged electrical insulation, poor electrical grounding, electrical switch failure, and abnormal electrical distribution box; the classifier uses a 10× second-dimensional weight matrix, combined with temperature characteristics and electrical parameters for comprehensive judgment; A fine-grained classifier for evacuation routes; it identifies six specific hazard types: blocked evacuation routes, missing evacuation signs, malfunctioning emergency lighting, insufficient width of evacuation routes, locked evacuation doors, and unclear evacuation routes; the classifier uses a 6× second-dimensional weight matrix to focus on analyzing spatial geometric features and signage text information; A fine-grained classifier for hazardous materials storage; it identifies 7 specific hazard types: improper storage of flammable and explosive materials, chemical leaks, non-standard hazardous materials labeling, abnormal storage temperature, ventilation system failure, explosion-proof equipment failure, and mixed storage of hazardous materials; the classifier uses a 7× second-dimensional weight matrix, integrating gas sensor data and visual features for identification; A fine-grained classifier for building structures identifies nine specific hazard types: incomplete fire compartmentation, non-flame-retardant building materials, abnormal structural load-bearing, peeling of fire-retardant coating, poor building sealing, malfunctioning fire-resistant roller shutters, building cracks and leaks, broken fire-resistant glass, and buildings that do not meet fire protection ratings. The classifier uses a 9× second-dimensional weight matrix to classify based on structural visual features and material properties.

[0032] S403, Uncertainty Quantification: The Monte Carlo Dropout method is used for uncertainty estimation. During the inference phase, the Dropout layer remains active, and 50 forward propagations are performed using the same input. The mean and variance of the prediction results are calculated, with the variance reflecting the degree of prediction uncertainty.

[0033] S404, Confidence Score Calculation: The comprehensive confidence score is calculated based on the predicted probability and uncertainty. The maximum predicted probability is extracted, an uncertainty penalty term (variance × 0.1) is calculated, and a confidence correction factor (1 - penalty term) is obtained. The final confidence score is the product of the maximum probability and the correction factor.

[0034] In some embodiments, a hazard identification method based on deep ensemble learning can also be employed. This method improves identification accuracy and the reliability of confidence estimation by training multiple classifiers with different architectures and combining them with Bayesian model averaging techniques. This method includes multiple backbone networks such as ResNet, EfficientNet, and Vision Transformer, each trained independently and then ensembled through weighted voting. The calculation process for the ensemble prediction includes the following steps:

[0035] Collect the predicted probabilities of all ensemble models for each class, and each model independently outputs its predicted probability distribution for each class; Each model is assigned a weight based on its performance on the validation set; the model with better performance receives a higher weight. Calculate the weighted average probability for each category, multiply the prediction probability of each model for that category by the corresponding model weight, and then sum all the weighted probabilities to obtain the final prediction probability of the ensemble model for that category.

[0036] The confidence score of the ensemble model is calculated by the consistency of predictions among the models, and the specific process is as follows: Calculate the standard deviation of the predicted probabilities of all models to measure the dispersion of the prediction results of different models. The larger the standard deviation, the greater the divergence between models. Calculate the mean of the predicted probabilities of all models, and use it as a baseline value for the prediction strength; Calculate the coefficient of variation, divide the standard deviation by the mean to obtain a measure of relative dispersion; specifically, statistically analyze the prediction results of all sub-models for the same hazard sample to obtain a set of predicted probabilities; calculate the mean of this set; then calculate the squared difference between each predicted probability and the mean, sum them, divide by the number of samples, and take the square root to obtain the standard deviation; finally, divide the standard deviation by the mean to obtain the coefficient of variation. The smaller the coefficient of variation, the higher the consistency between models; subtract the coefficient of variation from 1 to obtain the final integration confidence score. The closer this value is to 1, the more consistent the predictions between models are, and the higher the confidence score.

[0037] S500, based on the hazard identification results and scene context information, uses template matching and multi-objective optimization methods to obtain a fire prevention solution; Specifically, the following steps are included: S501, Fire Hazard Rectification Template Matching; Based on the existing fire hazard rectification suggestion template library, a multi-dimensional matching algorithm is used to obtain a set of candidate fire hazard rectification solutions. For the rectification needs of different types of fire hazards such as fire protection facilities, electrical safety, and evacuation routes, the template matching process includes the following steps:

[0038] Check whether the applicable fire hazard type of each rectification plan in the template library is completely consistent with the detected fire hazard type. Only templates with completely matching types can enter the candidate set. Perform scenario type compatibility matching to check whether the applicable scenario type of the template is included in the compatible scenario list of the current detection scenario to ensure that the rectification plan is suitable for the current environmental conditions. Severity range matching is performed, and the absolute value of the difference between the applicable severity level of the template and the severity level of the detected hazard is calculated. When the difference is less than or equal to a matching tolerance of 0.2, the severity matching is considered successful. S502, Cost-Benefit Assessment of Fire Hazard Rectification; A multi-objective optimization algorithm is used to obtain a comprehensive score for fire hazard rectification plans; The comprehensive score calculation process, considering the cost-benefit characteristics of different fire hazard rectification plans, includes the following steps: The effectiveness score of the fire hazard rectification plan is calculated based on historical data on the effectiveness of fire hazard rectification, reflecting the expected effect of the plan in solving the fire hazard; the cost-benefit score is calculated by normalizing the cost of the plan and then subtracting the normalized cost from 1 to obtain the cost-benefit score, with the lower the cost, the higher the benefit score; the feasibility score is calculated by comprehensively considering the technical difficulty and implementation conditions to evaluate the feasibility of the plan in the current environment. The three scoring items are weighted and summed to obtain the final comprehensive score, which comprehensively reflects the effectiveness, economy and feasibility of the plan.

[0039] S503, Time Prediction Model; Based on historical rectification plans, regression analysis is used to obtain a predicted estimate of the rectification time. The time prediction process includes the following steps:

[0040] A multiple linear regression model is established based on the fire hazard rectification plan. The standardized fire hazard rectification plan is used to predict the rectification time as the dependent variable. The regression model is trained using historical data to determine the regression intercept and the regression coefficients of each variable, and to establish the mathematical relationship for time prediction. The standardized fire hazard rectification plan is input into the trained regression model, and the predicted rectification time is output.

[0041] S504, Multiple Scheme Generation and Ranking: Based on the cost-benefit assessment results of rectification, the Top-K selection algorithm is used to obtain the ranked rectification suggestions; the scheme includes: specific measures description, estimated cost range, suggested completion time limit, relevant legal provisions, risk assessment and other information.

[0042] S600, based on the fire prevention scheme, uses experience playback and parameter regularization methods to update and optimize the existing fire hazard identification model, resulting in an updated fire hazard identification model. Specifically, the following steps are included: S601, Fire Hazard Identification Feedback Data Collection and Preprocessing; Obtaining user-confirmed fire hazard feedback; Fire hazard feedback data includes: fire hazard confirmation labels (correct / incorrect), fire hazard rectification effectiveness scores (1-5 points), newly added fire hazard samples, etc. Preprocessing of the fire hazard feedback data includes outlier detection, label consistency checks, and sample balance adjustments.

[0043] S602, Experience Replay Mechanism; Based on the historical sample buffer, an importance sampling method is used to obtain a mixed training dataset. The experience replay strategy includes the following steps:

[0044] Collect new feedback data, including user-confirmed hazard identification results and rectification effectiveness evaluation data; Historical samples are selected from the historical sample buffer according to the importance sampling method, and the number of samples is determined according to the ratio of new samples to historical samples of 1:3. The newly collected feedback data is merged with the sampled historical samples to form a mixed training dataset. The mixed dataset is subjected to quality checks and balance adjustments to ensure the diversity and representativeness of the training data.

[0045] S603, Parameter Regularization Update: Based on the Elastic Weight Consolidation (EWC) method, a parameter importance-weighted regularization strategy is adopted to obtain an updated hierarchical fire hazard identification network; where the update loss function is cross-entropy loss. S604, A / B testing verification: Based on a small-scale test set, a controlled experiment method was used to obtain the performance verification results of the new hierarchical fire hazard identification network. The A / B testing process includes: randomly selecting 10% of users as the test group, deploying the new model for 7 days of testing, comparing the key indicators of the test group and the control group, including accuracy, recall, and user satisfaction. When the performance improvement of the new model exceeds 2% and there is no negative impact, full deployment is carried out.

[0046] S700, based on the updated fire hazard identification model and hazard identification results, uses RESTful API and real-time statistical methods to obtain multi-dimensional statistical analysis reports and collaborative management instructions; Specifically, the following steps are included: S701, a standardized data exchange interface for fire hazards; designed based on a RESTful architecture, it adopts a structured data transmission protocol in JSON format to achieve a unified standard for fire hazard data exchange. The fire hazard management API interface design includes:

[0047] Fire hazard reporting interface: used to report newly discovered fire hazards; Fire hazard rectification progress update interface: used to update the rectification progress of fire hazards; Fire hazard statistics query interface: used to obtain statistical analysis results of fire hazards; Permission verification interface: used for user authentication. S702, role-based fire safety access control; employs a hierarchical access control strategy to achieve a differentiated fire safety access control mechanism. By matching and verifying user role levels with resource access permissions, secure access control for fire hazard data is achieved.

[0048] S703, Real-time Statistical Analysis of Fire Hazards; Based on a sliding time window, an incremental calculation method is used to obtain real-time statistical indicators of fire hazards. The formula for calculating the Fire Hazard Management Control Index (FHMCI) is as follows:

[0049] ; The rectification completion rate is the ratio of the number of rectified items to the total number of items found; the time efficiency is the ratio of the standard time to the actual time; and the quality score is based on the evaluation of the rectification effect.

[0050] S704, Ranking of Fire Hazardous Areas and Generation of Collaborative Management Instructions; Based on multi-dimensional fire safety assessment indicators, a comprehensive scoring algorithm is used to obtain rankings of fire hazard management areas and collaborative management suggestions. The formula for calculating the area ranking score is as follows:

[0051] ; Based on the ranking results, the system automatically generates fire safety collaborative management instructions, including key attention area identification, fire resource allocation suggestions, and special inspection plans.

[0052] In some embodiments, a fire hazard identification deployment architecture that combines edge computing and cloud computing can also be adopted. The computationally intensive fire hazard feature extraction and model inference tasks are deployed on edge devices, while the fire hazard data storage and statistical analysis tasks are deployed on cloud servers. Low-latency fire safety data transmission is achieved through 5G networks, thereby reducing network bandwidth requirements and cloud computing costs while ensuring real-time fire hazard identification. The edge device configuration includes an NVIDIA Jetson Xavier NX compute module, equipped with 8GB LPDDR4x memory and 256GB NVMe SSD storage, supporting multiple fire monitoring video inputs and fire safety sensor interfaces. The cloud server adopts a distributed architecture, including a load balancer, a fire hazard application server cluster, a fire safety database cluster, and a file storage system.

[0053] In one embodiment of the present invention, a fire hazard identification system based on a multimodal large model is provided, such as... Figure 2 As shown, it includes: The data acquisition module is used to acquire the raw data stream of fire hazards, process the raw data stream, and obtain a standardized set of fire hazard monitoring data. The feature extraction module is used to process the standardized fire hazard monitoring data set using feature engineering to obtain multimodal feature representations; The feature fusion module is used to perform weighted fusion processing on multimodal features to obtain fused multimodal features and scene context information; The fire hazard identification module constructs a fire hazard classification network based on fused multimodal features, thereby obtaining a fire hazard identification model and hazard identification results. It is recommended that the generation module, based on the hazard identification results and scenario context information, adopt template matching and multi-objective optimization methods to obtain fire prevention solutions; The model update module, based on the fire prevention plan, uses experience playback and parameter regularization methods to update and optimize the existing fire hazard identification model, resulting in an updated fire hazard identification model. The collaborative management module, based on the updated fire hazard identification model and hazard identification results, uses RESTful API and real-time statistical methods to obtain multi-dimensional statistical analysis reports and collaborative management instructions.

[0054] In one embodiment of the present invention, an application example for fire hazard identification is provided: An industrial park has deployed the fire hazard identification system of this invention. The park contains various types of buildings, including chemical plants, machine shops, and warehouses. The system monitors the safety status of the park in real time through a multimodal sensor network deployed at various key locations.

[0055] During a routine inspection, the system detected an anomaly in an electrical device within the chemical plant workshop. The specific detection process is as follows:

[0056] Data acquisition: The infrared thermal imager detected an abnormal temperature in the electrical cabinet, reaching 65°C, exceeding the normal range of 45°C. Video surveillance revealed slight smoke around the electrical cabinet, and environmental sensors detected localized temperature increases and a slight odor. Data preprocessing: Since temperature data and other sensor data have different dimensions and numerical ranges, the system performs standardization processing on the multi-source sensor data; the standardized temperature data has the characteristics of zero mean and unit variance. Gas concentration data normalization: The normalized data ranges from 0 to 1, ensuring that data from different types of sensors are fused and calculated within the same numerical range; Feature extraction and fusion: The system extracts temperature distribution features from infrared images, smoke texture features from visible light images, and temporal features of temperature and gas concentration from sensors, and obtains a comprehensive feature representation through adaptive weighted fusion; Fire hazard identification: The hierarchical identification network first classifies it as an electrical safety fire hazard, and then further subdivides it into electrical equipment overheating type, with a confidence score of 0.92.

[0057] Rectification suggestion generation: The system matches the corresponding rectification template and generates specific suggestions: Immediately disconnect the power and check the internal connections of the electrical cabinet; check whether the electrical cabinet's heat dissipation system is working properly; replace aging electrical components; estimated rectification cost: 8,000-12,000 yuan; suggested completion time: within 24 hours; Results Verification: After the company made the necessary rectifications as suggested, the temperature at that location returned to normal, and the system continued to monitor the area for 7 days without detecting any further abnormalities.

[0058] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A fire hazard identification method based on a multi-modal large model, characterized in that, Includes the following steps: The raw data stream of fire hazards is acquired, and the raw data stream is processed to obtain a standardized set of fire hazard monitoring data. A standardized fire hazard monitoring dataset is processed using feature engineering to obtain a multimodal feature representation, including: A deep learning network architecture is adopted, and a transfer learning strategy is used to obtain visual feature vectors of fire hazards; A pre-trained language model is used to obtain the semantic vector of fire hazard text through a context-aware semantic encoding method; A convolutional network is used to extract feature vectors from fire hazard environmental sensors using a sliding window feature extraction method. A multilayer perceptron network is used, and a nonlinear mapping method is employed to obtain a unified-dimensional multimodal feature representation. The multimodal features are weighted and fused to obtain the fused multimodal features and scene context information, including: A lightweight classification network is used to obtain scene context information; Attention weights are calculated using a single-layer attention network based on scene context information to obtain dynamic weights for each modality. Weighted feature fusion is performed by using a weighted average method based on the calculated attention weights to obtain the fused multimodal features. Based on the fused multimodal features, a fire hazard classification network is constructed; a fire hazard identification model and hazard identification results are obtained, including: A multi-class neural network is used to obtain the probability distribution of fire hazards; a classifier is then used to obtain the specific types of fire hazards. Based on the stochastic forward propagation method, the mean and variance of the prediction results are calculated to obtain the uncertainty estimate; based on the prediction variance, a normalization method is used to obtain the confidence score. Based on the hazard identification results and scenario context information, a fire prevention plan is obtained using template matching and multi-objective optimization methods; including: The rectification template matching is based on a pre-built rectification suggestion template library and uses a multi-dimensional matching algorithm to obtain a set of candidate rectification solutions. Cost-benefit assessment, based on a predefined cost model and benefit assessment function, employs a multi-objective optimization algorithm to obtain a comprehensive score for the rectification plan; The time prediction model, based on historical rectification data, uses regression analysis to obtain a predicted estimate of the rectification time. Multiple schemes are generated and sorted. Based on the scoring results, a selection algorithm is used to obtain the sorted fire prevention schemes. Based on the fire prevention plan, the existing fire hazard identification model is updated and optimized using experience playback and parameter regularization methods to obtain the updated fire hazard identification model. Based on the updated fire hazard identification model and hazard identification results, a multi-dimensional statistical analysis report and collaborative management instructions are obtained using RESTful API and real-time statistical methods. 2.The fire hazard identification method based on a multi-modal large model according to claim 1, wherein, The steps of processing the raw data stream to obtain a standardized fire hazard monitoring data set include: A unified timestamp is obtained based on GPS clock synchronization technology and network time protocols; A sharpness assessment algorithm based on the Laplacian operator is used to obtain video frame data for fire hazard monitoring through image gradient calculation. Based on the statistical method of the 3σ criterion, fire hazard environmental sensor data are obtained through sliding window anomaly detection. Based on the BERT pre-trained model, word segmentation and entity recognition technologies are used, and a fire safety terminology dictionary matching method is adopted to obtain standardized fire hazard inspection text data. A data quality score is obtained by weighting the scores based on three dimensions: completeness, accuracy, and timeliness. 3.The fire hazard identification method based on a multi-modal large model according to claim 1, wherein, The steps for updating and optimizing the existing fire hazard identification model using experience replay and parameter regularization methods to obtain the updated fire hazard identification model include: Feedback data collection and preprocessing: Based on the fire hazard identification results and rectification effect evaluation confirmed by users, data cleaning and labeling methods are used to obtain processed feedback data and use it as training samples. The experience replay mechanism, based on the historical sample buffer, uses an importance sampling method to obtain a mixed training dataset; The parameter regularization update is based on the elastic weight consolidation method and adopts a parameter importance weighted regularization strategy to obtain the updated fire hazard identification model. A / B testing was used to validate the new model by employing a controlled experiment method based on a small-scale test set. 4.The fire hazard identification method based on a multi-modal large model according to claim 1, wherein, The steps for obtaining multi-dimensional statistical analysis reports and collaborative management instructions based on the updated fire hazard identification model and hazard identification results, using RESTful API and real-time statistical methods, include: A standardized data exchange interface, based on a RESTful architecture, adopts a structured data transmission protocol in JSON format, resulting in a unified data exchange standard. Role-based access control employs a hierarchical access control strategy, resulting in a differentiated access control mechanism. Real-time statistical analysis, based on a sliding time window, uses an incremental calculation method to obtain real-time statistical indicators; Regional ranking and collaborative instruction generation are based on multi-dimensional evaluation indicators and a comprehensive scoring algorithm to obtain regional ranking and collaborative management suggestions.

5. The fire hazard identification method based on a multimodal large model according to claim 1, characterized in that, Also includes: Ensemble learning for fire hazard identification is based on multiple classifiers with different architectures, and Bayesian model averaging is used to obtain ensemble prediction results. The consistency assessment of predictions among models is based on the standard deviation and mean of the predicted probabilities of all models in the integrated prediction results, resulting in an integrated confidence score.

6. A fire hazard identification system based on a multimodal large model, used to execute the fire hazard identification method based on a multimodal large model as described in any one of claims 1-5, characterized in that, include: The data acquisition module is used to acquire the raw data stream of fire hazards, process the raw data stream, and obtain a standardized set of fire hazard monitoring data. The feature extraction module is used to process the standardized fire hazard monitoring data set using feature engineering to obtain multimodal feature representations; The feature fusion module is used to perform weighted fusion processing on multimodal features to obtain fused multimodal features and scene context information; The fire hazard identification module constructs a fire hazard classification network based on fused multimodal features; Obtain the fire hazard identification model and hazard identification results; It is recommended that the generation module, based on the hazard identification results and scenario context information, adopt template matching and multi-objective optimization methods to obtain fire prevention solutions; The model update module, based on the fire prevention plan, uses experience playback and parameter regularization methods to update and optimize the existing fire hazard identification model, resulting in an updated fire hazard identification model. The collaborative management module, based on the updated fire hazard identification model and hazard identification results, uses RESTful API and real-time statistical methods to obtain multi-dimensional statistical analysis reports and collaborative management instructions.

Images