A multi-modal AI-based construction fire problem identification method and system

Abstract

This invention relates to the field of fire protection technology, and in particular to a method and system for identifying fire safety issues in construction projects based on multimodal AI. The method includes the following steps: performing cross-modal spatiotemporal alignment and modal missing compensation processing on visible light images, thermal images, and construction text records at the construction site to generate a standardized multimodal data stream; performing targeted feature extraction on different modal data in the standardized multimodal data stream to obtain geometric semantic features of the visible light images, frequency domain features of abnormal heat sources in the thermal images, and entity relationship features in the text records; inputting the geometric semantic features, frequency domain features of abnormal heat sources, and entity relationship features into a dynamic fusion module based on an attention mechanism, and generating a deeply correlated unified multimodal feature tensor through cross-modal feature interaction and adaptive weight allocation; and inputting the unified multimodal feature tensor into a multi-task recognition network capable of parallel decoding.

Description

Technical Field

[0001] This invention relates to the field of fire protection technology, and in particular to a method and system for identifying fire protection problems in construction projects based on multimodal AI. Background Technology

[0002] Construction sites are complex and dynamic environments, with numerous and often concealed fire safety hazards. Traditional fire safety inspections rely primarily on manual patrols and periodic testing, which presents the following problems: Inspection results are easily affected by factors such as personnel experience and fatigue, making it difficult to achieve standardized and systematic hazard identification and assessment; Common detection methods are mostly based on a single data source (such as visible light images or thermal imaging), lacking the fusion and collaborative analysis of multi-source information, and making it difficult to fully reflect the spatial, temporal and semantic characteristics of potential hazards; Manual inspections are time-consuming and inefficient, making it difficult to detect dynamic hazards in a timely manner and unable to achieve real-time monitoring and risk warning. Traditional methods struggle to deeply integrate identification results with Building Information Modeling (BIM), failing to visually represent the location, type, risk level, and potential development and spread trends of hazards in three-dimensional space.

[0003] In recent years, computer vision-based recognition methods have gradually emerged, but they still have the following limitations: It focuses on a single modality (such as image or temperature) and does not fully integrate multimodal information such as text and time series; it lacks cross-modal spatiotemporal alignment and missing data compensation mechanisms; the recognition results are mostly isolated classifications or detection boxes, lacking pixel-level fine segmentation and semantic interpretation; it fails to be deeply integrated with the BIM system and is difficult to support risk inference and decision support based on three-dimensional scenes. Summary of the Invention

[0004] Therefore, it is necessary to provide a construction fire protection problem identification method and system based on multimodal AI to address the above-mentioned technical problems. By integrating visible light images, thermal images and construction text records, the method achieves complementarity of visual, temperature and semantic information, overcomes the limitations of a single data source, and significantly improves the coverage and accuracy of hazard identification.

[0005] This invention provides a method for identifying fire safety issues in construction projects based on multimodal AI, the method comprising: Cross-modal spatiotemporal alignment and modal missing compensation processing are performed on visible light images, thermal images, and construction text records at the construction site to generate a standardized multimodal data stream; Targeted feature extraction is performed on different modal data in the standardized multimodal data stream to obtain geometric semantic features of visible light images, frequency domain features of abnormal heat sources in thermal images, and entity relationship features in text records. The geometric semantic features, abnormal heat source frequency domain features, and entity relationship features are input into a dynamic fusion module based on an attention mechanism. Through cross-modal feature interaction and adaptive weight allocation, a deeply correlated unified multimodal feature tensor is generated. The unified multimodal feature tensor is input into a multi-task recognition network capable of parallel decoding, thereby decoding to obtain pixel-level hazard area segmentation map, hazard type classification and risk level quantification value; The hazard area segmentation map is post-processed and optimized to generate a high-precision hazard area mask and a corresponding visual interpretation heatmap; The hidden danger area mask, hidden danger type, risk level, and the visual interpretation heat map are semantically mapped and dynamically bound to the components in the building information model. Based on the risk diffusion model, situation inference is performed to generate a dynamic three-dimensional fire risk map that integrates real-time identification results, risk prediction and hidden danger visual interpretation.

[0006] In one embodiment, the cross-modal spatiotemporal alignment and modal missing compensation processing of visible light images, thermal images, and construction text records at the construction site to generate a standardized multimodal data stream includes: Feature point matching based on scale-invariant feature transformation is performed on visible light images and thermal images to achieve spatial alignment, and time synchronization is achieved through timestamp interpolation to establish a unified spatiotemporal coordinate system. A text parser based on regular expressions is used to extract key time and location information from the construction text and automatically associate and match it with the spatiotemporal coordinate system. The missing modal fragments after alignment are identified by a modal missing detector, and a pre-trained conditional generative adversarial network model is invoked to generate synthetic data of the missing modal based on the existing modal data. The complete multimodal data stream is Z-score normalized and then encapsulated into a normalized multimodal data stream according to the time series.

[0007] In one embodiment, the targeted feature extraction of different modal data in the standardized multimodal data stream to obtain geometric semantic features of the visible light image, frequency domain features of abnormal heat sources in the thermal image, and entity relationship features in the text record includes: The visible light image is input into a pre-trained masked region convolutional neural network to extract pixel-level geometric semantic features containing object boundaries and positional relationships. Short-time Fourier transform was performed on the thermal imaging sequence to extract high-frequency energy spectrum features of temperature changes, and frequency domain feature enhancement was performed by combining the historical anomalous heat source database. We employ a bidirectional encoder-based representation and a bidirectional long short-term memory conditional random field model to perform named entity recognition and relation extraction on construction text, thereby constructing a structured entity relation feature map.

[0008] In one embodiment, the step of inputting the geometric semantic features, abnormal heat source frequency domain features, and entity relationship features into a dynamic fusion module based on an attention mechanism, and generating a deeply correlated unified multimodal feature tensor through cross-modal feature interaction and adaptive weight allocation, includes: Construct a cross-modal multi-head attention layer, calculate the correlation matrix between features of different modalities, and realize the interaction between features; Learnable modal weight parameters are introduced, and weights are assigned through a normalized exponential function to concatenate the weighted features. The concatenated features are input into a fully connected layer for dimensionality reduction and nonlinear enhancement, generating a unified multimodal feature tensor.

[0009] In one embodiment, inputting the unified multimodal feature tensor into a multi-task recognition network capable of parallel decoding to obtain pixel-level hazard region segmentation maps, hazard type classifications, and risk level quantification values ​​includes: The unified multimodal feature tensor is input into the shared encoder based on the residual network to extract deep shared features; Use a U-shaped network decoder to output a hazard area segmentation map; Use a fully connected classifier to output a hazard type classification; Use a regression network to output a quantitative value for the risk level.

[0010] In one embodiment, the post-processing optimization of the hazard area segmentation map to generate a high-precision hazard area mask and a corresponding visual interpretation heatmap includes: Morphological closing operations are performed on the segmentation map of the potential hazard area to eliminate small voids, and adjacent areas are merged through connected component analysis. The optimized segmentation image is binarized and thresholded to generate a high-precision mask for potential hazard areas. Based on the gradient-weighted class activation mapping method, a visually interpretable heatmap is generated using gradient information in a multi-task recognition network.

[0011] In one embodiment, the hazard area mask, hazard type, risk level, and visual interpretation heatmap are semantically mapped and dynamically bound to components in the building information model. Based on a risk diffusion model, situational analysis is performed to generate a dynamic three-dimensional fire risk map that integrates real-time identification results, risk prediction, and visual interpretation of hazards. This includes: The hazard area mask is mapped onto the building information model through a coordinate transformation matrix, and then matched with the corresponding components for identification and semantic association. The hazard type, risk level, and heat map are dynamically attached to the building information model component object; Based on the cellular automata risk diffusion model, the spread path and impact range of hidden dangers in three-dimensional space are simulated to generate a dynamic three-dimensional fire risk map.

[0012] The present invention also provides a construction fire protection problem identification system based on multimodal AI, the device comprising: The data stream alignment module is used to perform cross-modal spatiotemporal alignment and modal missing compensation processing on visible light images, thermal images and construction text records at the construction site, and generate standardized multimodal data streams; The feature extraction module is used to perform targeted feature extraction on different modal data in the standardized multimodal data stream to obtain geometric semantic features of visible light images, frequency domain features of abnormal heat sources in thermal images, and entity relationship features in text records. The dynamic fusion module is used to input the geometric semantic features, abnormal heat source frequency domain features and entity relationship features into an attention mechanism-based dynamic fusion module, and generate a deeply associated unified multimodal feature tensor through cross-modal feature interaction and adaptive weight allocation. The multi-task recognition module is used to input the unified multimodal feature tensor into a multi-task recognition network that can decode in parallel, thereby decoding to obtain pixel-level hazard area segmentation map, hazard type classification and risk level quantification value; The post-processing module is used to perform post-processing optimization on the hazard area segmentation map to generate a high-precision hazard area mask and a corresponding visual interpretation heat map. The map generation module is used to semantically map and dynamically bind the hazard area mask, hazard type, risk level, and visual interpretation heat map with the components in the building information model, and perform situational inference based on the risk diffusion model to generate a dynamic three-dimensional fire risk map that integrates real-time identification results, risk prediction, and visual interpretation of hazards.

[0013] The present invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the construction fire protection problem identification method based on multimodal AI as described above.

[0014] The present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements the construction fire protection problem identification method based on multimodal AI as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the construction fire protection problem identification method based on multimodal AI as described above.

[0016] The aforementioned construction fire safety problem identification method and system based on multimodal AI achieves complementary visual, temperature, and semantic information by fusing visible light images, thermal images, and construction text records, overcoming the limitations of single data sources and significantly improving the coverage and accuracy of hazard identification. Employing techniques such as feature point matching, timestamp interpolation, and conditional generative adversarial networks, it achieves spatiotemporal synchronization of multi-source data and intelligent compensation for missing modalities, ensuring the integrity and consistency of the data stream and enhancing the system's adaptability in complex on-site environments. Through a dynamic fusion module based on an attention mechanism and a multi-task recognition network, it achieves pixel-level hidden... Parallel decoding of hazard segmentation, type classification, and risk level quantification outputs results with both spatial details and semantic information, supporting refined safety management. By dynamically binding the identification results to the BIM model and simulating risk diffusion based on a cellular automata model, a dynamic 3D fire risk map integrating real-time identification, risk prediction, and visual interpretation is generated, providing an intuitive and scientific basis for fire safety management. The entire process is automated, from data alignment and feature extraction to identification output and map generation, significantly reducing manual intervention and improving the efficiency and objectivity of hazard investigation, making it suitable for large-scale, long-term construction safety monitoring. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 A flowchart of the construction fire protection problem identification method based on multimodal AI provided by the present invention; Figure 2 A module diagram of a construction fire protection problem identification system based on multimodal AI provided by this invention; Figure 3 An internal structural diagram of the computer device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] The following is combined with Figures 1 to 3 This invention describes a method and system for identifying fire safety issues in construction projects based on multimodal AI.

[0021] In one embodiment, a construction fire safety problem identification method based on multimodal AI includes the following steps: Step S100: Perform cross-modal spatiotemporal alignment and modal missing compensation processing on the visible light images, thermal images and construction text records of the construction site to generate a standardized multimodal data stream; Step S200: Targeted feature extraction is performed on different modal data in the standardized multimodal data stream to obtain geometric semantic features of visible light images, frequency domain features of abnormal heat sources in thermal imaging images, and entity relationship features in text records. Step S300: Input the geometric semantic features, abnormal heat source frequency domain features and entity relationship features into a dynamic fusion module based on an attention mechanism, and generate a deeply correlated unified multimodal feature tensor through cross-modal feature interaction and adaptive weight allocation. Step S400: Input the unified multimodal feature tensor into a multi-task recognition network capable of parallel decoding, thereby decoding to obtain pixel-level hazard area segmentation map, hazard type classification and risk level quantification value; Step S500: Post-processing optimization is performed on the hazard area segmentation map to generate a high-precision hazard area mask and a corresponding visual interpretation heat map. Step S600: Semantically map and dynamically bind the hazard area mask, hazard type, risk level, and visual interpretation heat map with the components in the building information model, and perform situational simulation based on the risk diffusion model to generate a dynamic three-dimensional fire risk map that integrates real-time identification results, risk prediction, and hazard visual interpretation.

[0022] The aforementioned construction fire safety problem identification method based on multimodal AI achieves complementary visual, temperature, and semantic information by fusing visible light images, thermal images, and construction text records. This overcomes the limitations of single data sources and significantly improves the coverage and accuracy of hazard identification. Employing techniques such as feature point matching, timestamp interpolation, and conditional generative adversarial networks, it achieves spatiotemporal synchronization of multi-source data and intelligent compensation for missing modalities, ensuring the integrity and consistency of the data stream and enhancing the system's adaptability in complex on-site environments. Furthermore, through a dynamic fusion module based on an attention mechanism and a multi-task identification network, it achieves pixel-level hazard identification. The parallel decoding of segmentation, type classification, and risk level quantification outputs results that combine spatial details and semantic information, supporting refined safety management. By dynamically binding the identification results with the BIM model and simulating risk diffusion based on the cellular automata model, a dynamic 3D fire risk map integrating real-time identification, risk prediction, and visual interpretation is generated, providing an intuitive and scientific basis for fire safety management. The entire process is automated, from data alignment and feature extraction to identification output and map generation, significantly reducing manual intervention and improving the efficiency and objectivity of hazard investigation, making it suitable for large-scale, long-term construction safety monitoring.

[0023] In one embodiment, the process of performing cross-modal spatiotemporal alignment and modal missing compensation processing on visible light images, thermal images, and construction text records at the construction site to generate a standardized multimodal data stream includes the following steps: Step S110: Perform feature point matching based on scale-invariant feature transformation on the visible light image and the thermal image to achieve spatial alignment, and achieve time synchronization through timestamp interpolation to establish a unified spatiotemporal coordinate system.

[0024] Specifically, an improved SIFT feature matching algorithm is employed. Addressing the characteristics of blurred edges and limited texture information in thermal images, the algorithm enhances the edges of the visible light image before feature extraction and performs adaptive histogram equalization on the thermal image to improve contrast. After matching, the RANSAC (Random Sample Consensus) algorithm is used to eliminate mismatched point pairs, ultimately yielding an accurate homography transformation matrix H.

[0025] This matrix can project each pixel (x_t, y_t) in the thermal image onto the coordinate system (x_v, y_v) of the visible light image, achieving sub-pixel level spatial registration; To handle asynchronous acquisition or frame drops, the system maintains a timing buffer based on the master clock (usually the timestamp of the visible light image stream). For thermal images, if there is no corresponding data at time t, a motion-compensated bilinear time interpolation method is used: the thermal image data at time t is synthesized using the two most recent thermal images (t_1, t_2) and the pixel displacement field estimated by optical flow, thus ensuring strict alignment with the visible light image on the time axis.

[0026] Step S120: Use a regular expression-based text parser to extract key time and location information from the construction text and automatically associate and match it with the spatiotemporal coordinate system.

[0027] Specifically, a domain-adaptive regular expression and keyword dictionary combination model is constructed. In addition to general time and location patterns, the dictionary includes location codes specific to engineering projects (such as "WB-3F-C-12" representing column No. 12 in section C on the 3rd floor of Building B in the West Zone) and process terms (such as "rebar tying work surface").

[0028] The model first matches explicit patterns using regular expressions, then uses keywords to perform contextual retrieval and complete implicit information (for example, inferring the location entity "floor slab" from the context of "pouring completed"). The extracted time and location information will be associated with the unified spatiotemporal coordinate system established in step S110. Location keywords will be converted into 3D coordinates or region IDs through a predefined project BIM coordinate-semantic location mapping table. Time information will be fuzzily matched with image timestamps (allowing a tolerance of ±30 seconds) to ultimately form a multimodal data record with a unified spatiotemporal label.

[0029] Step S130: Identify the missing modal segments after alignment using a modal missing detector, and call a pre-trained conditional generative adversarial network model to generate synthetic data of the missing modal segments based on the existing modal data.

[0030] Specifically, the system monitors the timestamp sequence of each modality's data stream in real time. If a modality has no data for N consecutive time intervals (e.g., 3 frames), it is marked as a missing modality segment. A pre-trained multimodal conditional generative adversarial network (cGAN) is invoked. This network is trained on paired data (visible light, thermal imaging, and text features). During inference, for example, if the thermal imaging modality is missing, the current visible light image and text features are used as conditional inputs, and the generator G outputs the corresponding synthetic thermal image. The discriminator D ensures during training that the generated image is consistent with the real data in terms of temperature distribution and abnormal heat source morphology.

[0031] Step S140: Perform Z-score normalization on the complete multimodal data stream and encapsulate it into a normalized multimodal data stream according to the time series.

[0032] Specifically, for each modality's data stream, its mean (μ) and standard deviation (σ) within a time window (e.g., the past hour) are calculated independently, and the current data is transformed: z = (x - μ) / σ. This process transforms the data of each modality into a distribution with a mean of 0 and a standard deviation of 1, eliminating the influence of dimensions, where x represents the original data point in a specific independent modality at the current time.

[0033] The standardized visible light image, thermal image, and text feature vector are packaged in chronological order to form a structured multimodal data sample, which serves as the input to the feature extraction module.

[0034] In one embodiment, the targeted feature extraction of different modal data in the standardized multimodal data stream to obtain geometric semantic features of visible light images, frequency domain features of abnormal heat sources in thermal images, and entity relationship features in text records includes the following steps: Step S210: Input the visible light image into a pre-trained mask region convolutional neural network to extract pixel-level geometric semantic features including object boundaries and positional relationships.

[0035] Specifically, a Mask R-CNN model finely tuned on a construction engineering scene dataset (such as ConstructSeg) is used. This pre-trained model can already identify objects such as workers, safety helmets, machinery, material piles, temporary facilities, and fire-fighting equipment in construction scenes quite well. The model not only outputs instance segmentation masks and categories, but also extracts high-level feature maps after the ROI-Align layer. These feature maps encode the precise boundaries of each target instance, its positional relationship relative to other objects in the image, and its semantic category information, collectively forming geometric semantic features used to determine spatial relationship hazards such as "flammable materials piled up next to a fire lane".

[0036] Step S220: Perform short-time Fourier transform on the thermal imaging sequence to extract high-frequency energy spectrum features of temperature changes, and enhance frequency domain features by combining them with a historical abnormal heat source database.

[0037] Specifically, a short-time Fourier transform (STFT) is performed pixel-by-pixel or region-by-region on the registered thermal image sequence (time window T). This transform converts the temperature-time variation signal into a time-frequency graph, allowing analysis of the temporal distribution of different frequency components (such as high frequencies corresponding to rapid heating). High-frequency energy spectrum features (e.g., energy integrals within the 2-10Hz band) are extracted from the time-frequency plot. These features are then compared with typical patterns in a historical anomalous heat source database (e.g., overheating spectra of motor bearings, deterioration spectra of cable joints). If the matching degree exceeds a threshold, the feature vector for that region is weighted and enhanced, significantly improving the model's sensitivity to potential thermal hazards.

[0038] The short-time Fourier transform formula is: in, This represents the time-frequency analysis result, indicating the time... and frequency The complex amplitude at the point (including amplitude and phase information); This refers to window functions (such as the Hanning window and the Gaussian window) used to measure time. A local segment of the signal is captured nearby. The center of the window is located at time. ; This represents the input time-domain signal, here referring to a pixel or region in the thermal imaging sequence at different times. Temperature value (or temperature change value); The kernel function represents the Fourier transform, used to convert a time-domain signal to the frequency domain; It represents a frequency variable, and the unit is usually Hertz (Hz). This indicates the center point of time, representing the current time position in the analysis. This represents the integral variable, and a point on the time axis.

[0039] Step S230: The construction text is named entity recognition and relation extraction are performed using a bidirectional encoder representation and a bidirectional long short-term memory conditional random field model to construct a structured entity relation feature map.

[0040] Specifically, a pipeline of "BERT + BiLSTM + CRF" is adopted. The pre-trained BERT model converts the text into word vectors rich in contextual information. The BiLSTM layer further captures long-distance dependencies in the sentence. The CRF layer acts as a decoder, considering the transition probability between labels (e.g., "material" is more likely to be followed by "name" than "time"), and performs globally optimal label sequence prediction; The model identifies entities such as "electric welding operation," "oxygen cylinder," and "no welding hopper," and extracts relationships such as "operation usage" and "violation of regulations." These entities and relationships are organized into a structured knowledge graph, which serves as the input for subsequent fusion modules as entity and relationship features.

[0041] For example, the map can clearly indicate a high-risk semantic event such as "open flame work is underway at a certain location".

[0042] In one embodiment, the step of inputting the geometric semantic features, abnormal heat source frequency domain features, and entity relationship features into a dynamic fusion module based on an attention mechanism, and generating a deeply correlated unified multimodal feature tensor through cross-modal feature interaction and adaptive weight allocation, includes the following steps: Step S310: Construct a cross-modal multi-head attention layer, calculate the correlation matrix between features of different modalities, and realize the interaction between features.

[0043] Specifically, a multi-head attention layer is constructed, treating the features of the three modalities as a set of query Q, key K, and value V. By calculating the dot product attention between Q and K, the model can automatically learn cross-modal correspondences such as "the 'wire connector' mentioned in the text should be associated with a certain geometric region in the image and a certain high-temperature point in the thermal imaging".

[0044] Step S320: Learnable modal weight parameters are introduced, and weights are allocated through a normalized exponential function to concatenate the weighted features.

[0045] Specifically, a set of learnable modality importance weight parameters [α_v, α_t, α_x] are introduced, corresponding to visual, thermal, and text modalities, respectively. The actual weights [w_v, w_t, w_x] are obtained by normalization using the softmax function. During training, the network automatically adjusts these weights based on the current input content. For example, in nighttime or smoky environments, the visible light image quality is poor, and its weight w_v will automatically decrease, while the weights for thermal imaging and text recording will increase accordingly.

[0046] Step S330: Input the spliced ​​features into the fully connected layer for dimensionality reduction and nonlinear enhancement to generate a unified multimodal feature tensor.

[0047] Specifically, the features after attention interaction and weighting are concatenated and fed into a multilayer perceptron (MLP) for dimensionality reduction and nonlinear fusion, ultimately outputting a compact and information-rich unified multimodal feature tensor.

[0048] In one embodiment, inputting the unified multimodal feature tensor into a multi-task recognition network capable of parallel decoding to obtain pixel-level hazard region segmentation maps, hazard type classifications, and risk level quantification values ​​includes the following steps: Step S410: Input the unified multimodal feature tensor into the shared encoder based on the residual network to extract deep shared features.

[0049] Specifically, the unified feature tensor is first deep encoded through a lightweight ResNet module to extract high-level abstract features shared across tasks.

[0050] Step S420: Use a U-shaped network decoder to output a hazard area segmentation map.

[0051] Specifically, the U-Net decoder is used to upsample the shared features and make skip connections with the shallow features in the encoder to gradually restore spatial details and output a pixel-level hazard segmentation map to accurately delineate the outline of the hazard area.

[0052] Step S430: Use a fully connected classifier to output the hazard type classification.

[0053] Specifically, after performing global average pooling on the shared features, the probability distribution of the type of hazard is output through a fully connected layer + Softmax (e.g., short circuit 0.7, flammable material accumulation 0.2, others 0.1).

[0054] Step S440: Use a regression network to output a quantified risk level value.

[0055] Specifically, another fully connected regression network takes into account factors such as the type, area, and temperature of the hazard and outputs a continuous risk level value (e.g., 0.85, indicating high risk).

[0056] In one embodiment, the post-processing optimization of the hazard area segmentation map to generate a high-precision hazard area mask and a corresponding visual interpretation heatmap includes the following steps: Step S510: Perform morphological closing operation on the segmentation map of the hidden danger area to eliminate small holes, and merge adjacent areas through connected component analysis.

[0057] Specifically, morphological closing operations (dilation followed by erosion) are performed on the initial segmentation image to fill small holes and smooth the edges. Then, connected component analysis is used to merge spatially adjacent suspected regions.

[0058] Step S520: The optimized segmentation image is subjected to binarization thresholding to generate a high-precision hazard area mask.

[0059] Specifically, a clear binary hazard area mask is generated using a fixed threshold or an adaptive threshold method.

[0060] Step S530: Based on the gradient-weighted class activation mapping method, a visual interpretation heatmap is generated using gradient information in the multi-task recognition network.

[0061] Specifically, the Grad-CAM++ method is used to generate visual explanations. This is achieved by calculating the uniform feature tensor of the target category c (e.g., "short circuit") in the classification task. The gradients are calculated, weighted, and summed, then activated using ReLU to obtain a class activation heatmap. This heatmap highlights the areas in the image that contribute the most to the network's "short circuit" judgment, making the AI ​​decision-making process "white-box" and facilitating review by safety personnel.

[0062] The gradient-weighted class activation mapping formula is: in, Indicates category The generated class activation heatmap (two-dimensional matrix) has the same size as the input feature map, and the numerical values ​​indicate whether the region is identified as a class. The importance of; The linear rectified function is defined as follows: This is used to filter out negative contributions and only retain contributions to the category. Areas with a positive impact; Indicates category The The weights of each feature map reflect the importance of that feature map to the category. The degree of importance. The calculation formula is: in, Indicates network to category The predicted score (logits before softmax); Indicates the first Each feature map in spatial location The activation value at that location; Indicates category score Activation values ​​of feature maps The gradient represents the degree of influence of the activation value on the category score; Represents the spatial dimensions of the feature map (i.e. The total number of pixels), used for global average pooling; Indicates the first Each feature map (two-dimensional matrix) comes from a deep convolutional layer (usually the last convolutional layer) in a multi-task recognition network.

[0063] In one embodiment, the hidden danger area mask, hidden danger type, risk level, and the visual interpretation heat map are semantically mapped and dynamically bound to the components in the building information model. Based on a risk diffusion model, situational analysis is performed to generate a dynamic three-dimensional fire risk map that integrates real-time identification results, risk prediction, and visual interpretation of hidden dangers. This includes the following steps: Step S610: The hazard area mask is mapped onto the building information model through a coordinate transformation matrix, and the corresponding components are matched and semantically associated.

[0064] Specifically, by using camera calibration parameters and the world coordinate system of the BIM model, the two-dimensional hazard mask is projected in reverse into three-dimensional space, and a collision detection algorithm is used to accurately associate it with components in the BIM (such as "Electrical cable tray on the third floor - No. E-003").

[0065] Step S620: The hazard type, risk level, and heat map are attached as dynamic attributes to the building information model component object.

[0066] Specifically, the identified hazard types, risk levels, and explanatory heatmaps are dynamically attached to the component object, realizing the "state empowerment" of the BIM model.

[0067] Step S630: Based on the cellular automata risk diffusion model, simulate the spread path and impact range of hidden dangers in three-dimensional space to generate a dynamic three-dimensional fire risk map.

[0068] Specifically, dynamic simulation is performed based on a cellular automata (CA) model. The BIM space is meshed into cells, each with a state (normal, fire, smoke, heat) and material properties. Based on the initial hazard location and type (e.g., "electrical fire"), and combined with data from the BIM such as material fire resistance ratings, spatial connectivity, and ventilation systems, iterative calculations are performed according to preset physical rules (e.g., heat conduction, flame spread, smoke diffusion rules) to simulate the fire's spread path and impact range within the next 5 and 10 minutes. Finally, all real-time identification results, attributes, and predicted ranges are rendered together to generate a dynamically updated 3D fire risk map, providing an intuitive global view for evacuation command and emergency decision-making.

[0069] The following describes the construction fire protection problem identification system based on multimodal AI provided by the present invention. The construction fire protection problem identification system based on multimodal AI described below can be referred to in correspondence with the construction fire protection problem identification method based on multimodal AI described above.

[0070] In one embodiment, a construction fire protection problem identification system based on multimodal AI includes a data flow alignment module, a feature extraction module, a dynamic fusion module, a multi-task identification module, a post-processing module, and a map generation module. The data stream alignment module is used to perform cross-modal spatiotemporal alignment and modal missing compensation processing on visible light images, thermal images and construction text records at the construction site, generating a standardized multimodal data stream; The feature extraction module is used to perform targeted feature extraction on different modal data in the standardized multimodal data stream to obtain geometric semantic features of visible light images, frequency domain features of abnormal heat sources in thermal images, and entity relationship features in text records. The dynamic fusion module is used to input the geometric semantic features, abnormal heat source frequency domain features and entity relationship features into an attention mechanism-based dynamic fusion module, and generate a deeply correlated unified multimodal feature tensor through cross-modal feature interaction and adaptive weight allocation. The multi-task recognition module is used to input the unified multimodal feature tensor into a multi-task recognition network that can decode in parallel, thereby decoding to obtain pixel-level hazard area segmentation map, hazard type classification and risk level quantification value; The post-processing module is used to perform post-processing optimization on the hazard area segmentation map to generate a high-precision hazard area mask and a corresponding visual interpretation heat map. The map generation module is used to semantically map and dynamically bind the hazard area mask, hazard type, risk level, and visual interpretation heat map with the components in the building information model, and perform situational inference based on the risk diffusion model to generate a dynamic three-dimensional fire risk map that integrates real-time identification results, risk prediction, and hazard visual interpretation.

[0071] Figure 3 This example illustrates a schematic diagram of the physical structure of an electronic device, which can be a smart terminal. Its internal structure diagram can be as follows: Figure 3 As shown, this electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal AI-based method for identifying construction fire safety issues.

[0072] Those skilled in the art will understand that Figure 3The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the electronic device to which the present invention is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0073] On the other hand, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, implements a construction fire protection problem identification method based on multimodal AI.

[0074] On another front, a computer program product or computer program is provided, comprising computer instructions stored in a computer storage medium. A processor of an electronic device reads the computer instructions from the computer storage medium, and when the processor executes the computer instructions, it implements a construction fire safety problem identification method based on multimodal AI.

[0075] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0076] By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0077] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0078] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A method for identifying fire safety issues in construction projects based on multimodal AI, characterized in that, The method includes: Cross-modal spatiotemporal alignment and modal missing compensation processing are performed on visible light images, thermal images, and construction text records at the construction site to generate a standardized multimodal data stream; Targeted feature extraction is performed on different modal data in the standardized multimodal data stream to obtain geometric semantic features of visible light images, frequency domain features of abnormal heat sources in thermal images, and entity relationship features in text records. The geometric semantic features, abnormal heat source frequency domain features, and entity relationship features are input into a dynamic fusion module based on an attention mechanism. Through cross-modal feature interaction and adaptive weight allocation, a deeply correlated unified multimodal feature tensor is generated. The unified multimodal feature tensor is input into a multi-task recognition network capable of parallel decoding, thereby decoding to obtain pixel-level hazard area segmentation map, hazard type classification and risk level quantification value; The hazard area segmentation map is post-processed and optimized to generate a high-precision hazard area mask and a corresponding visual interpretation heatmap; The hidden danger area mask, hidden danger type, risk level, and the visual interpretation heat map are semantically mapped and dynamically bound to the components in the building information model. Based on the risk diffusion model, situation inference is performed to generate a dynamic three-dimensional fire risk map that integrates real-time identification results, risk prediction and hidden danger visual interpretation.

2. The construction fire safety problem identification method based on multimodal AI according to claim 1, characterized in that, The process of performing cross-modal spatiotemporal alignment and modal missing compensation processing on visible light images, thermal images, and construction text records at the construction site to generate a standardized multimodal data stream includes: Feature point matching based on scale-invariant feature transformation is performed on visible light images and thermal images to achieve spatial alignment, and time synchronization is achieved through timestamp interpolation to establish a unified spatiotemporal coordinate system. A text parser based on regular expressions is used to extract key time and location information from the construction text and automatically associate and match it with the spatiotemporal coordinate system. The missing modal fragments after alignment are identified by a modal missing detector, and a pre-trained conditional generative adversarial network model is invoked to generate synthetic data of the missing modal based on the existing modal data. The complete multimodal data stream is Z-score normalized and then encapsulated into a normalized multimodal data stream according to the time series.

3. The construction fire safety problem identification method based on multimodal AI according to claim 2, characterized in that, The targeted feature extraction of different modal data in the standardized multimodal data stream yields geometric semantic features of visible light images, frequency domain features of abnormal heat sources in thermal images, and entity relationship features in text records, including: The visible light image is input into a pre-trained masked region convolutional neural network to extract pixel-level geometric semantic features containing object boundaries and positional relationships. Short-time Fourier transform was performed on the thermal imaging sequence to extract high-frequency energy spectrum features of temperature changes, and frequency domain feature enhancement was performed by combining the historical anomalous heat source database. We employ a bidirectional encoder-based representation and a bidirectional long short-term memory conditional random field model to perform named entity recognition and relation extraction on construction text, thereby constructing a structured entity relation feature map.

4. The construction fire safety problem identification method based on multimodal AI according to claim 3, characterized in that, The geometric semantic features, abnormal heat source frequency domain features, and entity relationship features are input into a dynamic fusion module based on an attention mechanism. Through cross-modal feature interaction and adaptive weight allocation, a deeply correlated unified multimodal feature tensor is generated, including: Construct a cross-modal multi-head attention layer, calculate the correlation matrix between features of different modalities, and realize the interaction between features; Learnable modal weight parameters are introduced, and weights are assigned through a normalized exponential function to concatenate the weighted features. The concatenated features are input into a fully connected layer for dimensionality reduction and nonlinear enhancement, generating a unified multimodal feature tensor.

5. The construction fire safety problem identification method based on multimodal AI according to claim 4, characterized in that, The step of inputting the unified multimodal feature tensor into a multi-task recognition network capable of parallel decoding, thereby decoding to obtain pixel-level hazard region segmentation maps, hazard type classifications, and risk level quantification values, includes: The unified multimodal feature tensor is input into the shared encoder based on the residual network to extract deep shared features; Use a U-shaped network decoder to output a hazard area segmentation map; Use a fully connected classifier to output a hazard type classification; Use a regression network to output a quantitative value for the risk level.

6. The construction fire safety problem identification method based on multimodal AI according to claim 5, characterized in that, The post-processing optimization of the hazard area segmentation map to generate a high-precision hazard area mask and a corresponding visual interpretation heatmap includes: Morphological closing operations are performed on the segmentation map of the potential hazard area to eliminate small voids, and adjacent areas are merged through connected component analysis. The optimized segmentation image is binarized and thresholded to generate a high-precision mask for potential hazard areas. Based on the gradient-weighted class activation mapping method, a visually interpretable heatmap is generated using gradient information in a multi-task recognition network.

7. The construction fire safety problem identification method based on multimodal AI according to claim 6, characterized in that, The hidden danger area mask, hidden danger type, risk level, and the visual interpretation heat map are semantically mapped and dynamically bound to the components in the building information model. Based on the risk diffusion model, situational simulation is performed to generate a dynamic three-dimensional fire risk map that integrates real-time identification results, risk prediction, and visual interpretation of hidden dangers, including: The hazard area mask is mapped onto the building information model through a coordinate transformation matrix, and then matched with the corresponding components for identification and semantic association. The hazard type, risk level, and heat map are dynamically attached to the building information model component object; Based on the cellular automata risk diffusion model, the spread path and impact range of hidden dangers in three-dimensional space are simulated to generate a dynamic three-dimensional fire risk map.

8. A construction fire safety problem identification system based on multimodal AI, characterized in that, The device includes: The data stream alignment module is used to perform cross-modal spatiotemporal alignment and modal missing compensation processing on visible light images, thermal images and construction text records at the construction site, and generate standardized multimodal data streams; The feature extraction module is used to perform targeted feature extraction on different modal data in the standardized multimodal data stream to obtain geometric semantic features of visible light images, frequency domain features of abnormal heat sources in thermal images, and entity relationship features in text records. The dynamic fusion module is used to input the geometric semantic features, abnormal heat source frequency domain features and entity relationship features into an attention mechanism-based dynamic fusion module, and generate a deeply associated unified multimodal feature tensor through cross-modal feature interaction and adaptive weight allocation. The multi-task recognition module is used to input the unified multimodal feature tensor into a multi-task recognition network that can decode in parallel, thereby decoding to obtain pixel-level hazard area segmentation map, hazard type classification and risk level quantification value; The post-processing module is used to perform post-processing optimization on the hazard area segmentation map to generate a high-precision hazard area mask and a corresponding visual interpretation heat map. The map generation module is used to semantically map and dynamically bind the hazard area mask, hazard type, risk level, and visual interpretation heat map with the components in the building information model, and perform situational inference based on the risk diffusion model to generate a dynamic three-dimensional fire risk map that integrates real-time identification results, risk prediction, and visual interpretation of hazards.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images