A fire hazard identification method for hot work based on multi-modal feature fusion

Abstract

The present application provides a kind of fire operation fire hazard identification method based on multi-modal feature fusion, it is related to the technical field of fire hazard identification.Acquire the multi-modal data of fire operation including thermal infrared and visible light image data;Based on visible light image data, extract color feature and the motion feature of splash, determine real splash target, extract real motion feature;Based on thermal infrared image data, extract the temperature feature of splash in fire operation;The motion feature and temperature feature of splash in fire operation are fused, and the fusion target is obtained, the region where splash disappears is framed and reserved, and the region of continuous attention is obtained, smoke detection and thermal infrared temperature analysis are carried out in the region, finally the two results are fused to weight score, and the fire hazard identification result is output.The present application improves the accuracy and robustness of detection in complex environment, and is suitable for real-time safety monitoring of fire operation scene such as high-rise building facade.

Description

Technical Field

[0001] This invention relates to the technical field of fire hazard identification, and more specifically, to a method for identifying fire hazards during hot work operations based on multimodal feature fusion. Background Technology

[0002] Hot work is a special type of operation with a high risk of fire, commonly seen in industrial production. Sparks, residues, and high temperatures generated during the process can easily ignite surrounding combustibles, causing fires, injuries, and property damage. Hot work environments are typically complex and variable; therefore, timely detection and early warning of fire hazards are crucial for ensuring safe production.

[0003] Currently, the mainstream technologies applied to hot work safety monitoring mainly fall into two categories: one is the traditional monitoring method that relies on manual inspection or fixed smoke and temperature sensors, which has a slow response, high false alarm rate, and is difficult to adapt to complex working environments; the other is intelligent monitoring methods based on computer vision and infrared thermal imaging, especially the multimodal fusion method that utilizes visible light and thermal infrared images.

[0004] However, existing monitoring methods based on multimodal fusion still have significant limitations: First, in complex scenarios involving dynamics, occlusion, or drastic changes in illumination, feature alignment between visible light and thermal infrared images still faces problems of insufficient accuracy and high computational cost; Second, existing multimodal information fusion strategies are relatively simple and fail to fully explore the deep correlations between different modes, resulting in insufficient sensitivity and accuracy in early hazard identification; Third, existing methods have weak trajectory prediction and continuous tracking capabilities for transient high-temperature targets such as splashing sparks and molten slag, making it difficult to achieve continuous and precise monitoring of hazard areas. Summary of the Invention

[0005] To address the low accuracy of existing methods in identifying fire hazards, this invention proposes a fire hazard identification method for hot work operations based on multimodal feature fusion. This method enables reliable identification of flying sparks, welding slag, and smoke, as well as early warning of fire hazards, thus improving the accuracy of fire hazard identification in complex environments.

[0006] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows: A method for identifying fire hazards during hot work operations based on multimodal feature fusion includes the following steps: Acquire multimodal data of hot work operations; the multimodal data includes thermal infrared image data and visible light image data; Based on the visible light image data, color features and motion features of splashes during the hot work are extracted; Based on color features and the motion characteristics of splashes during the hot work, the real splash targets are determined, and the real motion characteristics of the splashes are extracted based on the real splash targets. Based on the thermal infrared image data, the temperature characteristics of the splashes during the hot work were extracted; The motion and temperature characteristics of the splashes during the hot work are fused to obtain a fused target. Based on the fused target, the area where the splashes disappear is selected and retained to obtain the area of ​​continuous interest. Within the area of ​​continuous attention, smoke detection and thermal infrared temperature processing are performed respectively to obtain smoke detection results and thermal infrared temperature processing results. The smoke detection results and temperature data processing results are combined and weighted to determine the fire hazard identification results.

[0007] Preferably, the process of extracting color features and motion features of splashes during the hot work is as follows: HSV thresholding is performed on the visible light RGB image frames in the visible light image data, and the bright areas are extracted as color feature detection results; Extract stable feature points from the visible light grayscale image in the visible light image data, and obtain the motion feature detection results of the splash based on the stable feature points; Based on color features and the motion characteristics of splashes during the hot work, the actual splash targets are determined. Based on the actual splash targets, the actual motion characteristics of the splashes are extracted. The process is as follows: Based on the cross-union criterion, the color feature detection results and motion feature detection results are comprehensively judged to determine the real splash target; By associating the color feature detection results with the motion feature detection results to the corresponding real splash target, the real motion features of the splash are obtained.

[0008] Preferably, the process of extracting the highlight region is as follows: converting the RGB image frame to the HSV color space, using an automatic parameter-finding mechanism based on the YOLO model to determine the optimal HSV threshold range, generating multiple candidate HSV threshold combinations in a preset HSV threshold grid, inputting the binary mask generated by each HSV threshold combination into a pre-trained YOLO model, and calculating a comprehensive score. Overall score The expression is:

[0009] in, This represents the sum of confidence scores for the bounding boxes detected by the YOLO model. This represents the contour statistical characteristics of the mapped splashes. This represents the ratio of the effective area of ​​the binary mask to the total image area. The effective area of ​​the binary mask is the connected region in the binary mask with a pixel value of 255. and All represent adjustment parameters; The HSV threshold combination with the highest comprehensive score is selected as the optimal threshold; the optimal threshold is applied to segment the RGB image frame to generate a color binary mask; Based on the color binary mask, the bright areas are extracted and used as the color feature detection results.

[0010] Preferably, the process of obtaining the motion feature detection results of the splashes is as follows: Calculate the gradient covariance matrix in the neighborhood of each pixel in the visible light grayscale image, and use the minimum eigenvalue of the gradient covariance matrix as the corner score. When the corner score is greater than the preset detection threshold, select the corresponding pixel as a stable feature point. The motion information of the stable feature points is obtained by using the sparse optical flow method. Based on the motion information, the points are filtered, compensated and integrated to obtain the detection results of the splash motion features.

[0011] Preferably, the highlighted area is selected to obtain a color feature detection box, and a motion feature detection box is obtained based on the motion feature detection result of the splash. Let the set of color feature detection boxes be denoted as . The set of motion feature detection boxes is For each pair of detection boxes, the cross-union ratio (CUR) is calculated. The expression for the CUR is:

[0012] in, This represents the area of ​​the overlapping region between the two types of detection boxes. This represents the area of ​​the union region of the two types of detection boxes; When the cross-union ratio of the detection frame is greater than the preset decision threshold, it is determined to be a real splash target; otherwise, it is regarded as noise or pseudo-motion area and is removed.

[0013] Preferably, the process of extracting the temperature characteristics of hot work splashes is as follows: Read the thermal infrared temperature data synchronized with the thermal infrared image data, obtain the thermal infrared video frames in the thermal infrared image data, and establish a mapping between the thermal infrared video frames and the temperature matrix. ,in, Indicates the frame number. Indicates the first Temperature matrix corresponding to the frame, matrix elements Indicates pixel position Temperature value at that location; Set a fixed temperature threshold The temperature matrix is ​​binarized to generate a temperature binary mask. The expression for the temperature binary mask is as follows: ; for the generated temperature binary mask Perform connected component analysis to extract a set of connected regions, and obtain the bounding box of the high-temperature region based on the set of connected regions.

[0014] Preferably, the process of obtaining the region of continuous interest is as follows: Based on the actual motion characteristics of the splashing object, the bounding box of the moving target is obtained; the overlap between the bounding box of the moving target and the bounding box of the high-temperature region is determined; and the intersection-overlap ratio (IoU) of the bounding boxes is used as the overlap metric. The expression for the IoU is:

[0015] in, Represents the bounding box of the moving target. Indicates the boundary box of the high-temperature region; For the bounding boxes of moving targets and high-temperature regions that meet the preset spatial overlap conditions, a feature concatenation method is used to generate fused targets. Each fused target includes bounding box coordinates, maximum temperature and average temperature. Temperature confidence factor calculated based on highest and average temperatures. The expression is:

[0016] in, Indicates the highest temperature. Indicates average temperature; For each fused target, a parabolic trajectory processing method is used to predict its motion trajectory; based on the motion trajectory, it is determined whether the fused target has disappeared. If it has disappeared, the bounding box coordinates of the frame before its disappearance are recorded. The expression for the bounding box coordinates is: , bounding box The corresponding areas are marked as areas of continued interest.

[0017] Preferably, the smoke detection process is as follows: HSV threshold segmentation is performed on the visible light image data to extract the color feature detection results of the smoke; the motion feature detection results of smoke diffusion are obtained using the sparse optical flow method; based on the color feature detection results and the motion feature detection results of smoke diffusion, smoke color detection boxes and motion detection boxes are obtained; the cross-union ratio (CUI) of the smoke is calculated for each pair of smoke color detection boxes and motion detection boxes, and the presence of smoke is comprehensively determined to generate a smoke detection result. , , where 1 indicates that smoke was detected and 0 indicates that smoke was not detected.

[0018] Preferably, the process of performing thermal infrared temperature processing is as follows: based on the set of continuously monitored regions... , For each area of ​​continuous attention Extract its value from the temperature matrix. Temperature sub-block at the corresponding position And calculate the temperature confidence factor corresponding to this region. The expression is:

[0019] in, , These represent areas of continued attention. The highest and average temperatures, This indicates the significance of the thermal anomaly in the region; Temperature sub-block Temperature data is processed for each pixel in the image to set a temperature threshold for fire hazards. and temperature rise rate threshold Determine whether the temperature value of each pixel exceeds the fire hazard temperature threshold. And calculate the temperature rise rate of each pixel. The expression for the rate of temperature rise is:

[0020] in, This indicates that the pixel is within a time interval. The amount of temperature change inside, Indicates the rate of temperature rise; When the temperature value of the pixel And the rate of temperature rise When this occurs, the pixel is determined to be a temperature anomaly point and recorded as... Otherwise, record as .

[0021] Preferably, the smoke detection results and temperature data processing results are fused and weighted for scoring, the process of which is as follows: Calculate the overall score for fire hazards The expression is:

[0022] in, , These represent the preset smoke detection weights and temperature detection weights, respectively, satisfying the following conditions: ; This indicates the total number of pixels with abnormal temperatures within the monitored area. This represents the total number of pixels in the area of ​​continuous focus; when If the value is greater than or equal to the preset value, the hot work is deemed to pose a fire hazard; otherwise, the hot work is deemed not to pose a fire hazard.

[0023] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: This invention proposes a fire hazard identification method for hot work operations based on multimodal feature fusion. For hot work scenarios, it analyzes the color features of visible light image data, the motion characteristics of spatter, and the temperature features of thermal infrared images. Based on the color features and the motion characteristics of spatter during hot work, it identifies the true spatter target, avoiding misjudgments caused by environmental interference in traditional single-sensor systems. Furthermore, by selecting and retaining areas where spatter disappears, it obtains areas of continuous interest, and continues smoke detection and temperature monitoring within these areas, extending the risk monitoring window. This allows for early warning by continuously monitoring secondary ignition signs in these areas before open flames occur. This invention achieves reliable identification and early fire hazard warning of spatter, welding slag, and smoke, improving the accuracy and robustness of detection in complex environments. It is suitable for real-time safety monitoring of hot work scenarios such as the facades of high-rise or super high-rise buildings. Attached Figure Description

[0024] Figure 1 The flowchart illustrates a method for identifying fire hazards during hot work operations based on multimodal feature fusion, as proposed in this embodiment of the invention. Figure 2 A first schematic diagram illustrating the fire hazard identification results proposed in this embodiment of the invention; Figure 3 A second schematic diagram illustrating the fire hazard identification results proposed in this embodiment of the invention; Figure 4 This is a third schematic diagram illustrating the fire hazard identification results proposed in this embodiment of the invention. Detailed Implementation

[0025] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts of the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions; It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings.

[0026] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] The positional relationships depicted in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. Example 1 This embodiment proposes a method for identifying fire hazards during hot work operations based on multimodal feature fusion. (See [link to relevant documentation]). Figure 1 This includes the following steps: S1: Acquire multimodal data of hot work operations; multimodal data includes thermal infrared image data and visible light image data; S2: Based on visible light image data, extract color features and motion characteristics of splashes during hot work operations; S3: Based on color features and the motion characteristics of splashes during hot work, identify real splash targets and extract real motion characteristics of splashes based on real splash targets; S4: Extract the temperature characteristics of splashes during hot work based on thermal infrared image data; S5: The motion and temperature characteristics of splashes during hot work are fused to obtain a fused target. Based on the fused target, the area where the splashes disappear is selected and preserved to obtain the area of ​​continuous interest. S6: Within the continuously monitored area, smoke detection and thermal infrared temperature processing are performed respectively to obtain smoke detection results and thermal infrared temperature processing results. S7: The smoke detection results and temperature data processing results are fused and weighted to determine the fire hazard identification results.

[0028] In this embodiment, the thermal infrared image data and visible light image data in the multimodal data are extracted from visible light video and thermal infrared video after spatiotemporal alignment processing. For hot work scenarios, the color features of the visible light image data are extracted and analyzed to locate bright areas. Static interference is then eliminated by combining the motion characteristics of the splashes, and finally, temperature features are introduced to confirm their high-temperature properties. This effectively distinguishes real splashes from reflective, cold, moving objects and other interference, significantly improving the accuracy of target detection. Only when a target possesses both high-temperature properties and a motion trajectory is it determined to be a real splash, avoiding misjudgments caused by environmental interference from traditional single sensors. By fusing motion trajectories and high-temperature areas, dangerous splashes are accurately located. Furthermore, a continuous monitoring area is established to periodically and precisely monitor areas that may ignite after the splashes disappear, extending the risk monitoring window and enabling early warnings before open flames appear. The fusion of motion features overcomes the problem of inaccurate temperature capture of rapidly moving targets by thermal infrared images, while temperature features compensate for the reduced recognition capability of visible light under smoke obscuration.

[0029] Example 2 This embodiment proposes a method for identifying fire hazards during hot work operations based on multimodal feature fusion. It extracts color features and the motion features of splashes during hot work operations. The process is as follows: HSV thresholding is performed on visible light RGB image frames in visible light image data, and the bright areas are extracted as color feature detection results. Stable feature points are extracted from the visible light grayscale image data, and the motion feature detection results of the splash are obtained based on the stable feature points. Based on color features and the motion characteristics of splashes during hot work, the actual splash targets are identified. Based on the actual splash targets, the actual motion characteristics of the splashes are extracted. The process is as follows: Based on the cross-union criterion, the color feature detection results and motion feature detection results are comprehensively judged to determine the real splash target; By associating the color feature detection results with the motion feature detection results to the corresponding real splash target, the real motion features of the splash are obtained.

[0030] The process of extracting highlight regions is as follows: RGB image frames are converted to the HSV color space; an automatic parameter-finding mechanism based on the YOLO model is used to determine the optimal HSV threshold range; multiple candidate HSV threshold combinations are generated in a preset HSV threshold grid; the binary mask generated by each HSV threshold combination is input into a pre-trained YOLO model, and a comprehensive score is calculated. Overall score The expression is:

[0031] in, This represents the sum of confidence scores for the bounding boxes detected by the YOLO model. This represents the contour statistical characteristics of the mapped splashes. This represents the ratio of the effective area of ​​the binary mask to the total image area. The effective area of ​​the binary mask is the connected region in the binary mask with a pixel value of 255. and All represent adjustment parameters; The HSV threshold combination with the highest comprehensive score is selected as the optimal threshold; the optimal threshold is applied to segment the RGB image frame to generate a color binary mask; Based on a color binary mask, the bright areas are extracted and used as the color feature detection results.

[0032] In this embodiment, RGB image frames are converted to the HSV color space, and the hue, saturation, and lightness components are used to effectively characterize the radiation characteristics at different temperatures. The expression for the HSV color space is:

[0033]

[0034]

[0035] in, , These represent the maximum and minimum values ​​in the RGB channels, respectively. Represents the hue angle, with a value range of [value missing]. ; Indicates saturation; This indicates the brightness value.

[0036] In overall score In the expression, Used to measure the semantic consistency between the binary mask and the YOLO model; The outline statistical features of the mapped spark; This indicates the ratio of the effective mask area to the total image area, used to control the rationality of the segmented regions; and This represents a balancing factor that adjusts the relative contributions of each component in the function.

[0037] By iterating through all HSV threshold combinations, the HSV threshold combination with the highest score is selected as the optimal threshold. The optimal HSV color threshold is then applied to segment the image, generating a binary color mask. In the color binary mask, all positions marked as 1 correspond to bright areas that meet the color and brightness requirements. The pixels at these 1 positions in the original image are retained, while the background pixels at the 0 positions are filtered out, thus completing the extraction of the bright areas. The bright areas are used as the detection results of color features.

[0038] In an optional embodiment, the process of obtaining the motion feature detection results of the splash is as follows: Calculate the gradient covariance matrix in the neighborhood of each pixel in the visible light grayscale image, and use the minimum eigenvalue of the gradient covariance matrix as the corner score. When the corner score is greater than the preset detection threshold, select the corresponding pixel as a stable feature point. The motion information of stable feature points is obtained by using the sparse optical flow method. Based on the motion information, the results of splash motion feature detection are obtained by filtering, compensation and integration.

[0039] In this embodiment, the Shi-Tomasi corner detection algorithm (GFTT) is used to extract stable feature points in the visible light grayscale image, and the gradient covariance matrix in the pixel neighborhood is calculated. The expression for the gradient covariance matrix is ​​as follows:

[0040] in, , These represent the gradients of image brightness in the horizontal and vertical directions, respectively.

[0041] The smallest eigenvalue of the gradient covariance matrix The scoring criterion for corner points is expressed as follows:

[0042] Set the detection threshold to ,when If the pixel has sufficient texture and orientation gradient, it is considered to be selected as a stable feature point.

[0043] The Lucas-Kanade sparse optical flow algorithm is used to estimate the positional changes of stable feature points between adjacent frames, assuming a local window. All pixels within a local window have the same motion speed. The optimal displacement vector is solved by minimizing the sum of squared brightness residuals within the local window. The expression is:

[0044] in, , These represent the gradients of image brightness in the horizontal and vertical directions, respectively. This represents the rate of change of image brightness over time.

[0045] The optimal displacement vectors calculated by the Lucas-Kanade sparse optical flow algorithm contain the velocity (amplitude) and direction of motion of the feature point. The optimal displacement vectors are the most original motion measurement data obtained from continuous video frames. These displacement vectors contain two parts of information: one is the real motion of the spatter (such as sparks and welding slag) itself; the other is the false global motion caused by factors such as camera shake.

[0046] To eliminate camera shake interference, an affine transformation compensation model based on RANSAC is introduced, which matches feature point pairs between adjacent frames. Estimate the global affine transformation matrix, expressed as:

[0047]

[0048] in, It is a two-dimensional partial affine matrix; , These represent the coordinates of the stable feature point in the previous frame and the current frame, respectively.

[0049] Then, by minimizing the reprojection error, the expression is: .

[0050] In an optional embodiment, a color feature detection box is obtained by selecting the highlighted area, and a motion feature detection box is obtained based on the motion feature detection results of the splash. Let the set of color feature detection boxes be denoted as . The set of motion feature detection boxes is For each pair of detection boxes, the cross-union ratio (CUR) is calculated. The expression for the CUR is:

[0051] in, This represents the area of ​​the overlapping region between the two types of detection boxes. This represents the area of ​​the union region of the two types of detection boxes; When the cross-union ratio of the detection frame is greater than the preset decision threshold, it is determined to be a real splash target; otherwise, it is regarded as noise or pseudo-motion area and is removed.

[0052] In this embodiment, by using cross-union matching of detection boxes, the motion attributes (velocity, trajectory) of the splash motion feature detection results are bound to a target consistent with the color detection result space, forming a target box with motion features. The decision threshold for fusion determination is set to 0.5. If the detection results of the two features are consistent in spatial location, that is, the area simultaneously possesses color and motion features, it is determined to be a real splash target; otherwise, it is considered noise or a pseudo-motion area and is discarded. Finally, the fused real motion features of the splash are obtained.

[0053] In an optional embodiment, the temperature characteristics of hot work splashes are extracted as follows: Read thermal infrared temperature data synchronized with thermal infrared image data, obtain thermal infrared video frames from the thermal infrared image data, and establish a mapping between thermal infrared video frames and temperature matrix. ,in, Indicates the frame number. Indicates the first Temperature matrix corresponding to the frame, matrix elements Indicates pixel position Temperature value at that location; Set a fixed temperature threshold The temperature matrix is ​​binarized to generate a temperature binary mask. The expression for the temperature binary mask is as follows: ; for the generated temperature binary mask Connectivity analysis is performed to extract a set of connected regions, and the bounding box of the high-temperature region is obtained based on the set of connected regions.

[0054] In this embodiment, matrix elements The unit is ℃; threshold binarization processing is performed on the thermal infrared image data using a fixed temperature threshold. A temperature binary mask is generated at 100℃. ; for the generated temperature binary mask Perform connected component analysis to extract the set of connected components. Calculate the area of ​​each connected region. Maximum temperature Average temperature .

[0055] In one optional embodiment, the process of obtaining the region of continuous interest is as follows: The bounding box of the moving target is obtained based on the actual motion characteristics of the splashing object; the overlap between the bounding box of the moving target and the bounding box of the high-temperature region is determined; and the intersection-union ratio (IUU) of the bounding boxes is used as the overlap metric. The expression for the IUU is:

[0056] in, Represents the bounding box of the moving target. Indicates the boundary box of the high-temperature region; For the bounding boxes of moving targets and high-temperature regions that meet the preset spatial overlap conditions, a feature concatenation method is used to generate fused targets. Each fused target includes bounding box coordinates, maximum temperature and average temperature. Temperature confidence factor calculated based on highest and average temperatures. The expression is:

[0057] in, Indicates the highest temperature. Indicates average temperature; For each fused target, a parabolic trajectory processing method is used to predict its motion trajectory. Based on the motion trajectory, it is determined whether the fused target has disappeared. If it has disappeared, the bounding box coordinates of the frame before disappearance are recorded. The expression for the bounding box coordinates is: , bounding box The corresponding areas are marked as areas of continued interest.

[0058] In this embodiment, for those that satisfy the spatial overlap condition Moving targets and high-temperature regions are compared using a feature concatenation method to generate fused targets. Each fused target includes bounding box coordinates, motion velocity vector, maximum temperature, and average temperature. Temperature confidence factors are stored as attributes of the fused targets, with a reference value of 300℃ for the maximum temperature and 200℃ for the average temperature. This indicates the significance of the thermal anomaly of the fusion target, with a value range of [value range missing]. .

[0059] For each fused target, its trajectory is predicted using a parabolic trajectory processing method, and the parabolic equation of the moving target is fitted, expressed as:

[0060] in, , , The fitting parameters for the parabola are obtained by fitting the historical trajectory points of the target using the least squares method. The target's position in subsequent frames is predicted based on the parabolic trajectory. The target is considered to have disappeared when it is not detected in three consecutive frames. In the original video image, the area corresponding to the bounding box is marked as a region of continued interest, and this bounding box is retained for 10 seconds starting from the disappearance frame, even if no new target appears during this period. The region of continued interest's ID, appearance time, disappearance time, historical temperature data, and temperature confidence factor are recorded for subsequent fire hazard identification.

[0061] In an optional embodiment, the smoke detection process is as follows: HSV threshold segmentation is performed on the visible light image data to extract the color feature detection results of the smoke; the motion feature detection results of smoke diffusion are obtained using the sparse optical flow method; based on the color feature detection results and the motion feature detection results of smoke diffusion, smoke color detection boxes and motion detection boxes are obtained; the cross-union ratio (CUI) of the smoke is calculated for each pair of smoke color detection boxes and motion detection boxes, and the presence of smoke is comprehensively determined to generate a smoke detection result. , , where 1 indicates that smoke was detected and 0 indicates that smoke was not detected.

[0062] The process of performing thermal infrared temperature processing is as follows: based on the set of regions of continuous interest , For each area of ​​continuous attention Extract its value from the temperature matrix. Temperature sub-block at the corresponding position And calculate the temperature confidence factor corresponding to this region. The expression is:

[0063] in, , These represent areas of continued attention. The highest and average temperatures, This indicates the significance of the thermal anomaly in the region; Temperature sub-block Temperature data is processed for each pixel in the image to set a temperature threshold for fire hazards. and temperature rise rate threshold Determine whether the temperature value of each pixel exceeds the fire hazard temperature threshold. And calculate the temperature rise rate of each pixel. The expression for the rate of temperature rise is:

[0064] in, This indicates that the pixel is within a time interval. The amount of temperature change inside, Indicates the rate of temperature rise; When the temperature value of the pixel And the rate of temperature rise When this occurs, the pixel is determined to be a temperature anomaly point and recorded as... Otherwise, record as .

[0065] In this embodiment, the area of ​​continuous attention The units for the highest and average temperatures are °C. This indicates the significance of the thermal anomaly in the region, with a value range of [value range missing]. Set fire hazard temperature thresholds The value range is 200℃~300℃.

[0066] In an optional embodiment, the smoke detection results and temperature data processing results are fused and weighted for scoring, the process of which is as follows: Calculate the overall score for fire hazards The expression is:

[0067] in, , These represent the preset smoke detection weights and temperature detection weights, respectively, satisfying the following conditions: ; This indicates the total number of pixels with abnormal temperatures within the monitored area. This represents the total number of pixels in the area of ​​continuous focus; when If the value is greater than or equal to the preset value, the hot work is deemed to pose a fire hazard; otherwise, the hot work is deemed not to pose a fire hazard.

[0068] In this embodiment, the ratio of the total number of temperature-abnormal pixels within the continuously monitored area to the total number of pixels in the continuously monitored area reflects the proportion of temperature-abnormal pixels within the area. When If a fire hazard is detected, it is considered a fire hazard; otherwise, it is considered a non-fire hazard. When a fire hazard is detected, the area of ​​continuous concern is marked with a preset-color box in the output video, and smoke detection and temperature anomaly indicators are overlaid. The fire hazard detection result, location coordinates, highest temperature value, time of occurrence, and duration are recorded.

[0069] Example 3 This embodiment, based on Embodiment 1 or Embodiment 2, further describes the effectiveness of a fire hazard identification method for hot work operations based on multimodal feature fusion, combined with simulation experiments.

[0070] 1. Simulation conditions: The hardware environment for the simulation experiment is NVIDIA GeForce RTX 3070 Laptop GPU (8GB), AMD Ryzen 7 5800H with Radeon Graphics (3.20 GHz, 16.0 GB RAM) and Windows 10 operating system; the simulation software is PyCharm 2024.1.7 and Python 3.9.

[0071] 2. Experimental content: To demonstrate the effectiveness of a fire hazard identification method for hot work operations based on multimodal feature fusion, this embodiment uses three spatiotemporally aligned visible light and thermal infrared video images of burning flame-retardant building materials collected in the experiment for simulation experiments.

[0072] In the subjective evaluation of image simulation experiments: the effectiveness of fire hazard identification is judged by observing the annotation effect of the recognition result annotation map. Figure 2 , Figure 3 and Figure 4 The results of fire hazard identification were displayed, with multiple detection boxes covering the trajectory segments and discrete spark points of the sparks, clearly showing the distribution of the sparks.

[0073] In the objective index image simulation experiment: based on three spatiotemporally aligned visible light and thermal infrared video images of fire-retardant building materials, the precision, recall, and F1 score for fire hazard identification were calculated. Table 1 shows the fire hazard identification results of this method.

[0074] Table 1 Fire Hazard Identification Results

[0075] Simulation experiments show that the proposed method performs stably across three test videos, maintaining a high overall detection performance with an average accuracy of 89.15%, a recall rate of 82.14%, and an F1 score of 85.81%. Combined with subjective visual recognition results, this demonstrates that the method achieves good detection effectiveness for fire hazards.

[0076] The embodiments described are merely examples to clearly illustrate the present invention and are not intended to limit the implementation of the invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively describe all possible implementations. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for identifying fire hazards during hot work operations based on multimodal feature fusion, characterized in that, Includes the following steps: Acquire multimodal data of hot work operations; the multimodal data includes thermal infrared image data and visible light image data; Based on the visible light image data, color features and motion features of splashes during the hot work are extracted; Based on color features and the motion characteristics of splashes during the hot work, the real splash targets are determined, and the real motion characteristics of the splashes are extracted based on the real splash targets. Based on the thermal infrared image data, the temperature characteristics of the splashes during the hot work were extracted; The motion and temperature characteristics of the splashes during the hot work are fused to obtain a fused target. Based on the fused target, the area where the splashes disappear is selected and retained to obtain the area of ​​continuous interest. Within the area of ​​continuous attention, smoke detection and thermal infrared temperature processing are performed respectively to obtain smoke detection results and thermal infrared temperature processing results. The smoke detection results and temperature data processing results are combined and weighted to determine the fire hazard identification results.

2. The method for identifying fire hazards during hot work operations based on multimodal feature fusion according to claim 1, characterized in that, The process of extracting color features and motion features of splashes during hot work is as follows: HSV thresholding is performed on the visible light RGB image frames in the visible light image data, and the bright areas are extracted as color feature detection results; Extract stable feature points from the visible light grayscale image in the visible light image data, and obtain the motion feature detection results of the splash based on the stable feature points; Based on color features and the motion characteristics of splashes during the hot work, the actual splash targets are determined. Based on the actual splash targets, the actual motion characteristics of the splashes are extracted. The process is as follows: Based on the cross-union criterion, the color feature detection results and motion feature detection results are comprehensively judged to determine the real splash target; By associating the color feature detection results with the motion feature detection results to the corresponding real splash target, the real motion features of the splash are obtained.

3. The method for identifying fire hazards during hot work operations based on multimodal feature fusion according to claim 2, characterized in that, The process of extracting the highlight region is as follows: convert the RGB image frame to the HSV color space, use an automatic parameter finding mechanism based on the YOLO model to determine the optimal HSV threshold range, generate multiple sets of candidate HSV threshold combinations in the preset HSV threshold grid, input the binary mask generated by each set of HSV threshold combinations into the pre-trained YOLO model, and calculate the comprehensive score. Overall score The expression is: in, This represents the sum of confidence scores for the bounding boxes detected by the YOLO model. This represents the contour statistical characteristics of the mapped splashes. This represents the ratio of the effective area of ​​the binary mask to the total image area. The effective area of ​​the binary mask is the connected region in the binary mask with a pixel value of 255. and All represent adjustment parameters; The HSV threshold combination with the highest comprehensive score is selected as the optimal threshold; the optimal threshold is applied to segment the RGB image frame to generate a color binary mask; Based on the color binary mask, the bright areas are extracted and used as the color feature detection results.

4. The method for identifying fire hazards during hot work operations based on multimodal feature fusion according to claim 2, characterized in that, The process of obtaining the motion feature detection results of the splashes is as follows: Calculate the gradient covariance matrix in the neighborhood of each pixel in the visible light grayscale image, and use the minimum eigenvalue of the gradient covariance matrix as the corner score. When the corner score is greater than the preset detection threshold, select the corresponding pixel as a stable feature point. The motion information of the stable feature points is obtained by using the sparse optical flow method. Based on the motion information, the points are filtered, compensated and integrated to obtain the detection results of the splash motion features.

5. The method for identifying fire hazards during hot work operations based on multimodal feature fusion according to claim 2, characterized in that, The highlighted area is selected to obtain a color feature detection box, and the motion feature detection box is obtained based on the motion feature detection results of the splash. Let the set of color feature detection boxes be denoted as . The set of motion feature detection boxes is For each pair of detection boxes, the cross-union ratio (CUR) is calculated. The expression for the CUR is: in, This represents the area of ​​the overlapping region between the two types of detection boxes. This represents the area of ​​the union region of the two types of detection boxes; When the cross-union ratio of the detection frame is greater than the preset decision threshold, it is determined to be a real splash target; otherwise, it is regarded as noise or pseudo-motion area and is removed.

6. The method for identifying fire hazards during hot work operations based on multimodal feature fusion according to claim 1, characterized in that, The process of extracting the temperature characteristics of splashes during hot work is as follows: Read the thermal infrared temperature data synchronized with the thermal infrared image data, obtain the thermal infrared video frames in the thermal infrared image data, and establish a mapping between the thermal infrared video frames and the temperature matrix. ,in, Indicates the frame number. Indicates the first Temperature matrix corresponding to the frame, matrix elements Indicates pixel position Temperature value at that location; Set a fixed temperature threshold The temperature matrix is ​​binarized to generate a temperature binary mask. The expression for the temperature binary mask is as follows: ; for the generated temperature binary mask Perform connected component analysis to extract a set of connected regions, and obtain the bounding box of the high-temperature region based on the set of connected regions.

7. The method for identifying fire hazards during hot work operations based on multimodal feature fusion according to claim 6, characterized in that, The process of obtaining the region of continuous interest is as follows: Based on the actual motion characteristics of the splashing object, the bounding box of the moving target is obtained; the overlap between the bounding box of the moving target and the bounding box of the high-temperature region is determined; and the intersection-overlap ratio (IoU) of the bounding boxes is used as the overlap metric. The expression for the IoU is: in, Represents the bounding box of the moving target. Indicates the boundary box of the high-temperature region; For the bounding boxes of moving targets and high-temperature regions that meet the preset spatial overlap conditions, a feature concatenation method is used to generate fused targets. Each fused target includes bounding box coordinates, maximum temperature and average temperature. Temperature confidence factor calculated based on highest and average temperatures. The expression is: in, Indicates the highest temperature. Indicates average temperature; For each fused target, a parabolic trajectory processing method is used to predict its motion trajectory; based on the motion trajectory, it is determined whether the fused target has disappeared. If it has disappeared, the bounding box coordinates of the frame before its disappearance are recorded. The expression for the bounding box coordinates is: , bounding box The corresponding areas are marked as areas of continued interest.

8. The method for identifying fire hazards during hot work operations based on multimodal feature fusion according to claim 4, characterized in that, The process of smoke detection is as follows: HSV threshold segmentation is performed on the visible light image data, the color feature detection results of the smoke are extracted, and the motion feature detection results of the smoke diffusion are obtained by using the sparse optical flow method; based on the color feature detection results of the smoke and the motion feature detection results of the smoke diffusion, the smoke color detection box and the motion detection box are obtained. For each pair of smoke color detection boxes and motion detection boxes, the cross-union ratio (CUI) of the smoke is calculated, and the presence of smoke is determined comprehensively to generate a smoke detection result. , , where 1 indicates that smoke was detected and 0 indicates that smoke was not detected.

9. A method for identifying fire hazards during hot work operations based on multimodal feature fusion according to claim 7, characterized in that, The process of performing thermal infrared temperature processing is as follows: based on the set of continuously monitored regions... , For each area of ​​continuous attention Extract its value from the temperature matrix. Temperature sub-block at the corresponding position And calculate the temperature confidence factor corresponding to this region. The expression is: in, , These represent areas of continued attention. The highest and average temperatures, This indicates the significance of the thermal anomaly in the region; Temperature sub-block Temperature data is processed for each pixel in the image to set a temperature threshold for fire hazards. and temperature rise rate threshold Determine whether the temperature value of each pixel exceeds the fire hazard temperature threshold. And calculate the temperature rise rate of each pixel. The expression for the rate of temperature rise is: in, This indicates that the pixel is within a time interval. The amount of temperature change inside, Indicates the rate of temperature rise; When the temperature value of the pixel And the rate of temperature rise When this occurs, the pixel is determined to be a temperature anomaly point and recorded as... Otherwise, record as .

10. A method for identifying fire hazards during hot work operations based on multimodal feature fusion according to claim 1, characterized in that, The smoke detection results and temperature data processing results are fused and weighted for scoring. The process is as follows: Calculate the overall score for fire hazards The expression is: in, , These represent the preset smoke detection weights and temperature detection weights, respectively, satisfying the following conditions: ; This indicates the total number of pixels with abnormal temperatures within the monitored area. This represents the total number of pixels in the area of ​​continuous focus; when If the value is greater than or equal to the preset value, the hot work is deemed to pose a fire hazard; otherwise, the hot work is deemed not to pose a fire hazard.

Images