Internet-of-things remote inspection system based on multi-mode fire-fighting sensing data

Abstract

The invention belongs to the technical field of remote control, and particularly relates to an internet-of-things remote inspection system based on multi-modal fire-fighting sensing data, which comprises a multi-modal data acquisition module, a judgment module and a fire behavior trend prediction module, and is characterized in that the multi-modal data acquisition module is used for acquiring an RGB three-dimensional image, a smoke concentration matrix and a thermal value matrix of a monitoring area in real time; the judgment module carries out feature fusion on the multiple data sources and then judges whether a fire exists, and the fire behavior trend prediction module predicts the space change trend of flames and the boundary change trend of smoke and then carries out space fusion on the space change trend and the boundary change direction of the smoke to generate a fire behavior trend chart. Compared with the judgment that only the change trend of the simple flame area is used as the development trend of the fire, the judgment on the spreading trend of the fire is quicker, and the accuracy is higher.

Description

Technical Field

[0001] The present invention belongs to the field of remote control technology, and in particular relates to an Internet of Things remote inspection system based on multimodal fire sensor data. Background Art

[0002] Multimodal Deep Learning is a subfield of artificial intelligence that focuses on developing models that can simultaneously process and learn multiple types of data. These data types, or modalities, can include text, images, audio, video, and sensor data. With the development and changes of artificial intelligence, multimodal deep learning technology has been widely used in fire inspections. By fusing data from multiple sources and types at the same inspected location and analyzing the characteristics of data from different sources and types, a corresponding modal dataset is constructed. This multi-dimensional data-based modal dataset is used to extract features from the multimodal data set to construct a fire trend prediction model to predict the development trend of the fire. However, the fire prediction model in the existing scheme only focuses on the spatiotemporal variation patterns of the flame area, and does not fully couple the impact of environmental factors on combustion behavior, resulting in the fire prediction model's lack of generalization for fire occurrence trends.

[0003] Specifically, wind is a key driving factor for the spread of fire. Due to the airflow transport effect, the wind speed vector directly changes the flame inclination angle and the direction of heat convection, causing the flame front to shift to the windward or crosswind side at a nonlinear rate. In addition, the increase in wind speed will increase the oxygen concentration gradient in the combustion area, accelerate the chemical reaction rate, and cause the flame height and radiation intensity to grow exponentially, which will cause the fire prediction model to have a lag in prediction. This lag directly weakens the timeliness of the emergency decision-making system and may lead to secondary risks such as delays in resource allocation and failure of safety isolation zone settings. Summary of the Invention

[0004] In order to solve the above-mentioned problems existing in the prior art, the present invention provides an Internet of Things remote inspection system based on multimodal fire sensor data, which solves the problem that the prior art lacks dynamic correlation between the development of the fire and changes in the wind environment, resulting in the inability to accurately assess the development trend of the fire.

[0005] The purpose of the present invention can be achieved through the following technical solutions: an Internet of Things remote inspection system based on multimodal fire sensor data, comprising: Multimodal data acquisition module: used to obtain RGB three-dimensional images, smoke concentration matrix, and thermal value matrix of the monitoring area in real time; A judgment module is used to perform feature fusion on the RGB three-dimensional image and the thermal value matrix using a feature fusion algorithm to generate fusion features, and then input the generated fusion features into a preset judgment model to output a judgment result to determine whether there is a fire in the layout point area; Computing module: Use image recognition algorithm to identify the flame outline in RGB 3D image and use U-Net network to segment the flame area; A fire trend prediction module, comprising a flame area prediction unit, a smoke diffusion direction prediction unit, and a fusion unit; The flame area prediction unit is used to identify the changes in the flame area through a machine learning algorithm and output the spatial change trend of the flame; The smoke diffusion direction prediction unit extracts features from the smoke matrix, and the GNN tracks and learns the smoke diffusion boundary to obtain the boundary change direction of the smoke; The fusion unit is used to spatially fuse the spatial change trend and the boundary change direction of the smoke to generate a fire trend map, and visualize the fire trend map and transmit it to the terminal.

[0006] Preferably, the data collected by the multimodal data collection module includes wind direction data and wind speed data of the layout points; The smoke diffusion direction prediction unit is used to obtain wind direction data and wind speed data at the center of the fire source, and use the wind data to construct a penalty function to optimize the GNN. The construction process includes: S1: The wind direction data obtained and wind speed data Perform preprocessing and use the preprocessed wind direction data and wind speed data to construct the wind vector ; S2: Extract the segmentation mask of the smoke matrix data of consecutive frames and input it into the GNN, outputting the smoke diffusion vector of the next frame ; S3: Building a penalty function , the penalty function is calculated as: ; in, is the unit vector predicting the diffusion direction, is the unit vector of the observed wind direction, The Euclidean distance between two unit vectors, is the wind speed penalty term, is the fitting coefficient; S4: Use the penalty function to construct the total loss function , use the total loss function to continuously optimize GNN until convergence.

[0007] Preferably, the total loss function The calculation formula is: ; in, It is the error function between the predicted value y of GNN and the true value x.

[0008] Preferably, each data frame of the RGB three-dimensional image, smoke concentration matrix and thermal value matrix carries a coordinate label and a time label.

[0009] Preferably, the judgment module obtains the fusion feature including: The RGB image data is spatially resampled according to the coordinate labels, and an RGB resampling matrix matching the spatial resolution of the thermal value matrix is ​​generated through an interpolation algorithm; The RGB resampling matrix and the thermal value matrix are spatially aligned and input into the two-channel convolutional neural network; The two convolution layers of the dual-channel convolutional neural network respectively extract features from the RGB resampling matrix and the thermal value matrix, and output fused features after the fully connected layer of the dual-channel convolutional neural network performs feature fusion.

[0010] Preferably, the interpolation algorithm is a Kriging algorithm, and the spatial resolution of the thermal value matrix is ​​used as the interpolation target grid, and the interpolation weight is solved, and the thermal value distribution matrix is ​​generated based on the interpolation weight.

[0011] Preferably, the calculation unit further includes a linkage area unit, and the linkage area identification unit is configured to: Extracting a set of contour coordinates of each flame region based on the multiple discrete flame regions obtained by segmentation using the U-Net network; Calculate the thermal radiation influence distance between any two flame areas; If the actual spatial distance between the two flame areas is less than the heat radiation influence distance, it is determined that the two flame areas are linked to form the same high-temperature linked area; The high-temperature linkage area is determined to be a flame area.

[0012] Preferably, the linkage area determination unit further includes: When the actual spatial distance between the two flame regions is greater than or equal to the thermal radiation influence distance, the boundaries of the two flame regions are extracted, a circumscribed circle is constructed with the boundaries of the two flames, and the circumscribed circle is marked as a high temperature region.

[0013] Preferably, the thermal radiation affects the distance The calculation formula is: ; in, is the thermal conductivity coefficient of the material, and are the effective areas of the two flame regions obtained by U-Net network segmentation, is the equivalent heat source radius.

[0014] The beneficial effects of the present invention are: This solution uses the fire trend prediction module to predict the changing trend of the flame area and the smoke diffusion direction of the fire through two sub-units. Compared with only using the changing trend of the flame area as the judgment of the development trend of the fire, the judgment of the spread trend of the fire is faster and more accurate. In feature extraction, a dual-channel convolutional neural network with dual input channels is used to extract features through independent convolutional layers. Specific features of different modalities are extracted and fused before the fully connected layer to retain spatial information, resulting in better fusion effects. In model prediction, the penalty term of GNN is constructed through wind direction data, and the penalty term is used to construct an optimization function for physical guidance, which improves the prediction robustness of GNN in complex wind fields, thereby improving the system's fire recognition speed and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] To facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0016] Figure 1 It is a system structure block diagram of the present invention; Figure 2 This is a block diagram of the prediction process of the fire trend prediction module of the present invention; Figure 3 A flowchart of the steps for constructing a penalty term based on wind data in the smoke diffusion direction prediction unit of the present invention; Figure 4 This is a flowchart of the steps of the method for obtaining fusion features by the judgment module of the present invention. DETAILED DESCRIPTION

[0017] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0018] See also Figures 1-4 This embodiment provides an Internet of Things remote inspection system based on multimodal fire sensor data, including a multimodal data acquisition module, a judgment module, a calculation module, and a fire trend prediction module connected in sequence, wherein: Multimodal data acquisition module: This module includes a distributed fiber optic temperature sensor, a visible light camera, a laser smoke sensor array, and an infrared thermal imager, which are used to acquire temperature data, RGB image data, smoke data, and thermal value data, respectively. The multiple sensors in the acquisition module are synchronized via the PTP (Precision Time Protocol) clock. Position tags are generated from a pre-calibrated sensor spatial coordinate mapping table. Each data frame carries a coordinate tag and a time tag. Preprocess each type of data, including denoising, normalization, and data enhancement to improve data quality. Data enhancement includes, but is not limited to, processing vibration and acoustic data based on frequency domain filtering and adaptive noise cancellation algorithms to reduce environmental noise and signal errors. The RGB image data, smoke data, and thermal value data are respectively used to generate an RGB three-dimensional image, a smoke concentration matrix, and a thermal value matrix. The matrix representation method here is to use gridded data composed of the position readings of sensors at different layout points combined with the position coordinates of the sensors themselves to discretely represent the smoke concentration field and thermal value field; The judgment module fuses the RGB three-dimensional image and the thermal value matrix through a feature fusion algorithm, and then inputs the feature into a preset judgment model to output the judgment result to determine whether there is a fire in the layout area. The judgment model here can use historical data to train the machine learning model. The historical data is generated by historical temperature matrix, historical RGB three-dimensional image, historical smoke concentration matrix, and historical thermal value matrix data. The feature fusion algorithm can be a GRU fusion algorithm or a convolutional neural network. The computing unit uses an image recognition algorithm to identify the flame outline in the RGB three-dimensional image and uses a U-Net network to segment and calculate the flame area. The U-Net network can use an Attention U-Net network with an added attention mechanism to improve the accuracy of the U-Net network in calculating the image segmentation. When calculating the flame area, it is necessary to use image resolution (pixels / meter) or calibration objects (such as reference objects of known size) to convert pixels into actual physical objects.

[0019] Training the Attention U-Net network requires a large amount of pre-training data. This data includes a large dataset of RGB images with annotated flame and smoke regions. The annotations must clearly distinguish between flames (high-temperature luminous areas) or combine them into the same category (e.g., "fire area"). To ensure data diversity, the training data set must cover different scenes (indoor / outdoor), lighting conditions, flame forms (static / dynamic), and smoke concentrations to improve the generalization ability of the Attention U-Net network.

[0020] The fire trend prediction module is used to predict the development trend of the fire, so as to facilitate the understanding of the subsequent development of the fire. The development trend of the fire has a diffusion direction in three-dimensional space and a spread direction in two-dimensional plane. Therefore, in order to better calculate the development trend of the fire, the fire trend prediction module includes a flame area prediction unit, a smoke diffusion direction prediction unit and a fusion unit. Because the ignition area of ​​a general fire is accompanied by a large amount of thick smoke, and the temperature of the thick smoke is very high and it will cover part of the flame itself, and the diffusion direction of the thick smoke is generally the diffusion direction of the fire source in the spatial dimension, so this solution uses the thick smoke (smoke) diffusion direction to predict the flame spread direction in the spatial dimension; The flame area prediction unit uses a machine learning algorithm to identify changes in the flame area and output the spatial variation trend of the flame. The machine learning algorithm here can be one or more of the GNN, RNN and LSTM algorithms. It can learn based on historical training data and predict the spatial variation trend of the flame area here. The smoke diffusion direction prediction unit uses GNN to track the smoke diffusion boundary and learn the direction of smoke boundary change; The fusion unit spatially fuses the spatial change trend and the boundary change direction of the smoke to generate a fire trend map, and then generates a visual alarm signal from the fire trend map and sends it to the terminal. The terminal staff formulates corresponding response measures based on the visual alarm signal.

[0021] This solution takes into account that the changing trend of smoke will greatly affect the changing area of ​​flames. The fire trend prediction module uses two sub-units to respectively predict the changing trend of the flame area of ​​frequently occurring fires and the trend of smoke diffusion direction. Compared with only using the changing trend of the simple flame area as the judgment of the development trend of the fire, the judgment of this solution is faster, thereby achieving faster and more accurate judgment of the development trend of the fire.

[0022] In the prediction of smoke diffusion direction, wind direction dominates the diffusion direction, and wind speed affects the diffusion intensity and range. Especially when a fire occurs, if there is a certain wind force at the fire point, it will increase the spread of the fire to a certain extent. In one embodiment, the data collected by the multimodal data acquisition module includes wind direction data and wind speed data at the layout point. The wind direction data is used to represent the wind direction, and the wind speed data is used to represent the wind speed. The data frames of the acquired wind direction data and wind speed data both carry time tags and location tags. The smoke diffusion direction prediction unit obtains the wind direction data at the center of the fire source; Then, the wind direction data is used to construct a penalty function to optimize the GNN. The construction process includes: S1: Preprocess the acquired wind direction data, including time alignment and space alignment, and use the preprocessed wind direction data to construct the wind vector , where d=3, representing the vector direction in three-dimensional space; S2: Extract the segmentation mask of the smoke of consecutive frames and input it into G, and output the smoke diffusion vector of the next frame , is the smoke diffusion direction vector output by the GNN layer, including the instantaneous diffusion direction of the smoke. S3: Building a penalty function , the expression of the penalty function is: ;in, is the unit vector predicting the diffusion direction, is the unit vector of the observed wind direction, The Euclidean distance between two unit vectors, which reflects the difference between their directions, The wind speed penalty item increases when the wind speed is lower than the preset low threshold, and decreases when the wind speed is higher than the preset high threshold. is the fitting coefficient, which is generated by fitting historical wind direction data and calculating through multiple iterations. ,The larger the value, the stronger the penalty, and also means that the direction of the smoke diffusion vector deviates more from the wind direction; S3: Use the penalty function to construct the total loss function , use the total loss function to continuously optimize GNN until the model converges.

[0023] Total loss function The expression is: ; in, It is the error function between the predicted value y of GNN and the true value x. The error function here can use the square loss function or the cross entropy loss function to calculate the total loss function required here.

[0024] The penalty term of GNN is constructed through wind direction data, and the penalty term is used to construct an optimization function for physical guidance, which improves the prediction robustness of GNN in complex wind fields and thus improves the system's fire recognition speed and accuracy.

[0025] In one embodiment, the temperature matrix is ​​set by representing a continuous thermal value distribution field through a discretized grid to represent a thermal map within a certain area. The multimodal data acquisition module uses a trained nonlinear regression network to convert the thermal value data collected by the infrared thermal imager into a thermal value matrix, including the following steps: The infrared color image output by the infrared thermal imager is reduced in channel dimension to a grayscale image; After flattening and batch normalizing the grayscale image, the preliminary thermal value matrix is ​​calculated using the fully connected module; The preliminary thermal value matrix is ​​post-processed including inverse standard normalization to obtain the thermal value matrix.

[0026] Among them, building and training a nonlinear regression network includes: Prepare infrared pseudo-color image samples and their temperature matrices as labels to construct a training dataset; build a nonlinear regression network and initialize the network weight parameters of the nonlinear regression network; The infrared pseudo-color image in the training data set is input into the nonlinear regression network for forward calculation and processing, and the regression loss between the predicted thermal value matrix output by the nonlinear regression network and the thermal value matrix as the label is calculated. The gradient of the loss function relative to the neuron weight of each network layer is reversely calculated based on the regression loss, and the network weight parameters of the nonlinear regression network are updated; Repeat the above steps until the specified number of trainings are completed or the regression loss is less than the set value. The temperature value of the temperature sensor is converted into a thermal value matrix through the set nonlinear regression network, and the thermal value matrix is ​​used to reflect the thermal monitoring values ​​of multiple pixels at the layout point.

[0027] Since the data between the thermal value matrix and the RGB image data are spatially discretized due to the different resolutions of the two sensors at the acquisition end, it is difficult to establish geometric consistency between the two. If they are directly extracted, the spatial position will be blurred, and it will be impossible to establish an accurate correspondence between the RGB image data and the thermal value matrix. Therefore, in order to improve the accuracy of the fusion of the two features, in one embodiment, the judgment module obtains the fusion feature by including the following steps: a: Use coordinate labels to spatially resample the RGB matrix. Based on the coordinate labels, calculate the corresponding coordinate position of each pixel of the thermal value matrix in the original RGB image. Use the Kriging algorithm (ordinary Kriging algorithm) and use the spatial resolution of the thermal value matrix as the interpolation target grid. Use ordinary Kriging to solve the interpolation weights to generate a resampled RGB matrix that matches the spatial resolution of the thermal value matrix. b: Align the original thermal value matrix and the resampled RGB matrix through the coordinate space of the pixel position to align the spatial position of each pixel, and verify the alignment accuracy by superimposing the displayed contour or key points; c: Spatially align the RGB resampling matrix and the thermal value matrix and input them into the two-channel convolutional neural network; d: The two convolutional layers of the dual-channel convolutional neural network extract features from the RGB resampling matrix and the thermal value matrix respectively, and output the fused features after feature fusion in the fully connected layer of the dual-channel convolutional neural network. Specifically: The resampled RGB matrix and the original thermal value matrix are input into the dual-channel convolutional neural network through two independent channels respectively. Then, RGB feature extraction and thermal feature extraction are performed on the two convolutional layers of the dual-channel convolutional neural network respectively. The output feature maps of the two branches are spliced ​​in the channel dimension to obtain the fusion feature.

[0028] Resampling resolves the resolution gap between RGB and thermal sensors, and pixel-level alignment is achieved through coordinate labels to ensure the effectiveness of fusion. The dual-channel convolutional neural network has dual input channels and performs feature extraction through independent convolutional layers. It can learn specific features of different modalities, such as RGB texture and thermal temperature gradient, and fuse the retained spatial information before the fully connected layer. This is better than the fusion effect of directly inputting 4 channels in the early stage of the network or splicing the outputs of independent networks in the late stage of fusion.

[0029] During the development of a fire, even if there is only a single ignition point initially, it may be affected by uncontrollable dynamic factors (such as burning materials falling from a height, strong winds causing flying fire to spread, etc.), which may trigger multiple secondary ignition points with discrete distribution in space. In such multi-point fire scenarios, each independent flame area needs to be monitored and evaluated separately. However, when the spatial distance between two flame areas is lower than the critical threshold, the thermal radiation energy released will produce a significant superposition effect, causing a high-temperature radiation field to form in the middle area. This phenomenon is extremely dangerous in terms of the spread trend of the fire and emergency rescue scenarios. Specifically: Since the thermal radiation flux decays with distance according to the inverse square law, when the distance between two flame areas is less than a certain threshold, the superposition of the radiation flux between the two flame areas will cause the temperature of the middle area to rise exponentially, significantly reducing the ignition time of combustible materials. Moreover, the superimposed radiation field can cause the heat flux density to exceed the human tolerance threshold, causing thermal exposure burns to rescue personnel or failure of protective equipment, so it is necessary to focus on identification.

[0030] In one embodiment, the calculation unit further includes a linkage area unit, and the linkage area identification unit is configured to: Based on the multiple discrete flame regions obtained by U-Net network segmentation, the contour coordinate set of each flame region is extracted; Calculate the thermal radiation influence distance between any two flame areas, the thermal radiation influence distance The calculation formula is: ; in, is the combustion object influence coefficient, which belongs to the Stefan-Boltzmann law correction term and its calculation formula is: ;in, is the Stefan-Boltzmann constant term, The maximum flame temperature can be extracted from the thermal map data or calibrated by presetting. is the thermal conductivity coefficient of the material of the burning object; and The actual spatial areas of the two flame regions are respectively segmented and calculated by the U-Net network and calculated using spatial mapping. is the equivalent heat source radius; If the actual spatial distance between the two flame areas is less than the thermal radiation influence distance, the two flame areas are judged to be linked to the same high-temperature linkage area; The high-temperature linkage area is determined as a flame area. The "flame area" here is the area of ​​the two flame areas plus the area between the two flame areas. The area is the area within the range of thermal radiation influence. When the actual spatial distance between the two flame areas is greater than or equal to the thermal radiation influence distance, the boundaries of the two flame areas are extracted, a circumscribed circle is constructed based on the boundaries of the two flames, and the circumscribed circle is marked as a high-temperature area; The high temperature area here is the impact area of ​​multiple flame areas, which has the same meaning as the "boundary fire scene" defined during fire fighting.

[0031] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any person skilled in the art can make some changes or modifications to equivalent embodiments using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. An Internet of Things remote inspection system based on multimodal fire sensor data, characterized by: include: Multimodal data acquisition module: used to obtain RGB three-dimensional images, smoke concentration matrix and thermal value matrix of the monitoring area in real time; A judgment module is used to fuse the RGB three-dimensional image and the thermal value matrix using a feature fusion algorithm to generate fusion features, input them into a preset judgment model, and output a judgment result to determine whether there is a fire in the layout point area; Computing module: used to identify the flame outline in the RGB three-dimensional image using the image recognition algorithm and segment the flame area using the U-Net network; A fire trend prediction module, comprising a flame area prediction unit, a smoke diffusion direction prediction unit, and a fusion unit; The flame area prediction unit is used to identify the changes in the flame area through a machine learning algorithm and output the flame space change trend; The smoke diffusion direction prediction unit extracts features from the smoke matrix and uses GNN to track and learn the smoke diffusion boundary to obtain the direction of smoke boundary change; The fusion unit is used to spatially fuse the spatial change trend and the boundary change direction of the smoke to generate a fire trend map, and visualize the fire trend map and transmit it to the terminal.

2. The IoT remote inspection system based on multimodal fire sensor data according to claim 1 is characterized by: The data collected by the multimodal data collection module includes wind direction data and wind speed data of the layout points; The smoke diffusion direction prediction unit is used to obtain wind direction data and wind speed data at the center of the fire source, and use the wind data to construct a penalty function to optimize the GNN. The construction process includes: S1: The wind direction data obtained and wind speed data Perform preprocessing and use the preprocessed wind direction data and wind speed data to construct the wind vector ; S2: Extract the segmentation mask of the smoke matrix data of consecutive frames and input it into the GNN, outputting the smoke diffusion vector of the next frame ; S3: Building a penalty function , the expression of the penalty function is: ； in, is the unit vector predicting the diffusion direction, is the unit vector of the observed wind direction, The Euclidean distance between two unit vectors, is the wind speed penalty term, is the fitting coefficient; S4: Use the penalty function to construct the total loss function , use the total loss function to continuously optimize GNN until convergence.

3. The IoT remote inspection system based on multimodal fire sensor data according to claim 2 is characterized by: The total loss function The calculation formula is: ； in, It is the error function between the predicted value y of GNN and the true value x.

4. The IoT remote inspection system based on multimodal fire sensor data according to claim 1 is characterized by: Each data frame of the RGB three-dimensional image, smoke concentration matrix and thermal value matrix carries a coordinate label and a time label.

5. The IoT remote inspection system based on multimodal fire sensor data according to claim 4 is characterized by: The judgment module obtains the fusion features including: The RGB image data is spatially resampled according to the coordinate labels, and an RGB resampling matrix matching the spatial resolution of the thermal value matrix is ​​generated through an interpolation algorithm; The RGB resampling matrix and the thermal value matrix are spatially aligned and input into the two-channel convolutional neural network; The two convolutional layers of the dual-channel convolutional neural network respectively extract features from the RGB resampling matrix and the thermal value matrix, and output fused features after the fully connected layer of the dual-channel convolutional neural network performs feature fusion.

6. The IoT remote inspection system based on multimodal fire sensor data according to claim 5, characterized in that: The interpolation algorithm is the Kriging algorithm, and the spatial resolution of the thermal value matrix is ​​used as the interpolation target grid, and the interpolation weight is solved, and the thermal value distribution matrix is ​​generated based on the interpolation weight.

7. The IoT remote inspection system based on multimodal fire sensor data according to claim 1, characterized in that: The calculation unit further includes a linkage area unit, and the linkage area identification unit is configured to: Extracting a set of contour coordinates of each flame region based on the multiple discrete flame regions obtained by segmentation using the U-Net network; Calculate the thermal radiation influence distance between any two flame areas; If the actual spatial distance between the two flame areas is less than the heat radiation influence distance, it is determined that the two flame areas are linked to form the same high-temperature linked area; The high-temperature linkage area is determined to be a flame area.

8. The IoT remote inspection system based on multimodal fire sensor data according to claim 7, characterized in that: The linkage area determination unit further includes: When the actual spatial distance between the two flame regions is greater than or equal to the heat radiation influence distance, the boundaries of the two flame regions are extracted, a circumscribed circle is constructed with the boundaries of the two flames, and the circumscribed circle is marked as a high temperature region.

9. The IoT remote inspection system based on multimodal fire sensor data according to claim 7, characterized in that: The thermal radiation affects the distance The calculation formula is: ； in, is the thermal conductivity coefficient of the material, and are the effective areas of the two flame regions obtained by U-Net network segmentation, is the equivalent heat source radius.

Images