An internet of things remote inspection system based on multi-modal fire fighting sensing data

Abstract

The application belongs to the technical field of remote control, and particularly relates to an Internet of Things remote inspection system based on multi-modal fire-fighting sensing data, which comprises a multi-modal data acquisition module, a judgment module and a fire trend prediction module. The multi-modal data acquisition module acquires the RGB three-dimensional image, the smoke concentration matrix and the thermal value matrix of a monitoring area in real time. The judgment module judges whether there is a fire after feature fusion of multiple data sources. The fire trend prediction module respectively predicts the spatial change trend of the flame and the boundary change trend of the smoke, and then performs spatial fusion on the spatial change trend and the boundary change direction of the smoke to generate a fire trend graph. Compared with the judgment of using only the change trend of the simple flame area as the development trend of the fire, the judgment of the spreading trend of the fire is more rapid and has higher precision.

Description

Technical Field

[0001] This invention belongs to the field of remote control technology, specifically relating to an Internet of Things (IoT) remote inspection system based on multimodal fire sensor data. Background Technology

[0002] Multimodal deep learning is a subfield of artificial intelligence that focuses on developing models capable of simultaneously processing and learning multiple types of data. These data types, or modalities, can include text, images, audio, video, and sensor data. With the development and evolution of artificial intelligence, multimodal deep learning technology has been widely applied in fire inspection. By fusing multiple data from different sources and types at the same inspection location and analyzing the characteristics of different sources and types of data, a corresponding modal dataset is constructed. This multi-dimensional data synthesis modal dataset is used to extract features from the multimodal dataset to build a fire trend prediction model to predict the development trend of fires. However, existing fire prediction models only focus on the spatiotemporal variation of the flame body area and do not fully couple the influence of environmental factors on combustion behavior, resulting in a lack of generalization ability of fire prediction models to predict the occurrence trend of fires.

[0003] Specifically, wind is a key driver of fire spread. Due to the airflow transport effect, the wind speed vector directly changes the flame tilt angle and the direction of thermal convection, causing the flame front to shift to the windward or crosswind side at a non-linear rate. Furthermore, the increase in wind speed will increase the oxygen concentration gradient in the combustion zone, accelerate the chemical reaction rate, and cause the flame height and radiation intensity to increase exponentially. This will cause the prediction of fire prediction models to lag. This lag directly weakens the timeliness of the emergency decision-making system and may lead to secondary risks such as delays in resource allocation and failure of safety isolation zone settings. Summary of the Invention

[0004] To address the aforementioned problems in existing technologies, this invention provides an IoT-based remote inspection system based on multimodal fire sensor data. This system solves the problem that existing technologies lack dynamic correlation between fire development and changes in wind conditions, thus making it impossible to accurately assess the development trend of a fire.

[0005] The objective of this invention can be achieved through the following technical solution: an IoT remote inspection system based on multimodal fire sensor data, comprising:

[0006] Multimodal data acquisition module: used to acquire RGB 3D images, smoke concentration matrix, and thermal value matrix of the monitoring area in real time;

[0007] The judgment module is used to perform feature fusion on the RGB three-dimensional image and thermal value matrix using a feature fusion algorithm to generate fused features, and then input them into a preset judgment model to output the judgment result to determine whether there is a fire in the area of ​​the deployment point.

[0008] Calculation module: Uses image recognition algorithms to identify flame outlines in RGB 3D images and uses a U-Net network to segment the flame region;

[0009] A fire trend prediction module, comprising a flame area prediction unit, a smoke diffusion direction prediction unit, and a fusion unit;

[0010] The flame region prediction unit is used to identify changes in the flame region and output the spatial change trend of the flame through machine learning algorithms.

[0011] The smoke diffusion direction prediction unit extracts features from the smoke matrix and uses a GNN to track and learn the smoke diffusion boundary to obtain the direction of smoke boundary change.

[0012] The fusion unit is used to spatially fuse the spatial change trend and the boundary change direction of the smoke to generate a fire trend map, and visualize the fire trend map and transmit it to the terminal.

[0013] Preferably, the data collected by the multimodal data acquisition module includes wind direction data and wind speed data at the deployment points;

[0014] The smoke diffusion direction prediction unit is used to acquire wind direction and wind speed data at the center of the fire source, and uses the wind data to construct a penalty term function to optimize the GNN. The construction process includes:

[0015] S1: The acquired wind direction data Wind speed data Preprocessing is performed, and the preprocessed wind direction and speed data are used to construct wind force vectors. ;

[0016] S2: Extract the segmentation mask of the continuous frame smoke matrix data and input it into the GNN. Output the smoke diffusion vector of the next frame. ;

[0017] S3: Construct the penalty function The formula for calculating the penalty function is:

[0018] ;

[0019] in, It is the unit vector that predicts the direction of diffusion. It is the unit vector for observing wind direction. The Euclidean distance between two unit vectors This is a wind speed penalty item. These are the fitting coefficients;

[0020] S4: Construct the total loss function using the penalty function. The GNN is continuously optimized using the total loss function until it converges.

[0021] Preferably, the total loss function The calculation formula is:

[0022] ;

[0023] in, Let y be the error function between the predicted value y and the true value x of the GNN.

[0024] Preferably, each data frame of the RGB three-dimensional image, smoke concentration matrix, and thermal value matrix carries coordinate labels and time labels.

[0025] Preferably, the fusion features obtained by the judgment module include:

[0026] Spatial resampling of RGB image data is performed based on coordinate labels, and an RGB resampling matrix matching the spatial resolution of the thermal value matrix is ​​generated through an interpolation algorithm.

[0027] The RGB resampling matrix and the thermal value matrix are spatially aligned and then input into a dual-channel convolutional neural network.

[0028] The two convolutional layers of the dual-channel convolutional neural network extract features from the RGB resampling matrix and the heatmap matrix, respectively. After feature fusion in the fully connected layer of the dual-channel convolutional neural network, the fused features are output.

[0029] Preferably, the interpolation algorithm is the Kriging algorithm, and the spatial resolution of the thermal value matrix is ​​used as the interpolation target grid. The interpolation weights are solved, and the thermal value distribution matrix is ​​generated based on the interpolation weights.

[0030] Preferably, the calculation unit further includes a linkage region unit, and the linkage region identification unit is configured as follows:

[0031] Based on the multiple discrete flame regions segmented by the U-Net network, the contour coordinate set of each flame region is extracted.

[0032] Calculate the distance of thermal radiation influence between any two flame regions;

[0033] If the actual spatial distance between the two flame regions is less than the thermal radiation influence distance, it is determined that the two flame regions are linked as the same high-temperature linkage region.

[0034] The high-temperature linkage area is identified as a flame area.

[0035] Preferably, the linkage area determination unit further includes:

[0036] When the actual spatial distance between the two flame regions is greater than or equal to the thermal radiation influence distance, the boundary between the two flame regions is extracted, an outer circle is constructed based on the boundary between the two flames, and the outer circle is marked as a high-temperature region.

[0037] Preferably, the thermal radiation influence distance The calculation formula is:

[0038] ;

[0039] in, The thermal conductivity coefficient of the material. and These represent the effective areas of the two flame regions obtained by segmentation using the U-Net network. The radius is the equivalent heat source radius.

[0040] The beneficial effects of this invention are as follows:

[0041] This solution uses a fire trend prediction module with two sub-units to predict the changing trends of the flame area and the direction of smoke spread in a fire. Compared to using only the simple changing trend of the flame area as a basis for judging the development trend of a fire, this solution is faster and more accurate in judging the spread trend of a fire.

[0042] In terms of feature extraction, a dual-channel convolutional neural network with dual input channels and separate convolutional layers is used to extract features. Specific features of different modalities are extracted and fused before the fully connected layer to retain spatial information, resulting in better fusion effect.

[0043] In terms of model prediction, a penalty term for the GNN is constructed using wind direction data, and an optimization function is built using the penalty term for physical guidance to improve the prediction robustness of the GNN under complex wind fields, thereby improving the system's speed and accuracy in identifying fires. Attached Figure Description

[0044] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0045] Figure 1 This is a system structure block diagram of the present invention;

[0046] Figure 2 This is a flowchart illustrating the prediction process of the fire trend prediction module of the present invention.

[0047] Figure 3 This is a flowchart illustrating the steps of the smoke diffusion direction prediction unit of the present invention in constructing a penalty term based on wind data;

[0048] Figure 4 This is a flowchart illustrating the steps of the method for obtaining fused features using the judgment module of the present invention. Detailed Implementation

[0049] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0050] Please see Figures 1-4 This embodiment provides an IoT remote inspection system based on multimodal fire sensor data, including a multimodal data acquisition module, a judgment module, a calculation module, and a fire trend prediction module connected in sequence, wherein:

[0051] Multimodal data acquisition module: includes distributed fiber optic temperature sensor, visible light camera, laser smoke sensor array, and infrared thermal imager, which are used to acquire temperature data, RGB image data, smoke data and thermal value data respectively. The multiple sensors of the acquisition module are synchronized through PTP (Precise Time Protocol) clock. The position tag is generated by a pre-calibrated sensor spatial coordinate mapping table. The data frames of each data are equipped with coordinate tags and time tags.

[0052] Preprocessing is performed on each type of data, including denoising, normalization, and data augmentation of the collected raw data to improve data quality. Data augmentation includes, but is not limited to, processing vibration and acoustic data based on frequency domain filtering and adaptive noise cancellation algorithms to reduce environmental noise and signal errors.

[0053] The RGB image data, smoke data, and thermal value data are used to generate RGB three-dimensional images, smoke concentration matrices, and thermal value matrices, respectively. Here, the matrix representation is based on the readings of the sensor positions at different locations combined with the sensor's own position coordinates, using gridded data to discretize the smoke concentration field and thermal value field.

[0054] The judgment module fuses the RGB 3D image and thermal value matrix using a feature fusion algorithm and inputs them into a pre-set judgment model. It then outputs the judgment result to determine whether there is a fire in the area where the deployment point is located. The judgment model here can be generated by training a machine learning model using historical data. The historical data is generated by historical temperature matrix, historical RGB 3D image, historical smoke concentration matrix, and historical thermal value matrix data. The feature fusion algorithm can be a GRU fusion algorithm or a convolutional neural network, etc.

[0055] The computational unit uses an image recognition algorithm to identify the flame outline in an RGB 3D image and uses a U-Net network to segment and calculate the flame region. The U-Net network can use an Attention U-Net network with an added attention mechanism to improve the accuracy of the U-Net network in image segmentation. When calculating the flame region, it is necessary to convert the pixels into actual physical values ​​by using the image resolution (pixels / meter) or a calibration object (such as a reference object of known size).

[0056] Training the Attention U-Net network requires acquiring a large amount of training data for pre-training. This training data includes a large dataset of RGB images labeled with flame and smoke regions. The labels must clearly distinguish between flames (high-temperature luminous areas) or group them into the same category (e.g., "on fire area"). To ensure data diversity, the training set data must cover different scenes (indoor / outdoor), lighting conditions, flame morphology (static / dynamic), and smoke concentration to improve the generalization ability of the Attention U-Net network.

[0057] The fire trend prediction module is used to predict the development trend of a fire, so as to grasp the subsequent development of the fire. The development trend of a fire has a three-dimensional diffusion direction and a two-dimensional spread direction. Therefore, in order to better calculate the development trend of a fire, the fire trend prediction module includes a flame area prediction unit, a smoke diffusion direction prediction unit, and a fusion unit. Because the ignition area of ​​a typical fire is accompanied by a large amount of dense smoke, and the temperature of the dense smoke is very high and it will cover part of the flame itself. Moreover, the diffusion direction of dense smoke is generally the diffusion direction of the fire source in the spatial dimension. Therefore, this solution predicts the diffusion direction of dense smoke (smoke) as the spread direction of the flame in the spatial dimension.

[0058] The flame region prediction unit uses machine learning algorithms to identify changes in the flame region and output the spatial change trend of the flame. The machine learning algorithm here can be one or more of GNN, RNN and LSTM algorithms. It can learn based on historical training data and predict the spatial change trend of the flame region.

[0059] The smoke diffusion direction prediction unit uses a GNN to track and learn the smoke diffusion boundary to obtain the direction of smoke boundary change;

[0060] The fusion unit spatially merges the spatial change trend and the boundary change direction of the smoke to generate a fire trend map, and then generates a visual alarm signal from the fire trend map and sends it to the terminal. The terminal staff can then formulate corresponding response measures based on the visual alarm signal.

[0061] This solution takes into account that the changing trend of smoke greatly affects the changing area of ​​the flame. The fire trend prediction module uses two sub-units to separately predict the changing trend of the flame area and the trend of smoke diffusion in multiple fires. Compared with using only the simple changing trend of the flame area as the judgment of the fire development trend, this solution can make a faster judgment, thus achieving a faster and more accurate judgment of the fire development trend.

[0062] In predicting the direction of smoke diffusion, wind direction dominates the diffusion direction, while wind speed affects the diffusion intensity and range. Especially when a fire occurs, if there is a certain amount of wind at the fire point, it will increase the spread of the fire to a certain extent. In one implementation, the data collected by the multimodal data acquisition module includes wind direction data and wind speed data of the deployment point. The wind direction data is used to characterize the direction of the wind, and the wind speed data is used to characterize the speed of the wind. The data frames of the acquired wind direction data and wind speed data both carry time tags and location tags.

[0063] The smoke diffusion direction prediction unit acquires wind direction data at the center of the fire source;

[0064] Then, a penalty function is constructed using wind direction data to optimize the GNN. The construction process includes:

[0065] S1: Preprocess the acquired wind direction data, including time alignment and spatial alignment, and construct a wind force vector using the preprocessed wind direction data. , where d=3, represents the vector direction in three-dimensional space;

[0066] S2: Extract the segmentation mask of smoke from consecutive frames, input it into G, and output the smoke diffusion vector of the next frame. , This is the smoke diffusion direction vector output by the GNN layer, including the instantaneous diffusion direction of the smoke.

[0067] S3: Construct the penalty function The expression for the penalty function is:

[0068]

[0069] ;in, It is the unit vector that predicts the direction of diffusion. It is the unit vector for observing wind direction.

[0070] The Euclidean distance between two unit vectors reflects the difference between their directions. This is a wind speed penalty; when the wind speed is below a preset low threshold, the penalty is increased, and when the wind speed is above a preset high threshold, the penalty is decreased. These are the fitting coefficients, generated by fitting historical wind direction data and calculating through multiple iterations. The larger the value, the stronger the penalty, and the more the smoke diffusion vector deviates from the wind direction;

[0071] S3: Construct the total loss function using the penalty function. The GNN is continuously optimized using the total loss function until the model converges.

[0072] Total loss function The expression is:

[0073] ;

[0074] in, Let y be the error function between the predicted value y and the true value x of the GNN. Here, the error function can be either the squared loss function or the cross-entropy loss function. The total loss function required here can be calculated.

[0075] By constructing a penalty term for the GNN using wind direction data and then using the penalty term to construct an optimization function for physical guidance, the predictive robustness of the GNN under complex wind fields is improved, thereby enhancing the system's speed and accuracy in identifying fires.

[0076] In one implementation, the temperature matrix is ​​set by using a discretized grid to represent the continuous thermal value distribution field to characterize the thermal map within a certain area. The multimodal data acquisition module uses a trained nonlinear regression network to convert the thermal value data acquired by the infrared thermal imager into a thermal value matrix, including the following steps:

[0077] The infrared color image output by the infrared thermal imager is reduced to grayscale by channel dimensionality reduction.

[0078] After flattening and batch normalizing the grayscale image, a preliminary thermodynamic matrix is ​​calculated using a fully connected module.

[0079] The initial thermodynamic value matrix is ​​post-processed with inverse standard normalization to obtain the thermodynamic value matrix.

[0080] The construction and training of the nonlinear regression network includes:

[0081] Prepare infrared pseudo-color image samples and their temperature matrices as labels to construct a training dataset; construct a nonlinear regression network and initialize the network weight parameters of the nonlinear regression network.

[0082] Infrared pseudo-color images from the training dataset are input into a nonlinear regression network for forward computation and processing. The regression loss between the predicted thermal value matrix output by the nonlinear regression network and the thermal value matrix used as the label is calculated. Based on the regression loss, the gradient of the loss function with respect to the weights of neurons in each network layer is calculated in reverse, and the network weight parameters of the nonlinear regression network are updated.

[0083] Repeat the above steps until the specified number of training iterations are completed or the regression loss is less than the set value. The temperature values ​​from the temperature sensor are converted into a thermal value matrix using a configured nonlinear regression network. This thermal value matrix reflects the thermal monitoring values ​​at multiple pixels of the placement point.

[0084] Because the resolutions of the two sensors differ at the acquisition end, the data acquired by the thermal matrix and RGB image data are spatially discretized, making it difficult to establish geometric consistency between them. Direct extraction would result in spatial ambiguity, making it impossible to accurately determine the correspondence between the RGB image data and the thermal matrix. Therefore, to improve the accuracy of feature fusion, in one implementation, the judgment module obtains the fused features by including the following steps:

[0085] a: Spatial resampling of the RGB matrix is ​​performed using coordinate labels. Based on the coordinate labels, the corresponding coordinate position of each pixel in the heat value matrix in the original RGB image is calculated. The Kriging algorithm (ordinary Kriging algorithm) is used, and the spatial resolution of the heat value matrix is ​​used as the interpolation target grid. The interpolation weights are solved using ordinary Kriging to generate a resampled RGB matrix with spatial resolution matching of the heat value matrix.

[0086] b: Align the original thermal matrix and the resampled RGB matrix in spatial position using the coordinate space of the pixel position, and verify the alignment accuracy by overlaying the display outline or key points;

[0087] c: Spatially align the RGB resampling matrix and the thermal matrix, and input them into a dual-channel convolutional neural network;

[0088] d: The two convolutional layers of the dual-channel convolutional neural network extract features from the RGB resampling matrix and the heatmap matrix, respectively, and then fuse these features in the fully connected layer of the dual-channel convolutional neural network to output the fused features. Specifically:

[0089] The resampled RGB matrix and the original thermal matrix are input into a dual-channel convolutional neural network through two independent channels. RGB feature extraction and thermal feature extraction are then performed on the two convolutional layers of the dual-channel convolutional neural network, respectively. The output feature maps of the two branches are then concatenated along the channel dimension to obtain the fused feature.

[0090] The resolution gap between the RGB and thermal sensors is addressed by resampling, and pixel-level alignment is achieved through coordinate labels to ensure the effectiveness of the fusion. Furthermore, the dual-channel convolutional neural network with dual input channels and separate convolutional layers for feature extraction can learn specific features of different modalities, such as the texture of RGB and the temperature gradient of thermal sensors. Spatial information is preserved by fusing before the fully connected layer, resulting in better fusion performance compared to direct input of 4 channels through early fusion or splicing of independent network outputs through late fusion.

[0091] During the development of a fire, even if there is only a single ignition point initially, uncontrollable dynamic factors (such as burning materials falling from heights or strong winds causing the spread of embers) can trigger multiple secondary ignition points that are spatially dispersed. In such multi-point fire scenarios, each independent flame area needs to be monitored and assessed separately. However, when the spatial distance between two flame areas is less than a critical threshold, the released thermal radiation energy will have a significant superposition effect, causing a high-temperature radiation field to form in the middle area. This phenomenon poses an extremely high risk to the spread of fire and emergency rescue scenarios. Specifically:

[0092] Since the thermal radiation flux decreases with distance according to the inverse square law, when the distance between two flame regions is less than a certain threshold, the superposition of the radiation flux between the two flame regions will cause the temperature in the middle region to rise exponentially, significantly reducing the ignition time of combustibles. Moreover, the superimposed radiation field can cause the heat flux density to exceed the human body's tolerance threshold, causing thermal exposure burns to rescuers or failure of protective devices. Therefore, it is necessary to focus on identifying this issue.

[0093] In one implementation, the computing unit further includes a linkage region unit, and the linkage region identification unit is configured as follows:

[0094] Based on the U-Net network segmentation of multiple discrete flame regions, the contour coordinate set of each flame region is extracted;

[0095] Calculate the thermal radiation influence distance between any two flame regions. The calculation formula is:

[0096] ;

[0097] in, This is the influence coefficient of the combustible material, which is a correction term in the Stefan-Boltzmann law. Its calculation formula is as follows: ;in, For the Stefan-Boltzmann constant term, The highest flame temperature can be extracted from heatmap data or through preset calibration. The thermal conductivity coefficient of the material in the combustion object;

[0098] and The actual spatial areas of the two flame regions, obtained by segmenting the U-Net network and calculating using spatial mapping, are respectively calculated. The radius of the equivalent heat source;

[0099] If the actual spatial distance between two flame regions is less than the distance affected by thermal radiation, it is determined that the two flame regions are linked as the same high-temperature linkage region.

[0100] The high-temperature linkage area is defined as a flame area. Here, "flame area" is the area of ​​two flame areas plus the area of ​​the area between the two flame areas. The area is the area within the range of heat radiation influence.

[0101] When the actual spatial distance between two flame regions is greater than or equal to the thermal radiation influence distance, the boundary between the two flame regions is extracted, an outer circle is constructed based on the boundary between the two flames, and the outer circle is marked as the high-temperature region.

[0102] The high-temperature area here is the area affected by multiple flame zones, which is consistent with the meaning of the "boundary fire field" defined during fire fighting.

[0103] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An IoT remote inspection system based on multimodal fire sensor data, characterized in that: include: Multimodal data acquisition module: used to acquire RGB 3D images, smoke concentration matrix and thermal value matrix of the monitoring area in real time; The judgment module is used to perform feature fusion on the RGB three-dimensional image and the thermal value matrix using a feature fusion algorithm to generate fused features, which are then input into a preset judgment model and the judgment result is output to determine whether there is a fire in the deployment point area. The computation module is used to identify flame outlines in RGB 3D images using image recognition algorithms and to segment the flame region using a U-Net network. A fire trend prediction module, comprising a flame area prediction unit, a smoke diffusion direction prediction unit, and a fusion unit; The flame region prediction unit is used to identify changes in the flame region and output the flame spatial change trend through machine learning algorithms. The smoke diffusion direction prediction unit extracts features from the smoke matrix and uses a GNN to track and learn the smoke diffusion boundary to obtain the direction of smoke boundary change. The fusion unit is used to spatially fuse the spatial change trend and the boundary change direction of the smoke to generate a fire trend map, and visualize the fire trend map and transmit it to the terminal. The multimodal data acquisition module collects wind direction and wind speed data at the deployment points; The smoke diffusion direction prediction unit is used to acquire wind direction and wind speed data at the center of the fire source, and uses the wind data to construct a penalty term function to optimize the GNN. The construction process includes: S1: The acquired wind direction data Wind speed data Preprocessing is performed, and the preprocessed wind direction and speed data are used to construct wind force vectors. ; S2: Extract the segmentation mask of the continuous frame smoke matrix data and input it into the GNN. Output the smoke diffusion vector of the next frame. ; S3: Construct the penalty function The expression for the penalty function is: ； in, It is the unit vector that predicts the direction of diffusion. It is the unit vector for observing wind direction. The Euclidean distance between two unit vectors This is a wind speed penalty item. These are the fitting coefficients; S4: Construct the total loss function using the penalty function. The GNN is continuously optimized using the total loss function until it converges.

2. The IoT remote inspection system based on multimodal fire sensor data according to claim 1, characterized in that: The total loss function The calculation formula is: ； in, Let y be the error function between the predicted value y and the true value x of the GNN.

3. The IoT remote inspection system based on multimodal fire sensor data according to claim 1, characterized in that: Each data frame of the RGB 3D image, smoke concentration matrix, and thermal value matrix carries coordinate labels and time labels.

4. The IoT remote inspection system based on multimodal fire sensor data according to claim 1, characterized in that: The fusion features obtained by the judgment module include: Spatial resampling of RGB image data is performed based on coordinate labels, and an RGB resampling matrix matching the spatial resolution of the thermal value matrix is ​​generated through an interpolation algorithm. The RGB resampling matrix and the thermal value matrix are spatially aligned and then input into a dual-channel convolutional neural network. The two convolutional layers of the dual-channel convolutional neural network extract features from the RGB resampling matrix and the heat value matrix, respectively, and output fused features after feature fusion in the fully connected layer of the dual-channel convolutional neural network.

5. The IoT remote inspection system based on multimodal fire sensor data according to claim 4, characterized in that: The interpolation algorithm is the Kriging algorithm, which uses the spatial resolution of the thermal value matrix as the interpolation target grid and solves for the interpolation weights to generate the thermal value distribution matrix based on the interpolation weights.

6. The IoT remote inspection system based on multimodal fire sensor data according to claim 1, characterized in that: The calculation module further includes a linkage region unit, and the linkage region identification unit is configured as follows: Based on the multiple discrete flame regions segmented by the U-Net network, the contour coordinate set of each flame region is extracted. Calculate the distance of thermal radiation influence between any two flame regions; If the actual spatial distance between the two flame regions is less than the thermal radiation influence distance, it is determined that the two flame regions are linked as the same high-temperature linkage region. The high-temperature linkage area is identified as a flame area.

7. The IoT remote inspection system based on multimodal fire sensor data according to claim 6, characterized in that: The linkage region identification unit also includes: When the actual spatial distance between the two flame regions is greater than or equal to the thermal radiation influence distance, the boundary between the two flame regions is extracted, an outer circle is constructed based on the boundary between the two flames, and the outer circle is marked as a high-temperature region.

8. The IoT remote inspection system based on multimodal fire sensor data according to claim 7, characterized in that: The distance affected by thermal radiation The calculation formula is: ； in, The thermal conductivity coefficient of the material. and These represent the effective areas of the two flame regions obtained by segmentation using the U-Net network. The radius is the equivalent heat source radius.

Images