Fire safety early warning and fire source detection system based on multi-image feature fusion

Abstract

This invention belongs to the field of fire safety alarm devices, specifically involving a fire safety early warning and fire source detection system based on multi-image feature fusion. It simultaneously acquires visible light image sequences and infrared thermal imaging image sequences, extracting visible light morphological texture feature vectors and infrared temperature gradient distribution feature vectors. A dual-stream spatiotemporal graph convolutional network is constructed, and a physical constraint mechanism based on the heat conduction equation is introduced into the feature fusion layer to calculate the spatiotemporal coupling degree between the visible light morphological change rate and the infrared temperature diffusion rate. Using the spatiotemporal coupling degree as a dynamic weight adjustment factor, the two feature vectors are weighted and fused channel-by-channel to suppress false fire source feature responses that do not conform to thermodynamic evolution laws. The fused features are input into a classification regression model to output early warning commands. This invention suppresses false fire source feature responses, reduces the false alarm rate in complex environments, reduces computational resource consumption and latency, and improves the sensitivity of early fire source identification.

Description

Technical Field

[0001] This invention belongs to the field of fire safety alarm devices, specifically involving a fire safety early warning and fire source detection system based on multi-image feature fusion. Background Technology

[0002] Existing fire safety early warning and fire source detection systems mostly use dual-modal imaging equipment (visible light and infrared) to acquire images of the monitored area. In the data processing stage, conventional solutions typically utilize a dual-stream network architecture to extract morphological texture features from the visible light image and temperature distribution features from the infrared image, respectively. Subsequently, existing technologies directly concatenate the feature vectors from the two modalities, or logically combine the recognition results output independently by the dual-stream network, determining whether a fire warning is triggered based on the confidence level of the combined result.

[0003] The aforementioned existing technologies have a core technical problem: they do not consider the inherent thermodynamic coupling relationship between flame shape changes and temperature diffusion during a fire. The fusion method of simple splicing or logical combination cannot filter out false feature correlations that do not conform to the physical laws of heat conduction, which makes the system prone to false alarms in complex environments with single-mode interference sources such as high-temperature equipment or strong light reflection. Summary of the Invention

[0004] The purpose of this invention is to provide a fire safety early warning and fire source detection system based on multi-image feature fusion, which can solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A fire safety early warning and fire source detection system based on multi-image feature fusion includes: a multimodal image acquisition front-end for simultaneously acquiring visible light image sequences and infrared thermal imaging image sequences of the monitored area; a feature extraction processor for extracting visible light morphological texture feature vectors from the visible light image sequences and infrared temperature gradient distribution feature vectors from the infrared thermal imaging image sequences; a feature fusion early warning processor for constructing a dual-stream spatiotemporal graph convolutional network, introducing a physical constraint mechanism based on the heat conduction equation into the feature fusion layer of the dual-stream spatiotemporal graph convolutional network, calculating the spatiotemporal coupling degree between the visible light morphological change rate of the visible light morphological texture feature vector and the infrared temperature diffusion rate of the infrared temperature gradient distribution feature vector, using the spatiotemporal coupling degree as a dynamic weight adjustment factor, performing channel-by-channel weighted fusion of the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector to obtain a fused feature vector, suppressing pseudo-fire source feature responses that do not conform to the thermodynamic evolution law; and a classification and regression early warning device for inputting the fused feature vector into a classification and regression model, triggering an alarm signal generation logic to output a fire safety early warning command corresponding to the spatial coordinates when the calculated global fire confidence exceeds the early warning threshold.

[0006] Preferably, the feature extraction processor extracts the visible light morphological texture feature vector of the visible light image sequence and the infrared temperature gradient distribution feature vector of the infrared thermal imaging image sequence by: performing local binary mode operation and Gaussian difference operation on the visible light image sequence to extract visible light edge flicker frequency features and color space distribution features, and cascading them to obtain the visible light morphological texture feature vector; performing non-subsampled contour wave transform on the infrared thermal imaging image sequence to extract high-frequency temperature fluctuation components and low-frequency temperature field distribution components of multi-directional sub-bands, calculating the partial derivative matrix of the high-frequency temperature fluctuation components and low-frequency temperature field distribution components in the spatial gradient direction, and performing tensor decomposition on the partial derivative matrix to obtain the infrared temperature gradient distribution feature vector.

[0007] Preferably, the feature fusion early warning processor introduces a physical constraint mechanism based on the heat conduction equation into the feature fusion layer of the dual-stream spatiotemporal graph convolutional network. Calculating the spatiotemporal coupling degree between the visible light morphological change rate and the infrared temperature diffusion rate includes: mapping the visible light morphological texture feature vector to a morphological evolution spatial coordinate sequence, and mapping the infrared temperature gradient distribution feature vector to a temperature field diffusion spatial coordinate sequence; constructing a heat flux density constraint matrix between the morphological evolution spatial coordinate sequence and the temperature field diffusion spatial coordinate sequence based on Fourier's law of thermal conduction; calculating the product of the heat flux density constraint matrix in the time dimension and the feature trace in the spatial dimension of the dual-stream spatiotemporal graph convolutional network, and normalizing the product to obtain the spatiotemporal coupling degree.

[0008] Preferably, the feature fusion early warning processor uses the spatiotemporal coupling degree as a dynamic weight adjustment factor to perform channel-by-channel weighted fusion of the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector, including: broadcasting and expanding the spatiotemporal coupling degree along the feature channel dimension to obtain a channel weight matrix with the same dimension as the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector; performing a Hadamard product operation on the visible light morphological texture feature vector and the channel weight matrix to obtain a weighted visible light feature matrix; performing a Hadamard product operation on the infrared temperature gradient distribution feature vector and the complementary channel weight matrix to obtain a weighted infrared feature matrix, wherein the complementary channel weight matrix is ​​obtained by subtracting the channel weight matrix from the identity matrix; and concatenating the weighted visible light feature matrix and the weighted infrared feature matrix to obtain the fused feature vector.

[0009] Preferably, the feature fusion early warning processor suppresses pseudo-fire source feature responses that do not conform to the thermodynamic evolution law by: constructing a spatiotemporal adjacency matrix in the graph convolutional layer of the dual-stream spatiotemporal graph convolutional network, wherein the edge weights of the spatiotemporal adjacency matrix are determined by the thermodynamic energy attenuation coefficient between adjacent nodes; inputting the fused feature vector into the graph convolutional layer containing the spatiotemporal adjacency matrix for feature propagation; and performing feature amplitude clipping operation on feature nodes that do not satisfy the thermodynamic energy conservation constraint during feature propagation to filter out discrete feature responses in the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector that do not satisfy the evolution logic of the thermodynamic energy attenuation coefficient, thereby obtaining a de-interference fused feature vector.

[0010] Preferably, the classification and regression early warning device inputs the fused feature vector into a classification and regression model. When the calculated global fire confidence exceeds the early warning threshold, the alarm signal generation logic outputs a fire safety early warning instruction with corresponding spatial coordinates, including: inputting the fused feature vector into a fully connected layer composed of multilayer sensing mechanisms to calculate the initial fire confidence; performing temperature calibration on the initial fire confidence by combining the temperature amplitude of the infrared thermal imaging image sequence to obtain the global fire confidence; comparing the global fire confidence with the early warning threshold, and when the global fire confidence is greater than the early warning threshold, extracting the response peak coordinates of the fused feature vector in the visible light image sequence and the infrared thermal imaging image sequence, converting the response peak coordinates into three-dimensional spatial physical coordinates, and packaging them to generate the fire safety early warning instruction.

[0011] Preferably, performing local binary mode (LMM) and difference-of-Gaussian (DAG) operations on the visible light image sequence to extract visible light edge flicker frequency features and color space distribution features, and cascading them to obtain the visible light morphological texture feature vector, includes: constructing a multi-scale difference-of-Gaussian pyramid; extracting candidate edge contour features at each level of the multi-scale difference-of-Gaussian pyramid; performing LMM encoding on the candidate edge contour features; statistically analyzing the histogram distribution of the LMM encoding; mapping the histogram distribution to the visible light edge flicker frequency features; separating the Cr and Cb chromaticity components in the YCrCb color space; calculating the spatial co-occurrence probability matrix of the Cr and Cb chromaticity components; extracting the main diagonal eigenvalues ​​of the spatial co-occurrence probability matrix as the color space distribution features; and concatenating the visible light edge flicker frequency features and the color space distribution features to form the visible light morphological texture feature vector.

[0012] Preferably, the heat flux density constraint matrix between the morphological evolution spatial coordinate sequence and the temperature field diffusion spatial coordinate sequence is constructed based on Fourier's law of thermal conductivity. The product of the feature traces of the heat flux density constraint matrix in the time dimension and the feature traces in the spatial dimension of the two-stream spatiotemporal graph convolutional network is calculated, and the product is normalized to obtain the spatiotemporal coupling degree. This includes: calculating the temperature gradient vector of the temperature field diffusion spatial coordinate sequence on the spatial coordinate axis; multiplying the temperature gradient vector by a preset material thermal conductivity diagonal matrix to obtain the heat flux density vector; calculating the morphological feature change rate vector of the morphological evolution spatial coordinate sequence over time; solving the inner product operation result of the heat flux density vector and the morphological feature change rate vector; constructing the diagonal matrix of the inner product operation result as the heat flux density constraint matrix; extracting the time feature trace and the spatial feature trace of the heat flux density constraint matrix on the time axis; calculating the product of the time feature trace and the spatial feature trace; and inputting the product into the Sigmoid activation function to obtain the spatiotemporal coupling degree.

[0013] Preferably, the process of performing a Hadamard product operation on the visible light morphological texture feature vector and the channel weight matrix to obtain a weighted visible light feature matrix, performing a Hadamard product operation on the infrared temperature gradient distribution feature vector and the complementary channel weight matrix to obtain a weighted infrared feature matrix, and concatenating the weighted visible light feature matrix and the weighted infrared feature matrix to obtain the fused feature vector includes: performing a global average pooling operation on the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector to obtain visible light channel descriptors and infrared channel descriptors; inputting the visible light channel descriptors and the infrared channel descriptors into a gated recurrent network to output adaptive visible light weight factors and adaptive infrared weight factors; correcting the channel weight matrix using the adaptive visible light weight factors, correcting the complementary channel weight matrix using the adaptive infrared weight factors, and performing a Hadamard product operation and concatenation operation based on the corrected channel weight matrix and the complementary channel weight matrix.

[0014] Preferably, the fused feature vector is input into a graph convolutional layer containing the spatiotemporal adjacency matrix for feature propagation. During feature propagation, feature amplitude clipping is performed on feature nodes that do not satisfy the thermodynamic energy conservation constraint to filter out discrete feature responses in the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector that do not satisfy the evolution logic of the thermodynamic energy attenuation coefficient. Obtaining the de-interference fused feature vector includes: in each feature propagation iteration of the graph convolutional layer, calculating the sum of the feature energy of the current feature node and the input energy of the adjacent feature nodes; comparing the sum of the input energy with the expected thermodynamic attenuation energy calculated according to the thermodynamic energy conservation law; when the sum of the input energy is greater than the expected thermodynamic attenuation energy, clipping the feature amplitude of the current feature node to the feature amplitude corresponding to the expected thermodynamic attenuation energy, and removing the discrete feature responses to obtain the de-interference fused feature vector.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention introduces a physical constraint mechanism based on the heat conduction equation into the feature fusion layer of a dual-stream spatiotemporal graph convolutional network. It calculates the spatiotemporal coupling degree between the visible light morphological change rate and the infrared temperature diffusion rate, using this spatiotemporal coupling degree as a dynamic weight adjustment factor to perform channel-by-channel weighted fusion of the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector. This processing method embeds the physical evolution law of fire into the feature fusion weight calculation process, suppressing false fire source feature responses that do not conform to the thermodynamic evolution law, overcoming false correlations caused by single-mode interference sources, and reducing the false alarm rate of fires in complex environments.

[0016] 2. This invention uses physical constraints to dynamically assign and fuse multimodal features, which limits the feature search space of the neural network and reduces resource consumption and early warning delay during model operation. The channel-wise weighted fusion guided by physical constraints enhances the feature representation of early small fire sources in background noise, thereby improving the system's sensitivity and response reliability in identifying early fire sources. Attached Figure Description

[0017] Figure 1 This is a flowchart of the multimodal image synchronous acquisition process of the present invention; Figure 2 This is a flowchart of the visible light morphological texture feature vector extraction process of the present invention; Figure 3 This is a flowchart of the infrared temperature gradient distribution feature vector extraction process of the present invention; Figure 4 This is a flowchart of the spatiotemporal coupling degree calculation based on the heat conduction equation of the present invention. Detailed Implementation

[0018] This invention relates to the field of fire safety early warning technology, specifically to a fire safety early warning and fire source detection system based on multi-image feature fusion. The invention will be described in detail below with reference to several embodiments.

[0019] refer to Figure 1 In one embodiment, the multi-image feature fusion-based fire safety early warning and fire source detection system includes a multimodal image acquisition front-end, a feature extraction processor, a feature fusion early warning processor, and a classification and regression early warning unit. The multimodal image acquisition front-end controls the exposure time of the visible light imaging unit and the infrared thermal imaging unit through a unified trigger signal, adding the same millisecond-level timestamp to each frame to ensure strict temporal alignment between the visible light image sequence and the infrared thermal imaging image sequence. The frame rate of the image sequence remains consistent, and the time interval between adjacent frames is fixed. The multimodal image acquisition front-end transmits the synchronously acquired visible light image sequence and infrared thermal imaging image sequence to the feature extraction processor.

[0020] refer to Figure 2 The feature extraction processor receives image sequences transmitted from the multimodal image acquisition front-end and extracts the visible light morphological texture feature vector from the visible light image sequence and the infrared temperature gradient distribution feature vector from the infrared thermal imaging image sequence, respectively. For the visible light image sequence, the feature extraction processor first constructs a multi-scale Gaussian difference pyramid. The construction process of the multi-scale Gaussian difference pyramid follows the following formula:

[0021] in, This is a Gaussian difference image. For Gaussian kernel function, These are the initial scale parameters. As a scale factor, ( () represents the image pixel coordinates. In this embodiment, the scale factor... Pick The Difference-of-Gaussian (DOB) pyramid comprises four scale levels. At each level of the multi-scale DOB pyramid, the feature extraction processor extracts candidate edge contour features by detecting local extrema. Local extrema are defined as the points with the maximum or minimum pixel values ​​within the current scale's 3x3 neighborhood and the 3x3 neighborhoods of the adjacent scales above and below.

[0022] The feature extraction processor performs Local Binary Pattern Encoding (LDE) on the extracted candidate edge contour features. The calculation process of LDE follows the formula:

[0023] in, Center pixel coordinates for Local binary pattern encoded value, The grayscale value of the center pixel. For the first grayscale values ​​of neighboring pixels The number of neighboring pixels. For a sign function, when hour, ,when hour, In this embodiment, a 3x3 neighborhood is used. Take 8. Feature extraction processor statistical continuity. The frequency variation of local binary pattern encoding at the same location in a frame image is used to obtain the visible light edge flicker frequency characteristics. The value is determined based on the frame rate to ensure that the flickering characteristics of the flame edge can be captured.

[0024] Subsequently, the feature extraction processor converts the visible light image from the RGB color space to the YCrCb color space, separating the Y luminance component, the Cr red chromaticity component, and the Cb blue chromaticity component. The feature extraction processor then calculates the spatial co-occurrence probability matrix of the Cr and Cb chromaticity components. The calculation of the spatial co-occurrence probability matrix follows the formula below:

[0025] in, The first of the spatial co-occurrence probability matrices Line 1 Column elements, and For the width and height of the image, This is the spatial offset. The Kronecker function is defined when the two parameters are equal. ,otherwise , For pixels The chromaticity component values, For pixels The chromaticity component values. In this embodiment, the spatial offset is taken as... This involves calculating the spatial co-occurrence probability in the horizontal direction. The feature extraction processor extracts the eigenvalues ​​of the main diagonal of the spatial co-occurrence probability matrix as color space distribution features. The main diagonal eigenvalues ​​reflect the degree of spatial clustering of pixels with the same chromaticity value; the Cr chromaticity component in the flame region typically exhibits high clustering.

[0026] In this embodiment, the fire feature discrimination data of different color space components are shown in Table 1.

[0027] Table 1 Comparison of fire feature distinguishability across different color space components Y 0.124 0.087 1.425 Cr 0.356 0.042 8.476 Cb 0.218 0.065 3.354 R 0.187 0.079 2.367 G 0.093 0.091 1.022 B 0.076 0.095 0.800 As shown in Table 1, the Cr chromaticity component exhibits the largest inter-class variance, the smallest intra-class variance, and the highest signal-to-noise ratio, making it the most effective at distinguishing flame regions from background regions. The Cb chromaticity component shows the second highest discriminative power, while the Y luminance component and the R, G, and B components have relatively low discriminative power. Therefore, in this embodiment, the Cr and Cb chromaticity components are selected for color space distribution feature extraction. The feature extraction processor concatenates the visible light edge flicker frequency features with the color space distribution features to obtain a visible light morphological texture feature vector.

[0028] refer to Figure 3 For infrared thermal imaging image sequences, the feature extraction processor first performs a non-subsampled contourlet transform. The non-subsampled contourlet transform can extract features at different scales and orientations without reducing image resolution. The decomposition process of the non-subsampled contourlet transform follows the formula:

[0029] in, The input is an infrared thermal imaging image. The number of scales for decomposition. For the first Number of directions of scale For the first Scale No. The contour wave basis function of the direction. For scaling basis functions, These are high-frequency coefficients. These are low-frequency coefficients. In this embodiment, the number of decomposition scales... Take 3, the number of directions at each scale Take 8. The high-frequency temperature fluctuation component and the low-frequency temperature field distribution component of the multi-directional subband output by the non-subsampled profile wave transform. The high-frequency temperature fluctuation component reflects the local rapid changes in the temperature field, while the low-frequency temperature field distribution component reflects the overall distribution trend of the temperature field.

[0030] The feature extraction processor calculates the partial derivative matrices of the high-frequency temperature fluctuation component and the low-frequency temperature field distribution component along the spatial gradient direction. The calculation of the partial derivative matrices follows the formula:

[0031] in, For a moment , The temperature field gradient vector at that location, For a moment , Temperature value at that location, Let be the partial derivative of temperature in the x-direction. Let be the partial derivative of temperature in the y-direction. In this embodiment, the central difference method is used to calculate the partial derivative to improve calculation accuracy. For pixels at image boundaries, a mirror continuation method is used to avoid the influence of boundary effects.

[0032] The feature extraction processor performs tensor decomposition on the partial derivative matrix to obtain the feature vector of the infrared temperature gradient distribution. In this embodiment, singular value decomposition (SVD) is used to decompose the partial derivative matrix. SVD decomposes the partial derivative matrix into the product of three matrices: ,in and It is an orthogonal matrix. It is a diagonal matrix, and the elements on the diagonal are singular values. This represents the transpose. The feature extraction processor extracts the singular vectors corresponding to the top 16 largest singular values ​​and concatenates them to form the infrared temperature gradient distribution feature vector.

[0033] The feature extraction processor transmits the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector to the feature fusion warning processor. The feature fusion warning processor constructs a two-stream spatiotemporal graph convolutional network, which contains two parallel spatiotemporal graph convolutional branches, processing the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector respectively. Each branch contains three spatiotemporal graph convolutional layers, each with 64 output channels. In the feature fusion layer of the two-stream spatiotemporal graph convolutional network, the feature fusion warning processor introduces a physical constraint mechanism based on the heat conduction equation.

[0034] refer to Figure 4 The feature fusion early warning processor first maps visible light morphological texture feature vectors to a morphological evolution spatial coordinate sequence and infrared temperature gradient distribution feature vectors to a temperature field diffusion spatial coordinate sequence. This mapping process is achieved through linear transformation, where each feature vector corresponds to a coordinate point in three-dimensional space, and feature vectors from multiple consecutive frames constitute a spatial coordinate sequence. Then, the feature fusion early warning processor constructs a heat flux density constraint matrix between the morphological evolution spatial coordinate sequence and the temperature field diffusion spatial coordinate sequence based on Fourier's law of thermal conductivity. The expression for Fourier's law of thermal conductivity is:

[0035] in, For a moment , The heat flux density vector at that location, This is a pre-defined diagonal matrix of material thermal conductivity. In this embodiment, the material thermal conductivity is taken as the thermal conductivity of air.

[0036] The feature fusion early warning processor calculates the rate of change vector of morphological features as the spatial coordinate sequence of morphological evolution changes over time. The calculation process of the rate of change vector of morphological features follows the formula:

[0037] in, For time t, The vector of morphological feature change rate at the location, For a moment Location The spatial coordinate vector of morphological evolution, It is the partial derivative of the morphological evolution spatial coordinate vector with respect to time.

[0038] The feature fusion early warning processor calculates the inner product of the heat flux density vector and the morphological feature change rate vector, and constructs a diagonal matrix of the inner product result as the heat flux density constraint matrix. The construction process of the heat flux density constraint matrix follows the formula below:

[0039] in, For a moment The heat flux density constraint matrix, Constructs a diagonal matrix by using the elements of the vector as the diagonal elements of the matrix.

[0040] The feature fusion early warning processor extracts the temporal feature trace of the heat flux density constraint matrix on the time axis and the spatial feature trace on the spatial coordinate axis. The temporal feature trace is the sum of all diagonal elements of the heat flux density constraint matrix in the time dimension, and the spatial feature trace is the sum of all diagonal elements of the heat flux density constraint matrix in the spatial dimension. The feature fusion early warning processor calculates the product of the temporal and spatial feature traces and inputs it into the Sigmoid activation function to obtain the spatiotemporal coupling degree. The calculation process of the spatiotemporal coupling degree follows the following formula:

[0041] in, For a moment spatiotemporal coupling degree It is the Sigmoid activation function. For time features, Q This represents a spatial feature trace. The spatiotemporal coupling degree ranges from 0 to 1, reflecting the degree of thermodynamic coupling between visible light morphology changes and infrared temperature diffusion. A higher coupling degree indicates that the feature is more likely to correspond to a real flame; a lower coupling degree indicates that the feature is more likely to correspond to a false flame source.

[0042] The feature fusion early warning processor uses spatiotemporal coupling as a dynamic weight adjustment factor to perform channel-by-channel weighted fusion of visible light morphological texture feature vectors and infrared temperature gradient distribution feature vectors. First, the processor broadcasts and expands the spatiotemporal coupling along the feature channel dimensions to obtain a channel weight matrix with dimensions consistent with both the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector. Broadcasting and expansion refers to copying the scalar spatiotemporal coupling to each feature channel, forming a weight matrix with the same dimensions as the feature vectors. Then, the processor performs a Hadamard product operation between the visible light morphological texture feature vector and the channel weight matrix to obtain a weighted visible light feature matrix, and performs a Hadamard product operation between the infrared temperature gradient distribution feature vector and the complementary channel weight matrix to obtain a weighted infrared feature matrix. The complementary channel weight matrix is ​​obtained by subtracting the channel weight matrix from the identity matrix. Finally, the processor concatenates the weighted visible light feature matrix and the weighted infrared feature matrix to obtain the fused feature vector. The channel-by-channel weighted fusion process follows the formula:

[0043] in, This is a weighted fusion feature vector of visible light and infrared light. For Hadamard product operation, This is the channel weight matrix. To and Identity matrices of the same dimension This represents the visible light morphological texture feature vector. This is the characteristic vector of the infrared temperature gradient distribution.

[0044] Furthermore, the feature fusion warning processor performs a global average pooling operation on the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector to obtain visible light channel descriptors and infrared channel descriptors. The global average pooling operation averages all elements of each feature channel to obtain a scalar channel descriptor. The feature fusion warning processor inputs the visible light channel descriptors and infrared channel descriptors into a gated recurrent network and outputs adaptive visible light weight factors and adaptive infrared weight factors. The output process of the gated recurrent network follows the formula:

[0045]

[0046] in, This is the visible light channel descriptor. For infrared channel descriptors, For gated cyclic networks, For adaptive visible light weighting factors, An adaptive infrared weighting factor is used. In this embodiment, the gated recurrent network contains one hidden layer with 32 neurons. The feature fusion warning processor uses an adaptive visible light weighting factor to correct the channel weight matrix and an adaptive infrared weighting factor to correct the complementary channel weight matrix. It then performs Hadamard product operations and concatenation operations based on the corrected channel weight matrices and the complementary channel weight matrices. The gated recurrent network can dynamically adjust the weighting factor according to historical feature information, further improving the adaptability of feature fusion.

[0047] The feature fusion early warning processor suppresses pseudo-fire source feature responses that do not conform to thermodynamic evolution laws. In the graph convolutional layer of the dual-stream spatiotemporal graph convolutional network, the feature fusion early warning processor constructs a spatiotemporal adjacency matrix. This matrix considers both spatially and temporally adjacent nodes, and the edge weights are determined by the thermodynamic energy decay coefficient between adjacent nodes. The calculation of the thermodynamic energy decay coefficient follows this formula:

[0048] in, For nodes With nodes Thermodynamic energy decay coefficient between them For nodes and nodes Spatial distance between them For nodes and nodes The time difference between them The spatial attenuation coefficient, This is the time decay coefficient. In this embodiment, the spatial decay coefficient... Set to 10, time decay coefficient Take 0.5.

[0049] The feature fusion early warning processor inputs the fused feature vector into a graph convolutional layer containing a spatiotemporal adjacency matrix for feature propagation. The feature propagation process follows the formula:

[0050] in, For the first The feature matrix of the layer, The normalized spatiotemporal adjacency matrix, For the first The weight matrix of the layer, For the first The layer's bias vector, The ReLU activation function is used. The normalized spatiotemporal adjacency matrix is ​​calculated using the following formula: ,in This is the original spatiotemporal adjacency matrix. For degree matrix, This is the original spatiotemporal adjacency matrix.

[0051] During feature propagation, the feature fusion warning processor performs feature amplitude clipping on feature nodes that do not satisfy the thermodynamic energy conservation constraint. In each feature propagation iteration of the graph convolutional layer, the feature fusion warning processor calculates the sum of the feature energy of the current feature node and the input energy of its neighboring feature nodes. Feature energy is defined as the square of the L2 norm of the feature vector. The feature fusion warning processor compares the sum of the input energies with the expected thermodynamic decay energy calculated based on the thermodynamic energy conservation law. The expression for the thermodynamic energy conservation constraint is:

[0052] in, The characteristic energy of the current node, The sum of the input energies of adjacent nodes is used. When the sum of the input energies exceeds the expected thermodynamic decay energy, the feature fusion early warning processor trims the feature amplitude of the current feature node to the feature amplitude corresponding to the expected thermodynamic decay energy, eliminating discrete feature responses to obtain a de-interference fused feature vector. Pseudo-fire source features that do not conform to the thermodynamic evolution law have energy propagation speeds and decay patterns different from real flames; these pseudo-features can be effectively filtered out through energy conservation constraints.

[0053] The feature fusion early warning processor transmits the interference-free fused feature vector to the classification and regression early warning unit. The classification and regression early warning unit inputs the fused feature vector into the classification and regression model. When the calculated global fire confidence score exceeds the early warning threshold, it triggers an alarm signal generation logic outputting a fire safety early warning command with corresponding spatial coordinates. First, the classification and regression early warning unit inputs the fused feature vector into a fully connected layer composed of a multilayer perceptron to calculate the initial fire confidence score. In this embodiment, the multilayer perceptron contains two hidden layers, each with 128 and 64 neurons respectively. Each hidden layer uses the ReLU activation function, and the output layer uses the Sigmoid activation function, outputting an initial fire confidence score between 0 and 1.

[0054] Then, the classification and regression early warning system combines the temperature amplitude of the infrared thermal imaging image sequence to perform temperature calibration on the initial fire confidence level, thereby obtaining the global fire confidence level. The temperature calibration process follows the formula:

[0055] in, The initial fire confidence level. This represents the highest temperature value in the infrared thermal imaging image. This is the ambient temperature value. The reference flame temperature value is 500°C. In this embodiment, the reference flame temperature value is 500°C. Temperature calibration can further improve the accuracy of fire confidence by utilizing temperature information from infrared images, because the temperature of a real flame is usually higher than the ambient temperature, while the temperature of a spurious fire source may be lower.

[0056] The classification and regression early warning system compares the global fire confidence level with the early warning threshold. When the global fire confidence level is greater than the early warning threshold, the system extracts the peak response coordinates of the fused feature vector in the visible light image sequence and the infrared thermal imaging image sequence. The peak response coordinates correspond to the location with the largest feature response, i.e., the location of the fire. The system converts the image coordinates into three-dimensional physical coordinates using the camera's intrinsic and extrinsic parameters and packages them to generate a fire safety early warning command. The fire safety early warning command includes the three-dimensional spatial coordinates of the fire, the global fire confidence level, and the alarm time.

[0057] This embodiment provides a basic implementation method for a fire safety early warning and fire source detection system based on multi-image feature fusion. It covers the entire process of multimodal image acquisition, feature extraction, feature fusion under physical constraints, false fire source suppression, and classification regression early warning. By introducing a physical constraint mechanism based on the heat conduction equation, the physical evolution law of fire is embedded into the feature fusion process, thereby suppressing the feature response of false fire sources.

[0058] In a preferred embodiment, the multimodal image acquisition front-end operates in the same manner as in the aforementioned embodiments. When extracting visible light morphological texture feature vectors, the feature extraction processor performs multi-scale expansion of the local binary pattern encoding. The feature extraction processor performs local binary pattern encoding on images at different scales, and then concatenates the encoded histograms at different scales to obtain multi-scale visible light edge flicker frequency features. In this embodiment, three images at different scales are used: the original scale, a 2x downsampling scale, and a 4x downsampling scale. Multi-scale local binary pattern encoding can capture flame edge features of different sizes, improving the ability to identify small fire sources.

[0059] When extracting the infrared temperature gradient distribution feature vector, the feature extraction processor optimizes the decomposition parameters of the non-subsampled profiler transform. For large-space monitoring areas, the processor increases the number of decomposition scales to four to capture large-scale temperature field changes; for complex environments, it increases the number of directions per scale to 16 to capture temperature gradient changes in different directions. During the decomposition of the non-subsampled profiler transform, the processor assigns different weights to subbands in different directions, focusing on retaining the features of subbands with high contribution. This is to illustrate the impact of temperature fluctuation components in different directional subbands on fire source detection.

[0060] In this embodiment, the contribution data of temperature fluctuation components in different directional sub-bands to fire source detection are shown in Table 2.

[0061] Table 2. Contribution of temperature fluctuation components in different directional sub-bands to fire source detection 0° 0.087 0.124 0.012 45° 0.215 0.346 0.038 90° 0.103 0.157 0.015 135° 0.198 0.312 0.035 180° 0.079 0.112 0.011 225° 0.092 0.136 0.013 270° 0.085 0.121 0.012 315° 0.097 0.143 0.014 As shown in Table 2, the sub-bands in the 45° and 135° directions exhibit the largest feature variance, the highest mutual information with fire tags, and the most significant improvement in classification accuracy. This is because the edges of flames typically display an irregular, oblique distribution, and these sub-bands are better able to capture the edge features of the flames. Therefore, in this embodiment, the feature extraction processor assigns a weight of 1.5 to the sub-bands in the 45° and 135° directions and a weight of 1.0 to the sub-bands in other directions.

[0062] The feature extraction processor employs a higher-order central difference method to further improve computational accuracy when calculating the partial derivative matrix. The higher-order central difference method uses more neighboring pixels to calculate the partial derivatives, reducing the impact of noise on the calculation results. For pixels at image boundaries, a symmetric extension method is used to further reduce the influence of boundary effects. When performing tensor decomposition on the partial derivative matrix, the feature extraction processor uses non-negative matrix decomposition. Non-negative matrix decomposition yields non-negative eigenvectors, which better reflects the physical meaning of temperature gradients because the magnitude of a temperature gradient is non-negative. Non-negative matrix decomposition decomposes the partial derivative matrix into the product of two non-negative matrices: ,in As a basis matrix, This is the coefficient matrix. The feature extraction processor uses the column vectors of the basis matrix as feature vectors for the infrared temperature gradient distribution.

[0063] The feature fusion early warning processor and the classification regression early warning system operate in the same manner as in the aforementioned embodiments. This embodiment refines the feature extraction process, particularly the extraction of the infrared temperature gradient distribution feature vector. By adjusting the decomposition parameters of the non-subsampled contour wave transform and selecting directional sub-bands with high contribution, the quality of feature extraction is improved, further enhancing the system's ability to identify fire sources.

[0064] In another preferred embodiment, the multimodal image acquisition front-end and feature extraction processor operate in the same manner as in the aforementioned embodiments. When constructing the spatiotemporal adjacency matrix, the feature fusion early warning processor adjusts edge weights by incorporating node feature similarity. Node feature similarity is obtained by calculating the cosine similarity between the feature vectors of two nodes. The formula for calculating cosine similarity is: ,in and They are nodes and nodes The feature vectors are then used. The feature fusion early warning processor multiplies the feature similarity of nodes by the thermodynamic energy decay coefficient to obtain the final edge weights. This allows the spatiotemporal adjacency matrix to reflect not only the adjacency relationships in the physical space but also the similarity relationships in the feature space, further improving the accuracy of feature propagation.

[0065] The feature fusion early warning processor adaptively adjusts the spatial and temporal decay coefficients when calculating the thermodynamic energy decay coefficient. It pre-collects flame data under different environmental conditions and trains a regression model to predict suitable decay coefficients. The input to the regression model is environmental parameters of the monitored area, including temperature, humidity, and wind speed; the output is the spatial and temporal decay coefficients. During system operation, the feature fusion early warning processor acquires environmental parameters of the monitored area in real time and inputs them into the regression model to obtain adaptive decay coefficients. For example, in well-ventilated areas, the flame energy propagation speed is faster, so the temporal decay coefficient is appropriately reduced; in areas with many obstacles, the flame energy propagation speed is slower, so the temporal decay coefficient is appropriately increased.

[0066] The feature fusion early warning processor employs soft clipping when performing feature amplitude clipping. The calculation formula for soft clipping is: ,in The original characteristic amplitude, For the thermodynamic decay expected energy, The sum of the input energies is used. When the sum of the input energies exceeds the expected thermodynamic decay energy, soft clipping reduces the characteristic amplitude below the expected thermodynamic decay energy by a certain proportion. Compared with hard clipping, soft clipping can retain more characteristic information and avoids excessive suppression of the characteristics of the real flame.

[0067] In this embodiment, the data on the suppression effect of pseudo-fire source features under different interference sources are shown in Table 3.

[0068] Table 3. Suppression effect of false fire source characteristics under different interference sources. High temperature equipment 0.187 0.032 0.945 strong light reflection 0.213 0.041 0.932 Smoke Shadow 0.156 0.028 0.951 Moving objects 0.124 0.023 0.963 As shown in Table 3, the false alarm rate under various interference sources was significantly reduced after the introduction of the physical constraint mechanism, while the feature retention rate remained at a high level. This indicates that the mechanism can effectively filter out false fire source features while retaining the features of real flames.

[0069] When extracting response peak coordinates, the classification and regression early warning system employs a non-maximum suppression algorithm (NMS) to remove duplicate response peaks. The NMS algorithm consists of the following steps: sorting all response peaks from highest to lowest confidence level; selecting the response peak with the highest confidence level and adding it to the final coordinate list; removing other response peaks whose overlap with this peak exceeds a threshold; and repeating this process until all response peaks have been processed. In this embodiment, the overlap threshold is set to 0.5. The NMS algorithm ensures that each fire source outputs only one coordinate, avoiding duplicate alarms.

[0070] This embodiment refines the false fire source feature suppression part. By adaptively adjusting the edge weights and thermodynamic energy attenuation coefficients of the spatiotemporal adjacency matrix, and by employing soft clipping operations, the effect of false fire source feature suppression is further improved, and the false alarm rate in complex environments is reduced.

Claims

1. A fire safety early warning and fire source detection system based on multi-image feature fusion, characterized in that, include: A multimodal image acquisition front-end is used to simultaneously acquire visible light image sequences and infrared thermal imaging image sequences of the monitored area; A feature extraction processor is used to extract the visible light morphological texture feature vector of the visible light image sequence and the infrared temperature gradient distribution feature vector of the infrared thermal imaging image sequence. A feature fusion early warning processor is used to construct a dual-stream spatiotemporal graph convolutional network. A physical constraint mechanism based on the heat conduction equation is introduced into the feature fusion layer of the dual-stream spatiotemporal graph convolutional network to calculate the spatiotemporal coupling degree between the visible light morphology change rate of the visible light morphological texture feature vector and the infrared temperature diffusion rate of the infrared temperature gradient distribution feature vector. The spatiotemporal coupling degree is used as a dynamic weight adjustment factor to perform channel-wise weighted fusion of the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector to obtain a fused feature vector and suppress pseudo-fire source feature responses that do not conform to the thermodynamic evolution law. The classification and regression early warning device is used to input the fused feature vector into the classification and regression model. When the calculated global fire confidence exceeds the early warning threshold, it triggers the alarm signal generation logic output to provide a fire safety early warning instruction with corresponding spatial coordinates. The feature fusion early warning processor introduces a physical constraint mechanism based on the heat conduction equation into the feature fusion layer of the dual-stream spatiotemporal graph convolutional network. The calculation of the spatiotemporal coupling degree between the visible light morphological change rate and the infrared temperature diffusion rate includes: mapping the visible light morphological texture feature vector to a morphological evolution space coordinate sequence, and mapping the infrared temperature gradient distribution feature vector to a temperature field diffusion space coordinate sequence. A heat flux density constraint matrix is ​​constructed between the morphological evolution spatial coordinate sequence and the temperature field diffusion spatial coordinate sequence based on Fourier's law of thermal conductivity. Calculate the product of the feature traces of the heat flux density constraint matrix in the time dimension and the feature traces in the spatial dimension of the two-stream spatiotemporal graph convolutional network, and normalize the product to obtain the spatiotemporal coupling degree; The feature fusion early warning processor suppresses pseudo-fire source feature responses that do not conform to the thermodynamic evolution law by: constructing a spatiotemporal adjacency matrix in the graph convolutional layer of the dual-stream spatiotemporal graph convolutional network, wherein the edge weights of the spatiotemporal adjacency matrix are determined by the thermodynamic energy attenuation coefficient between adjacent nodes; The fused feature vector is input into a graph convolutional layer containing the spatiotemporal adjacency matrix for feature propagation. During feature propagation, feature amplitude clipping is performed on feature nodes that do not satisfy the thermodynamic energy conservation constraint to filter out discrete feature responses in the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector that do not satisfy the evolution logic of the thermodynamic energy attenuation coefficient, thereby obtaining the interference-free fused feature vector.

2. The fire safety early warning and fire source detection system based on multi-image feature fusion according to claim 1, characterized in that, The feature extraction processor extracts the visible light morphological texture feature vector of the visible light image sequence and the infrared temperature gradient distribution feature vector of the infrared thermal imaging image sequence by: performing local binary mode operation and Gaussian difference operation on the visible light image sequence, extracting visible light edge flicker frequency features and color space distribution features, and cascading them to obtain the visible light morphological texture feature vector; The infrared thermal imaging image sequence is subjected to non-downsampled contour wave transform to extract the high-frequency temperature fluctuation component and the low-frequency temperature field distribution component of the multi-directional sub-band. The partial derivative matrix of the high-frequency temperature fluctuation component and the low-frequency temperature field distribution component in the spatial gradient direction is calculated. The partial derivative matrix is ​​then subjected to tensor decomposition to obtain the infrared temperature gradient distribution feature vector.

3. The fire safety early warning and fire source detection system based on multi-image feature fusion according to claim 1, characterized in that, The feature fusion early warning processor uses the spatiotemporal coupling degree as a dynamic weight adjustment factor to perform channel-by-channel weighted fusion of the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector, including: broadcasting and expanding the spatiotemporal coupling degree along the feature channel dimension to obtain a channel weight matrix consistent with the dimensions of the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector. The weighted visible light feature matrix is ​​obtained by performing a Hadamard product operation between the visible light morphological texture feature vector and the channel weight matrix. The weighted infrared feature matrix is ​​obtained by performing a Hadamard product operation on the infrared temperature gradient distribution feature vector and the complementary channel weight matrix. The complementary channel weight matrix is ​​obtained by subtracting the channel weight matrix from the identity matrix. The fused feature vector is obtained by concatenating the weighted visible light feature matrix and the weighted infrared feature matrix.

4. The fire safety early warning and fire source detection system based on multi-image feature fusion according to claim 1, characterized in that, The classification and regression early warning device inputs the fused feature vector into the classification and regression model. When the calculated global fire confidence exceeds the early warning threshold, it triggers the alarm signal generation logic to output the fire safety early warning instruction corresponding to the spatial coordinates. This includes: inputting the fused feature vector into a fully connected layer composed of a multilayer sensing mechanism to calculate the initial fire confidence. The initial fire confidence level is calibrated by combining the temperature amplitude of the infrared thermal imaging image sequence to obtain the global fire confidence level; By comparing the global fire confidence score with the warning threshold, when the global fire confidence score is greater than the warning threshold, the response peak coordinates of the fused feature vector in the visible light image sequence and the infrared thermal imaging image sequence are extracted, the response peak coordinates are converted into three-dimensional spatial physical coordinates and packaged to generate the fire safety warning instruction.

5. The fire safety early warning and fire source detection system based on multi-image feature fusion according to claim 2, characterized in that, The visible light image sequence is subjected to local binary mode operation and Gaussian difference operation to extract visible light edge flicker frequency features and color space distribution features, and the visible light morphological texture feature vector is obtained by cascading: constructing a multi-scale Gaussian difference pyramid, and extracting candidate edge contour features at each level of the multi-scale Gaussian difference pyramid; Local binary pattern encoding is performed on the candidate edge contour features, the histogram distribution of the local binary pattern encoding is statistically analyzed, and the histogram distribution is mapped to the visible light edge flicker frequency features; In the YCrCb color space, the Cr chromaticity component and the Cb chromaticity component are separated, the spatial co-occurrence probability matrix of the Cr chromaticity component and the Cb chromaticity component is calculated, and the eigenvalues ​​of the main diagonal of the spatial co-occurrence probability matrix are extracted as the distribution features of the color space. The visible light edge flicker frequency feature and the color space distribution feature are combined to form the visible light morphological texture feature vector.

6. The fire safety early warning and fire source detection system based on multi-image feature fusion according to claim 1, characterized in that, Based on Fourier's law of thermal conductivity, a heat flux density constraint matrix is ​​constructed between the morphological evolution spatial coordinate sequence and the temperature field diffusion spatial coordinate sequence. The product of the feature traces of the heat flux density constraint matrix in the time dimension and the feature traces in the spatial dimension of the two-stream spatiotemporal graph convolutional network is calculated, and the spatiotemporal coupling degree is obtained by normalizing the product. This includes: calculating the temperature gradient vector of the temperature field diffusion spatial coordinate sequence on the spatial coordinate axis, and multiplying the temperature gradient vector with a preset material thermal conductivity diagonal matrix to obtain the heat flux density vector. Calculate the morphological feature change rate vector of the morphological evolution spatial coordinate sequence over time; Solve the inner product of the heat flux density vector and the morphological characteristic change rate vector, and construct the diagonal matrix of the inner product result as the heat flux density constraint matrix; Extract the temporal feature trace of the heat flux density constraint matrix on the time axis and the spatial feature trace on the spatial coordinate axis, calculate the product of the temporal feature trace and the spatial feature trace, and input it into the Sigmoid activation function to obtain the spatiotemporal coupling degree.

7. The fire safety early warning and fire source detection system based on multi-image feature fusion according to claim 3, characterized in that, The process of obtaining the weighted visible light feature matrix by performing a Hadamard product operation on the visible light morphological texture feature vector and the channel weight matrix, and obtaining the weighted infrared feature matrix by performing a Hadamard product operation on the infrared temperature gradient distribution feature vector and the complementary channel weight matrix, and concatenating the weighted visible light feature matrix and the weighted infrared feature matrix to obtain the fused feature vector includes: performing a global average pooling operation on the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector to obtain a visible light channel descriptor and an infrared channel descriptor; The visible light channel descriptor and the infrared channel descriptor are input into a gated recurrent network, and the adaptive visible light weighting factor and the adaptive infrared weighting factor are output. The channel weight matrix is ​​corrected using the adaptive visible light weight factor, and the complementary channel weight matrix is ​​corrected using the adaptive infrared weight factor. Hadamard product operation and splicing operation are then performed based on the corrected channel weight matrix and the complementary channel weight matrix.

8. The fire safety early warning and fire source detection system based on multi-image feature fusion according to claim 1, characterized in that, The fused feature vector is input into a graph convolutional layer containing the spatiotemporal adjacency matrix for feature propagation. During feature propagation, feature amplitude clipping is performed on feature nodes that do not satisfy the thermodynamic energy conservation constraint to filter out discrete feature responses in the visible light morphological texture feature vector and the infrared temperature gradient distribution feature vector that do not satisfy the evolution logic of the thermodynamic energy attenuation coefficient. Obtaining the de-interference fused feature vector includes: in each feature propagation iteration of the graph convolutional layer, calculating the sum of the feature energy of the current feature node and the input energy of the adjacent feature nodes; Compare the sum of the input energies with the expected thermodynamic decay energy calculated based on the thermodynamic law of conservation of energy; When the sum of the input energies is greater than the expected thermodynamic decay energy, the feature amplitude of the current feature node is clipped to the feature amplitude corresponding to the expected thermodynamic decay energy, and the discrete feature response is removed to obtain the interference-free fusion feature vector.

Images