Fire smoke image early recognition method based on multi-modal fusion

Abstract

The invention discloses a fire smoke image early recognition method based on multi-modal fusion. The method comprises the following steps: S1, collecting multi-modal data and carrying out time alignment; s2, constructing a visible light branch, an infrared branch and a photoacoustic branch, generating multi-modal features, and storing the multi-modal features in a shared feature pool; s3, embedding a space-time tracking module in the shared feature pool, and generating global feature representation; s4, generating a dynamic background template by adopting a low-rank matrix decomposition algorithm, and separating and removing background noise; s5, multi-modal features are fused in the shared feature pool through a lightweight self-attention mechanism, and missing features are compensated; s6, enhancing an early smoke signal through a sparse feature enhancement algorithm, and performing fire state classification and smoke diffusion path prediction by applying a multi-task classification network; and S7, updating the shared feature pool and the space-time tracking module through a dynamic feedback mechanism. According to the invention, early-stage accurate identification of fire smoke is realized by using multi-modal fusion, space-time tracking and dynamic feedback methods.

Description

Technical Field

[0001] The present invention relates to the technical field of fire monitoring, and in particular to an early recognition method of fire smoke images based on multi-modal fusion. Background Art

[0002] Fire, as a common disaster, not only poses a huge threat to the safety of life and property, but also has a long-term destructive impact on the environment. Traditional fire monitoring methods mainly rely on equipment such as smoke sensors, temperature sensors or single optical cameras. These methods usually have problems such as poor real-time performance, limited coverage, and weak adaptability to complex environments. Especially in the early stages of a fire, smoke signals are often sparse and not obvious, and traditional single-modality monitoring equipment is difficult to effectively identify them, resulting in high lag and false alarm rates in fire alarms. As fire monitoring scenarios become increasingly complex, there is an urgent need for a technical solution that can integrate information from multiple data sources and achieve high-precision fire monitoring.

[0003] In recent years, multimodal data fusion technology has been gradually applied to the field of fire monitoring. Multimodal technology integrates and analyzes data from different sources such as visible light, infrared spectrum, sound, and gas sensors in order to overcome the problem of insufficient information in single modal data. However, the existing multimodal fusion technology still has some obvious limitations. First, the time difference and spatial distribution inconsistency between data acquisition devices lead to feature alignment deviation in the data fusion process, which affects the data integration effect; secondly, due to the diversity and complexity of multimodal data itself, the existing technology lacks a fine processing mechanism for the characteristics of each modal data during the fusion process, which may cause key information to be diluted or even lost, making it difficult to fully reflect the characteristics of the fire. In addition, there is a certain degree of noise and missing in multimodal data, especially in complex environmental scenarios (such as industrial sites or high humidity environments). Data missing may have a significant impact on the monitoring results, and the existing compensation methods are mostly static rules, which are difficult to adapt to dynamic environmental changes in real time.

[0004] Another technical bottleneck is that the smoke characteristics in the early stages of a fire are often sparse and change rapidly. Existing technologies usually use static image processing algorithms or traditional machine learning methods, which have poor capture capabilities for early, sparse smoke signals and are prone to missed or false alarms due to noise interference. At the same time, traditional methods find it difficult to effectively model the diffusion path of smoke and changes in the location of fire sources. Fire monitoring requires not only accurate identification of the occurrence of a fire, but also real-time positioning of the fire source and prediction of its future diffusion trend, which is of great significance for optimizing firefighting strategies and improving emergency response efficiency. However, existing technologies lack effective models and technical support for fire source positioning and diffusion path prediction.

[0005] In the existing technology, some studies have attempted to model fire characteristics by combining time series analysis models, such as long short-term memory networks (LSTM) and other methods. However, these methods usually have limited modeling capabilities for time series, especially in the case of multimodal data, and cannot effectively capture the dynamic interaction relationship between different modes. Some other studies have attempted to introduce attention mechanisms to improve the efficiency of feature selection and fusion, but due to their high complexity and high computational cost, it is difficult to achieve real-time requirements in resource-constrained scenarios. In addition, most of the current methods rely on static background modeling technology for background noise removal, which is difficult to adapt to the interference of dynamic backgrounds in fire scenes (such as light changes, wind flow effects, etc.).

[0006] Therefore, how to provide an early recognition method for fire smoke images based on multimodal fusion is an urgent problem that technicians in this field need to solve. Summary of the invention

[0007] One purpose of the present invention is to propose an early recognition method for fire smoke images based on multimodal fusion. The present invention uses advanced methods such as multimodal fusion, spatiotemporal tracking, low-rank matrix decomposition and dynamic feedback to comprehensively utilize visible light, infrared and photoacoustic spectral data to achieve early and accurate recognition of fire smoke. Through sparse feature enhancement algorithm and lightweight self-attention mechanism, the present invention effectively improves the capture ability of smoke signals and solves the problems of complex background interference and data loss. It has the advantages of strong real-time performance, high recognition accuracy and strong environmental adaptability. It is suitable for complex fire monitoring scenes such as industrial plants and large public buildings, and can significantly reduce the missed alarm rate and false alarm rate of fire.

[0008] According to an embodiment of the present invention, a method for early recognition of fire smoke images based on multimodal fusion includes the following steps:

[0009] S1. Collect multimodal data through a visible light camera, an infrared camera and a photoacoustic spectroscopy sensor, and perform time alignment on the multimodal data based on timestamp synchronization;

[0010] S2, constructing a visible light branch, an infrared branch, and a photoacoustic branch, respectively extracting smoke diffusion morphology, temperature gradient, and dynamic change characteristics of gas composition from the multimodal data, generating multimodal features, and storing them in a shared feature pool;

[0011] S3, embedding a spatiotemporal tracking module in the shared feature pool to capture the changing rules of the multimodal features in the time axis and spatial distribution, and generating a global feature representation with spatiotemporal correlation characteristics;

[0012] S4. Based on the global feature representation, a low-rank matrix decomposition algorithm is used to generate a dynamic background template, separate and remove background noise, and dynamically optimize the multimodal features in the shared feature pool;

[0013] S5. The multimodal features are fused in the shared feature pool and the missing features are compensated through the lightweight self-attention mechanism to generate a complete multimodal feature vector.

[0014] S6. Based on the complete multimodal feature vector, the early smoke signal is enhanced by a sparse feature enhancement algorithm, and a multi-task classification network is applied to perform fire state classification and smoke diffusion path prediction;

[0015] S7. According to the real-time environmental data deviation, the shared feature pool and spatiotemporal tracking module are updated through a dynamic feedback mechanism to optimize the fire source positioning accuracy.

[0016] Optionally, the S2 specifically includes:

[0017] S21, collecting visible light image data through a visible light camera to construct a visible light branch, wherein the visible light image data is used to obtain diffusion morphological characteristics of smoke, including edge characteristics, diffusion speed, and diffusion direction of smoke;

[0018] S22, collecting infrared image data through an infrared camera to construct an infrared branch, wherein the infrared image data is used to obtain temperature gradient characteristics of the smoke area, including temperature changes in the smoke diffusion area and intensity of the hot spot area;

[0019] S23, collecting gas composition data through a photoacoustic spectroscopy sensor to construct a photoacoustic branch, wherein the gas composition data is used to obtain dynamic change characteristics of gas in smoke;

[0020] S24, standardize the features extracted from each branch, generate multimodal features and store them in a shared feature pool.

[0021] Optionally, the S3 specifically includes:

[0022] S31, constructing a spatiotemporal tracking module, wherein the spatiotemporal tracking module is embedded in the shared feature pool, and the spatiotemporal tracking module includes a temporal convolutional network and a spatiotemporal attention mechanism, wherein the temporal convolutional network is used to capture the changing law of multimodal features in the temporal dimension, and the spatiotemporal attention mechanism is used to capture the distribution characteristics of multimodal features in the spatial dimension;

[0023] S32, applying a temporal convolutional network to perform temporal modeling on the multimodal features, and generating a temporal feature representation at each time step:

[0024] T(t)=σ(W t *X(t)+b t );

[0025] Among them, T(t) represents the temporal feature representation of time step t, σ represents the activation function, and Wt represents the time convolution kernel weight, * represents the convolution operation, X(t) represents the multimodal feature input at time step t, and b t represents the bias vector;

[0026] S33, applying the spatiotemporal attention mechanism to perform weighted processing on the multimodal features in the spatial dimension, and generating a spatial feature representation at each spatial position:

[0027]

[0028] Among them, A(s) represents the spatial feature representation of spatial position s, Q(s) represents the query matrix of spatial position s, K(s) represents the key matrix of spatial position s, V(s) represents the value matrix of spatial position s, Softmax represents the normalization operation, and d k represents the dimension of the key vector;

[0029] S34, fuse the temporal feature representation T(t) and the spatial feature representation A(s) to generate a global feature representation with temporal and spatial correlation:

[0030]

[0031] Among them, G(t,s) represents the global feature representation with spatiotemporal correlation, α, β and γ represent weight parameters, Represents the tensor product operation, which is used to capture the interaction between temporal and spatial features.

[0032] Optionally, the S4 specifically includes:

[0033] S41. Based on the global feature representation, applying a low-rank matrix decomposition algorithm to decompose the global feature representation into a low-rank matrix L and a sparse matrix S;

[0034] S42. Optimize the low-rank matrix L and the sparse matrix S by using the nuclear norm minimization method:

[0035] min L,S ∥L∥ * +λ∥S∥1;

[0036] Among them, ∥L∥ * represents the nuclear norm of the low-rank matrix L, |S||1 represents the L1 norm of the sparse matrix S, and λ represents the regularization parameter;

[0037] S43, using the alternating direction multiplier method to iterate and solve, the specific update rule is:

[0038]

[0039] Y k+1 =Y k+G(t,s)-L k+1 -S k+1 ;

[0040] Among them, L k+1 represents the low-rank matrix in the k+1th iteration, S k+1 represents the sparse matrix in the k+1th iteration, Y k +1 represents the Lagrange multiplier in the k+1th iteration, ∥·∥ F represents the Frobenius norm, S k represents the sparse matrix in the kth iteration, Y k represents the Lagrangian multiplier in the kth iteration, ρ represents the augmented Lagrangian parameter, and G(t,s) represents the global feature representation with spatiotemporal correlation;

[0041] S44. After completing the iteration, obtain the optimized low-rank matrix L * And the optimized sparse matrix S * , and remove the background noise from the global feature representation to obtain the denoised feature representation:

[0042] G′(t,s)=G(t,s)-L * ;

[0043] Among them, G'(t,s) represents the denoising feature representation;

[0044] S45. Dynamically optimize the denoising feature representation G'(t,s), and use a weighting mechanism based on the gradient boosting decision tree to adjust the weights of the multimodal features in the shared feature pool:

[0045]

[0046] in, represents the updated weight of the i-th modal feature in the shared feature pool, represents the original weight of the i-th modal feature in the shared feature pool, exp represents the exponential function, η represents the learning rate, Represents the gradient of the loss function J;

[0047] S46. Update the optimized denoising feature representation G'(t,s) back to the shared feature pool.

[0048] Optionally, the S5 specifically includes:

[0049] S51, constructing a lightweight self-attention mechanism module, wherein the lightweight self-attention mechanism module includes a multi-head attention unit and a feature reconstruction unit for real-time fire monitoring;

[0050] S52, inputting the extracted multimodal feature vector into the multi-head attention unit in the lightweight self-attention mechanism module, performing correlation analysis between the multimodal features, dynamically calculating the importance weight of each modal feature through the self-attention mechanism, and generating a weighted feature representation;

[0051] S53, reconstructing the weighted feature representation using a feature reconstruction unit to generate an enhanced modal feature vector, wherein the reconstruction process includes feature fusion and redundant information suppression;

[0052] S54, adopting the self-attention compensation strategy based on context dependency, by analyzing the correlation between existing modal features, inferring and compensating the feature information of the missing modality, and generating a complete multimodal feature vector:

[0053]

[0054] Among them, F complete represents the complete multimodal feature vector, ω represents the nonlinear transformation function, represents the concatenation operation, Q i represents the query matrix of the i-th modal feature, K i represents the bond matrix of the i-th modal feature, V i represents the value matrix of the i-th modal feature, n represents the total number of modes, M represents the index set of missing modes, θ j represents the compensation weight factor of the jth missing mode, C j (X j ) represents the compensatory feature generation function of the jth missing mode.

[0055] Optionally, the S6 specifically includes:

[0056] S61, using the complete multimodal feature vector as input to the sparse feature enhancement algorithm, and initializing the regularization parameter and the sparsity threshold;

[0057] S62. The complete multimodal feature vector is sparsely processed by a sparse feature enhancement algorithm. A sparse coding mechanism is used to represent each modal feature as a linear combination of the original features. The optimization goal is to minimize the reconstruction error while maintaining the sparsity of the modal feature representation.

[0058] S63, solving the sparse coding matrix by an iterative optimization method, and combining it with the feature dictionary to generate an enhanced sparse feature matrix;

[0059] S64, inputting the enhanced sparse feature matrix into a multi-task classification network, wherein task 1 is fire state classification, and task 2 is smoke diffusion path prediction. The multi-task classification network extracts common features through a shared layer, and processes feature representations of the two tasks in a task-specific layer, thereby forming a parallel structure of classification and prediction.

[0060] S65. Define the objective function of multi-task classification and perform joint optimization in combination with the sparse feature matrix. The objective function is expressed as the weighted sum of the two task loss functions, where the weight coefficient is dynamically adjusted according to the importance of the task.

[0061] S66. Generate fire state classification results and smoke diffusion path prediction results through a multi-task classification network.

[0062] Optionally, the S7 specifically includes:

[0063] S71, collecting environmental data in real time, and calculating the deviation matrix E between the multimodal feature data of the current fire monitoring and the prediction results;

[0064] S72. Based on the deviation matrix E, the feature weights in the shared feature pool are optimized and adjusted through a dynamic feedback mechanism:

[0065]

[0066] in, represents the updated weight of the i-th modal feature in the shared feature pool, represents the original weight of the i-th modality feature in the shared feature pool, η1 represents the learning rate, represents the gradient of the deviation matrix with respect to the modal feature weight;

[0067] S73, inputting the optimized shared feature pool into the embedded spatiotemporal tracking module to update and calculate the fire source location;

[0068] S74, predicting the location of the fire source at a future time according to the optimized fire source location and combining environmental characteristics, and generating trajectory prediction results, including the future fire source location and diffusion range;

[0069] S75, evaluating the trajectory prediction results, monitoring the error change trend in real time, judging whether the prediction accuracy reaches the expected level by the change amplitude of the deviation matrix E, and further optimizing the parameters of the spatiotemporal tracking module to improve the positioning accuracy if the amplitude of the deviation matrix E exceeds the set threshold;

[0070] S76. Feedback the optimized trajectory prediction results to the spatiotemporal tracking module of the shared feature pool, thereby improving the response speed and accuracy to changes in the fire source location in the next round of monitoring, forming a dynamically optimized closed-loop feedback mechanism.

[0071] The beneficial effects of the present invention are:

[0072] First, the present invention realizes the comprehensive extraction of dynamic change characteristics of smoke diffusion morphology, temperature gradient and gas composition by fusing three modal data of visible light, infrared and photoacoustic spectroscopy, overcoming the problems of incomplete information and low recognition accuracy in traditional single modal methods. By constructing a shared feature pool to uniformly store multimodal features and embedding a spatiotemporal tracking module to capture the dynamic changes of multimodal features in time and space dimensions, the present invention can efficiently model the spatiotemporal correlation characteristics of smoke signals. Compared with the prior art method of extracting only static features, the present invention provides a solution that dynamically adapts to the complex changes in the early stages of a fire, significantly improving the accuracy of fire source positioning and the ability to predict smoke diffusion paths.

[0073] Secondly, the present invention generates a dynamic background template by adopting a low-rank matrix decomposition algorithm, removes background noise from the global feature representation, and effectively addresses complex background problems such as light changes and wind interference in fire monitoring scenes. Unlike traditional static background modeling methods, the present invention can dynamically adjust the background template according to the real-time changes of multimodal data, thereby ensuring the stability and accuracy of feature extraction. In addition, the sparse feature enhancement algorithm targets the characteristics of sparse smoke signals in early fires, and combines a sparse coding mechanism to amplify key features, thereby avoiding the problem of weak smoke signals being masked by background interference. Through this feature enhancement process, the present invention significantly reduces the underreporting rate and false alarm rate, providing strong support for the accurate identification of early fires.

[0074] In addition, the present invention introduces a lightweight self-attention mechanism, which effectively solves the dynamic optimization problem of feature weight allocation in the process of multimodal data fusion. By weighting multimodal features through the self-attention mechanism, the present invention can adaptively highlight the importance of different modalities in different scenarios, thereby improving the efficiency of feature fusion. At the same time, the context compensation strategy is adopted to solve the problem of missing multimodal data, ensuring the integrity of feature fusion and the robustness of the system. Compared with traditional static fusion methods, the present invention has significant improvements in fusion efficiency and missing data compensation capabilities, especially in scenarios with complex environments and incomplete data, showing stronger adaptability.

[0075] Finally, the present invention constructs a closed-loop optimization system through a dynamic feedback mechanism, compares the real-time collected environmental data with the system prediction results, dynamically adjusts the parameters of the shared feature pool and the spatiotemporal tracking module, and realizes the continuous optimization of the fire source positioning accuracy. This mechanism can correct the prediction deviation in real time to ensure the adaptability and accuracy of the system in a dynamic environment. In addition, by combining the fire state classification and smoke diffusion path prediction tasks, the present invention realizes comprehensive monitoring of the development of the fire and provides forward-looking support for fire fighting decisions. The overall system shows significant advantages in accuracy, response speed and robustness, and is particularly suitable for complex scenes such as industrial plants and large public buildings. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0077] Figure 1 This is an overall flow chart of a method for early recognition of fire smoke images based on multimodal fusion proposed by the present invention;

[0078] Figure 2 This is a schematic diagram of the shared feature pool structure of the embedded spatiotemporal tracking module of the fire smoke image early recognition method based on multimodal fusion proposed in the present invention. DETAILED DESCRIPTION

[0079] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0080] refer to Figure 1 and Figure 2 , a fire smoke image early recognition method based on multimodal fusion, comprising the following steps:

[0081] S1. Collect multimodal data through a visible light camera, an infrared camera and a photoacoustic spectroscopy sensor, and perform time alignment on the multimodal data based on timestamp synchronization;

[0082] S2, constructing a visible light branch, an infrared branch, and a photoacoustic branch, respectively extracting smoke diffusion morphology, temperature gradient, and dynamic change characteristics of gas composition from the multimodal data, generating multimodal features, and storing them in a shared feature pool;

[0083] S3, embedding a spatiotemporal tracking module in the shared feature pool to capture the changing rules of the multimodal features in the time axis and spatial distribution, and generating a global feature representation with spatiotemporal correlation characteristics;

[0084] S4. Based on the global feature representation, a low-rank matrix decomposition algorithm is used to generate a dynamic background template, separate and remove background noise, and dynamically optimize the multimodal features in the shared feature pool;

[0085] S5. The multimodal features are fused in the shared feature pool and the missing features are compensated through the lightweight self-attention mechanism to generate a complete multimodal feature vector.

[0086] S6. Based on the complete multimodal feature vector, the early smoke signal is enhanced by a sparse feature enhancement algorithm, and a multi-task classification network is applied to perform fire state classification and smoke diffusion path prediction;

[0087] S7. According to the real-time environmental data deviation, the shared feature pool and spatiotemporal tracking module are updated through a dynamic feedback mechanism to optimize the fire source positioning accuracy.

[0088] In this implementation, S2 specifically includes:

[0089] S21, collecting visible light image data through a visible light camera to construct a visible light branch, wherein the visible light image data is used to obtain diffusion morphological characteristics of smoke, including edge characteristics, diffusion speed, and diffusion direction of smoke;

[0090] S22, collecting infrared image data through an infrared camera to construct an infrared branch, wherein the infrared image data is used to obtain temperature gradient characteristics of the smoke area, including temperature changes in the smoke diffusion area and intensity of the hot spot area;

[0091] S23, collecting gas composition data through a photoacoustic spectroscopy sensor to construct a photoacoustic branch, wherein the gas composition data is used to obtain dynamic change characteristics of gas in smoke;

[0092] S24, standardize the features extracted from each branch, generate multimodal features and store them in a shared feature pool.

[0093] In this implementation, S3 specifically includes:

[0094] S31, constructing a spatiotemporal tracking module, wherein the spatiotemporal tracking module is embedded in the shared feature pool, and the spatiotemporal tracking module includes a temporal convolutional network and a spatiotemporal attention mechanism, wherein the temporal convolutional network is used to capture the changing law of multimodal features in the temporal dimension, and the spatiotemporal attention mechanism is used to capture the distribution characteristics of multimodal features in the spatial dimension;

[0095] S32, applying a temporal convolutional network to perform temporal modeling on the multimodal features, and generating a temporal feature representation at each time step:

[0096] T(t)=σ(W t *X(t)+bt );

[0097] Among them, T(t) represents the temporal feature representation of time step t, σ represents the activation function, and W t represents the time convolution kernel weight, * represents the convolution operation, X(t) represents the multimodal feature input at time step t, and b t represents the bias vector;

[0098] S33, applying the spatiotemporal attention mechanism to perform weighted processing on the multimodal features in the spatial dimension, and generating a spatial feature representation at each spatial position:

[0099]

[0100] Among them, A(s) represents the spatial feature representation of spatial position s, Q(s) represents the query matrix of spatial position s, K(s) represents the key matrix of spatial position s, V(s) represents the value matrix of spatial position s, Softmax represents the normalization operation, and d k represents the dimension of the key vector;

[0101] S34, fuse the temporal feature representation T(t) and the spatial feature representation A(s) to generate a global feature representation with temporal and spatial correlation:

[0102]

[0103] Among them, G(t,s) represents the global feature representation with spatiotemporal correlation, α, β and γ represent weight parameters, Represents the tensor product operation, which is used to capture the interaction between temporal and spatial features.

[0104] In this implementation manner, the S4 specifically includes:

[0105] S41. Based on the global feature representation, applying a low-rank matrix decomposition algorithm to decompose the global feature representation into a low-rank matrix L and a sparse matrix S;

[0106] S42. Optimize the low-rank matrix L and the sparse matrix S by using the nuclear norm minimization method:

[0107] min L,S ∥L∥ * +λ∥S∥1;

[0108] Among them, ∥L∥ * represents the nuclear norm of the low-rank matrix L, |S||1 represents the L1 norm of the sparse matrix S, and λ represents the regularization parameter;

[0109] S43, using the alternating direction multiplier method to iterate and solve, the specific update rule is:

[0110]

[0111] Y k+1 =Y k +G(t,s)-L k+1 -S k+1 ;

[0112] Among them, L k+1 represents the low-rank matrix in the k+1th iteration, S k+1 represents the sparse matrix in the k+1th iteration, Y k +1 represents the Lagrange multiplier in the k+1th iteration, ∥·∥ F represents the Frobenius norm, S k represents the sparse matrix in the kth iteration, Y k represents the Lagrangian multiplier in the kth iteration, ρ represents the augmented Lagrangian parameter, and G(t,s) represents the global feature representation with spatiotemporal correlation;

[0113] S44. After completing the iteration, obtain the optimized low-rank matrix L * And the optimized sparse matrix S * , and remove the background noise from the global feature representation to obtain the denoised feature representation:

[0114] G′(t,s)=G(t,s)-L * ;

[0115] Among them, G'(t,s) represents the denoising feature representation;

[0116] S45. Dynamically optimize the denoising feature representation G'(t,s), and use a weighting mechanism based on the gradient boosting decision tree to adjust the weights of the multimodal features in the shared feature pool:

[0117]

[0118] in, represents the updated weight of the i-th modal feature in the shared feature pool, represents the original weight of the i-th modal feature in the shared feature pool, exp represents the exponential function, η represents the learning rate, Represents the gradient of the loss function J;

[0119] S46. Update the optimized denoising feature representation G'(t,s) back to the shared feature pool.

[0120] In this implementation manner, S5 specifically includes:

[0121] S51, constructing a lightweight self-attention mechanism module, wherein the lightweight self-attention mechanism module includes a multi-head attention unit and a feature reconstruction unit for real-time fire monitoring;

[0122] S52, inputting the extracted multimodal feature vector into the multi-head attention unit in the lightweight self-attention mechanism module, performing correlation analysis between the multimodal features, dynamically calculating the importance weight of each modal feature through the self-attention mechanism, and generating a weighted feature representation;

[0123] S53, reconstructing the weighted feature representation using a feature reconstruction unit to generate an enhanced modal feature vector, wherein the reconstruction process includes feature fusion and redundant information suppression;

[0124] S54, adopting the self-attention compensation strategy based on context dependency, by analyzing the correlation between existing modal features, inferring and compensating the feature information of the missing modality, and generating a complete multimodal feature vector:

[0125]

[0126] Among them, F complete represents the complete multimodal feature vector, ω represents the nonlinear transformation function, represents the concatenation operation, Q i represents the query matrix of the i-th modal feature, K i represents the bond matrix of the i-th modal feature, V i represents the value matrix of the i-th modal feature, n represents the total number of modes, M represents the index set of missing modes, θ j represents the compensation weight factor of the jth missing mode, C j (X j ) represents the compensatory feature generation function of the jth missing mode.

[0127] In this implementation manner, S6 specifically includes:

[0128] S61, using the complete multimodal feature vector as input to the sparse feature enhancement algorithm, and initializing the regularization parameter and the sparsity threshold;

[0129] S62. The complete multimodal feature vector is sparsely processed by a sparse feature enhancement algorithm. A sparse coding mechanism is used to represent each modal feature as a linear combination of the original features. The optimization goal is to minimize the reconstruction error while maintaining the sparsity of the modal feature representation.

[0130] S63, solving the sparse coding matrix by an iterative optimization method, and combining it with the feature dictionary to generate an enhanced sparse feature matrix;

[0131] S64, inputting the enhanced sparse feature matrix into a multi-task classification network, wherein task 1 is fire state classification, and task 2 is smoke diffusion path prediction. The multi-task classification network extracts common features through a shared layer, and processes feature representations of the two tasks in a task-specific layer, thereby forming a parallel structure of classification and prediction.

[0132] S65. Define the objective function of multi-task classification and perform joint optimization in combination with the sparse feature matrix. The objective function is expressed as the weighted sum of the two task loss functions, where the weight coefficient is dynamically adjusted according to the importance of the task.

[0133] S66. Generate fire state classification results and smoke diffusion path prediction results through a multi-task classification network.

[0134] In this implementation manner, the S7 specifically includes:

[0135] S71, collecting environmental data in real time, and calculating the deviation matrix E between the multimodal feature data of the current fire monitoring and the prediction results;

[0136] S72. Based on the deviation matrix E, the feature weights in the shared feature pool are optimized and adjusted through a dynamic feedback mechanism:

[0137]

[0138] in, represents the updated weight of the i-th modal feature in the shared feature pool, represents the original weight of the i-th modality feature in the shared feature pool, η1 represents the learning rate, represents the gradient of the deviation matrix with respect to the modal feature weight;

[0139] S73, inputting the optimized shared feature pool into the embedded spatiotemporal tracking module to update and calculate the fire source location;

[0140] S74, predicting the location of the fire source at a future time according to the optimized fire source location and combining environmental characteristics, and generating trajectory prediction results, including the future fire source location and diffusion range;

[0141] S75, evaluating the trajectory prediction results, monitoring the error change trend in real time, judging whether the prediction accuracy reaches the expected level by the change amplitude of the deviation matrix E, and further optimizing the parameters of the spatiotemporal tracking module to improve the positioning accuracy if the amplitude of the deviation matrix E exceeds the set threshold;

[0142] S76. Feedback the optimized trajectory prediction results to the spatiotemporal tracking module of the shared feature pool, thereby improving the response speed and accuracy to changes in the fire source location in the next round of monitoring, forming a dynamically optimized closed-loop feedback mechanism.

[0143] Embodiment 1:

[0144] In order to verify the feasibility of the present invention in implementation, the present invention is applied to the fire early warning system of a large industrial park. The industrial park is located in the suburbs of a city, covers an area of ​​about 50,000 square meters, has multiple high-temperature production workshops and flammable material storage warehouses, has a high fire risk, and has a complex environment, including high temperature, smoke interference, and dynamic light changes. This test aims to evaluate the actual performance of the fire smoke image early recognition method based on multimodal fusion proposed in the present invention, especially the performance under the conditions of sparse smoke signals, complex background noise, and missing data.

[0145] In this embodiment, the system deploys visible light cameras, infrared cameras and photoacoustic spectrometer sensors, which are used to collect the diffusion morphology, temperature gradient and dynamic change characteristics of gas composition of smoke respectively. All devices synchronize data through timestamps, store multimodal features in a shared feature pool, and perform dynamic modeling and feature optimization by the embedded spatiotemporal tracking module. This method combines the low-rank matrix decomposition algorithm to generate dynamic background templates, remove background noise from the global feature representation, and optimize multimodal features. In addition, a lightweight self-attention mechanism is used for multimodal feature fusion, and the early sparse smoke signal is amplified by a sparse feature enhancement algorithm, thereby completing the fire state classification and smoke diffusion path prediction.

[0146] During the experiment, a variety of fire scenarios were simulated, including normal state, early smoke diffusion, and gradual development of fire sources. Under normal conditions, the system's anti-interference ability and false alarm rate were tested; in the early smoke diffusion stage, the sparse feature enhancement algorithm's ability to recognize weak smoke signals was verified; in the fire source development stage, the accuracy of fire state classification and the reliability of diffusion path prediction were tested. The specific experimental sites covered the production workshops, warehouses, and control rooms of the industrial park. Ten monitoring points were selected, and the data collection frequency of each monitoring point was 10 frames per second, ultimately generating about 5.2TB of experimental data.

[0147] Table 1 Performance comparison of the present invention and the prior art in early fire identification

[0148] Test indicators The present invention Prior art Improvement Recognition accuracy (%) 98.5 91.2 +7.3 False alarm rate (%) 4.7 12.3 -7.6 Classification accuracy (%) 97.8 89.5 +8.3 Diffusion path prediction error (meters) 1.2 3.6 -2.4 Data compensation accuracy (%) 95.4 78.6 +16.8 Average delay time (seconds) 0.8 1.7 -0.9

[0149] According to the experimental results in the table above, the recognition accuracy of the present invention reaches 98.5%, which is 7.3 percentage points higher than the 91.2% of the prior art. This result shows that the present invention can more accurately capture the smoke characteristics of the early stage of the fire and significantly reduce the risk of missed reports. In addition, in terms of false alarm rate, the present invention reduces the false alarm rate from 12.3% of the prior art to 4.7% through dynamic background templates and feature optimization algorithms, reducing the impact of environmental background interference on fire monitoring, especially in complex dynamic scenes.

[0150] In terms of classification accuracy, the classification accuracy of the present invention is 97.8%, which is 8.3 percentage points higher than the 89.5% of the prior art. This shows that the present invention has higher reliability in distinguishing fire states (such as normal, early smoke, and fire source development), providing a more reliable basis for fire emergency decision-making. At the same time, in terms of the performance of diffusion path prediction, the average error distance of the present invention is only 1.2 meters, while the prior art is 3.6 meters, and the prediction error is significantly reduced. This improvement effectively improves the accuracy of fire source diffusion path prediction, allowing the fire monitoring system to provide a more accurate reference for the deployment of firefighting measures.

[0151] For the problem of missing data, the present invention improves the accuracy of data compensation to 95.4% through a contextual compensation strategy, which is 16.8 percentage points higher than the 78.6% of the traditional interpolation algorithm. This performance proves that the present invention can effectively compensate for feature information when processing incomplete modal data, thereby ensuring the integrity and consistency of multimodal data fusion. In addition, in terms of system response time, the average delay time from data acquisition to output of recognition results of the present invention is 0.8 seconds, which is 0.9 seconds less than the 1.7 seconds of the prior art. This significant performance improvement has brought the real-time performance of the present invention to a new level in early fire detection.

[0152] The data analysis of this embodiment fully demonstrates that the present invention effectively solves the deficiencies of traditional fire monitoring technology in sparse smoke recognition, dynamic scene adaptation, data loss compensation and real-time response through technologies such as multimodal fusion, dynamic background processing, sparse feature enhancement and dynamic feedback optimization. The present invention has significant advantages in recognition accuracy, real-time and adaptability, provides reliable technical support for early warning of fires, and has broad application prospects.

[0153] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A fire smoke image early recognition method based on multimodal fusion, characterized in that: The steps include: S1. Collect multimodal data through a visible light camera, an infrared camera and a photoacoustic spectroscopy sensor, and perform time alignment on the multimodal data based on timestamp synchronization; S2, constructing a visible light branch, an infrared branch, and a photoacoustic branch, respectively extracting smoke diffusion morphology, temperature gradient, and dynamic change characteristics of gas composition from the multimodal data, generating multimodal features, and storing them in a shared feature pool; S3, embedding a spatiotemporal tracking module in the shared feature pool to capture the changing rules of the multimodal features in the time axis and spatial distribution, and generating a global feature representation with spatiotemporal correlation characteristics; S4. Based on the global feature representation, a low-rank matrix decomposition algorithm is used to generate a dynamic background template, separate and remove background noise, and dynamically optimize the multimodal features in the shared feature pool; S5. The multimodal features are fused in the shared feature pool and the missing features are compensated through the lightweight self-attention mechanism to generate a complete multimodal feature vector. S6. Based on the complete multimodal feature vector, the early smoke signal is enhanced by a sparse feature enhancement algorithm, and a multi-task classification network is applied to perform fire state classification and smoke diffusion path prediction; S7. According to the real-time environmental data deviation, the shared feature pool and spatiotemporal tracking module are updated through a dynamic feedback mechanism to optimize the fire source positioning accuracy.

2. The method for early recognition of fire smoke images based on multimodal fusion according to claim 1 is characterized in that: The S2 specifically includes: S21, collecting visible light image data through a visible light camera to construct a visible light branch, wherein the visible light image data is used to obtain diffusion morphological characteristics of smoke, including edge characteristics, diffusion speed, and diffusion direction of smoke; S22, collecting infrared image data through an infrared camera to construct an infrared branch, wherein the infrared image data is used to obtain temperature gradient characteristics of the smoke area, including temperature changes in the smoke diffusion area and intensity of the hot spot area; S23, collecting gas composition data through a photoacoustic spectroscopy sensor to construct a photoacoustic branch, wherein the gas composition data is used to obtain dynamic change characteristics of gas in smoke; S24, standardize the features extracted from each branch, generate multimodal features and store them in a shared feature pool.

3. The method for early recognition of fire smoke images based on multimodal fusion according to claim 1 is characterized in that: The S3 specifically includes: S31, constructing a spatiotemporal tracking module, wherein the spatiotemporal tracking module is embedded in the shared feature pool, and the spatiotemporal tracking module includes a temporal convolutional network and a spatiotemporal attention mechanism, wherein the temporal convolutional network is used to capture the changing law of multimodal features in the temporal dimension, and the spatiotemporal attention mechanism is used to capture the distribution characteristics of multimodal features in the spatial dimension; S32, applying a temporal convolutional network to perform temporal modeling on the multimodal features, and generating a temporal feature representation at each time step: T(t)=σ(W t *X(t)+b t ); Among them, T(t) represents the temporal feature representation of time step t, σ represents the activation function, and W t represents the time convolution kernel weight, * represents the convolution operation, X(t) represents the multimodal feature input at time step t, and b t represents the bias vector; S33, applying the spatiotemporal attention mechanism to perform weighted processing on the multimodal features in the spatial dimension, and generating a spatial feature representation at each spatial position: Among them, A(s) represents the spatial feature representation of spatial position s, Q(s) represents the query matrix of spatial position s, K(s) represents the key matrix of spatial position s, V(s) represents the value matrix of spatial position s, Softmax represents the normalization operation, and d k represents the dimension of the key vector; S34, fuse the temporal feature representation T(t) and the spatial feature representation A(s) to generate a global feature representation with temporal and spatial correlation: Among them, G(t,s) represents the global feature representation with spatiotemporal correlation, α, β and γ represent weight parameters, Represents the tensor product operation, which is used to capture the interactive relationship between temporal and spatial features.

4. The method for early recognition of fire smoke images based on multimodal fusion according to claim 1 is characterized in that: The S4 specifically includes: S41. Based on the global feature representation, applying a low-rank matrix decomposition algorithm to decompose the global feature representation into a low-rank matrix L and a sparse matrix S; S42. Optimize the low-rank matrix L and the sparse matrix S by using the nuclear norm minimization method: min L,S ∥L∥ * +λ∥S∥1; Among them, ∥L∥ * represents the nuclear norm of the low-rank matrix L, |S||1 represents the L1 norm of the sparse matrix S, and λ represents the regularization parameter; S43, using the alternating direction multiplier method to iterate and solve, the specific update rule is: Y k+1 =Y k +G(t,s)-L k+1 -S k+1 ; Among them, L k+1 represents the low-rank matrix in the k+1th iteration, S k+1 represents the sparse matrix in the k+1th iteration, Y k+1 represents the Lagrange multiplier in the k+1th iteration, ∥·∥ F represents the Frobenius norm, S k represents the sparse matrix in the kth iteration, Y k represents the Lagrangian multiplier in the kth iteration, ρ represents the augmented Lagrangian parameter, and G(t,s) represents the global feature representation with spatiotemporal correlation; S44. After completing the iteration, obtain the optimized low-rank matrix L * And the optimized sparse matrix S * , and remove the background noise from the global feature representation to obtain the denoised feature representation: G′(t,s)=G(t,s)-L * ; Among them, G'(t,s) represents the denoising feature representation; S45. Dynamically optimize the denoising feature representation G'(t,s), and use a weighting mechanism based on the gradient boosting decision tree to adjust the weights of the multimodal features in the shared feature pool: in, represents the updated weight of the i-th modal feature in the shared feature pool, represents the original weight of the i-th modal feature in the shared feature pool, exp represents the exponential function, η represents the learning rate, Represents the gradient of the loss function J; S46. Update the optimized denoising feature representation G'(t,s) back to the shared feature pool.

5. The method for early recognition of fire smoke images based on multimodal fusion according to claim 1 is characterized in that: The S5 specifically includes: S51, constructing a lightweight self-attention mechanism module, wherein the lightweight self-attention mechanism module includes a multi-head attention unit and a feature reconstruction unit for real-time fire monitoring; S52, inputting the extracted multimodal feature vector into the multi-head attention unit in the lightweight self-attention mechanism module, performing correlation analysis between the multimodal features, dynamically calculating the importance weight of each modal feature through the self-attention mechanism, and generating a weighted feature representation; S53, reconstructing the weighted feature representation using a feature reconstruction unit to generate an enhanced modal feature vector, wherein the reconstruction process includes feature fusion and redundant information suppression; S54, adopting the self-attention compensation strategy based on context dependency, by analyzing the correlation between existing modal features, inferring and compensating the feature information of the missing modality, and generating a complete multimodal feature vector: Among them, F complete represents the complete multimodal feature vector, ω represents the nonlinear transformation function, represents the splicing operation, Q i represents the query matrix of the i-th modal feature, K i represents the bond matrix of the i-th modal feature, V i represents the value matrix of the i-th modal feature, n represents the total number of modes, M represents the index set of missing modes, θ j represents the compensation weight factor of the jth missing mode, C j (X j ) represents the compensatory feature generation function of the jth missing mode.

6. The method for early recognition of fire smoke images based on multimodal fusion according to claim 1 is characterized in that: The S6 specifically includes: S61, using the complete multimodal feature vector as input to the sparse feature enhancement algorithm, and initializing the regularization parameter and the sparsity threshold; S62. The complete multimodal feature vector is sparsely processed by a sparse feature enhancement algorithm. A sparse coding mechanism is used to represent each modal feature as a linear combination of the original features. The optimization goal is to minimize the reconstruction error while maintaining the sparsity of the modal feature representation. S63, solving the sparse coding matrix by an iterative optimization method, and combining it with the feature dictionary to generate an enhanced sparse feature matrix; S64, inputting the enhanced sparse feature matrix into a multi-task classification network, wherein task 1 is fire state classification, and task 2 is smoke diffusion path prediction. The multi-task classification network extracts common features through a shared layer, and processes feature representations of the two tasks in a task-specific layer, thereby forming a parallel structure of classification and prediction. S65. Define the objective function of multi-task classification and perform joint optimization in combination with the sparse feature matrix. The objective function is expressed as the weighted sum of the two task loss functions, where the weight coefficient is dynamically adjusted according to the importance of the task. S66. Generate fire state classification results and smoke diffusion path prediction results through a multi-task classification network.

7. The method for early recognition of fire smoke images based on multimodal fusion according to claim 1 is characterized in that: The S7 specifically includes: S71, collecting environmental data in real time, and calculating the deviation matrix E between the multimodal feature data of the current fire monitoring and the prediction results; S72. Based on the deviation matrix E, the feature weights in the shared feature pool are optimized and adjusted through a dynamic feedback mechanism: in, represents the updated weight of the i-th modal feature in the shared feature pool, represents the original weight of the i-th modality feature in the shared feature pool, η1 represents the learning rate, represents the gradient of the deviation matrix with respect to the modal feature weight; S73, inputting the optimized shared feature pool into the embedded spatiotemporal tracking module to update and calculate the fire source location; S74, predicting the location of the fire source at a future time according to the optimized fire source location and combining environmental characteristics, and generating trajectory prediction results, including the future fire source location and diffusion range; S75, evaluating the trajectory prediction results, monitoring the error change trend in real time, judging whether the prediction accuracy reaches the expected level by the change amplitude of the deviation matrix E, and further optimizing the parameters of the spatiotemporal tracking module to improve the positioning accuracy if the amplitude of the deviation matrix E exceeds the set threshold; S76. Feedback the optimized trajectory prediction results to the spatiotemporal tracking module of the shared feature pool, thereby improving the response speed and accuracy to changes in the fire source location in the next round of monitoring, forming a dynamically optimized closed-loop feedback mechanism.

Images