Fire early warning method and system for printing workshop based on multi-source data fusion

Abstract

The application provides a printing workshop fire warning method and system based on multi-source data fusion, which comprises collecting multi-modal original data such as printing workshop video, thermal infrared and gas concentration, extracting smoke texture entropy, flame flicker spectrum centroid, high-temperature area, average temperature gradient, and the smooth ratio of VOC and CO concentration normalized time derivative, and constructing a multi-modal feature set through time series resampling and dimension mapping; a variational autoencoder network containing a sharp peak thick tail mixed prior distribution is constructed, a fusion feature representation is obtained through random channel mask, cross attention fusion and decoder training; after training, the feature importance distribution is determined based on random channel mask disturbance forward propagation, and the feature selection threshold is adjusted according to the divergence change between the approximate posterior distribution before and after input channel disturbance and the prior distribution, and the optimal feature subset is selected; during the warning, the to-be-tested data features are processed by the mask based on the optimal feature subset, and the fire warning level is output by the decoder.

Description

Technical Field

[0001] This application belongs to the field of multi-source data fusion, and in particular relates to a method and system for fire early warning in printing workshops based on multi-source data fusion. Background Technology

[0002] Printing workshops typically store large quantities of flammable materials such as paper and cardboard. The production process also involves the use of inks, cleaning agents, and organic solvents containing volatile organic compounds (VOCs). When these substances accumulate in a confined space, they can easily ignite if exposed to sources of ignition such as static electricity from equipment, short circuits, or high temperatures from mechanical friction. The fire can spread rapidly due to paper stacking and air circulation. Traditional fire early warning systems rely heavily on smoke sensors, temperature sensors, or single video monitoring methods. However, printing workshop environments often contain interfering factors such as production dust, normal equipment heating, ink volatiles, and changes in lighting. Single-dimensional monitoring struggles to accurately distinguish between normal operating conditions and early fire anomalies, easily leading to false alarms or missed alarms.

[0003] With the development of multimodal sensing technology, multi-source data such as video images, thermal infrared images, and gas concentrations can jointly reflect the on-site conditions of a workshop, providing a more comprehensive information foundation for early fire warning. However, while multimodal data improves information coverage, it also brings problems such as high data dimensionality, large modal differences, and significant feature redundancy. In different stages of fire evolution, smoke texture, flame flicker, heat diffusion, and changes in volatile gases and carbon monoxide concentrations have different temporal characteristics and contribution levels. Directly inputting all features into the early warning model not only increases the computational burden but may also introduce low-contribution or interfering features, reducing the model's judgment stability. Existing methods often use fixed thresholds, manual empirical rules, or dimensionality reduction methods for feature selection, making it difficult to fully explore the dependencies between different modal features and quantify the importance and uncertainty of features in industrial scenarios. Therefore, how to select more representative and stable feature combinations for fire early warning from high-dimensional, multi-source heterogeneous data has become an urgent problem to be solved in intelligent early warning of fires in printing workshops. Summary of the Invention

[0004] To address the problems in existing technologies for fire early warning in printing workshops, such as reliance on a single monitoring dimension, difficulty in handling redundancy of multi-source heterogeneous features, and inability to quantify the importance and uncertainty of features, which leads to a high rate of false alarms and missed alarms, this invention proposes a fire early warning method and system for printing workshops based on multi-source data fusion.

[0005] Firstly, this invention proposes a fire early warning method for printing workshops based on multi-source data fusion, comprising:

[0006] The process involves acquiring multimodal raw data from the printing workshop, extracting the gray-level co-occurrence matrix texture entropy of the smoke region and the scintillation spectrum centroid of the flame region from the video image, extracting the area of ​​the high-temperature region and the average temperature gradient from the thermal infrared image, calculating the smoothing ratio of the normalized time derivative of the concentrations of volatile organic compounds and carbon monoxide, resampling the extracted features in time series, and mapping them to a unified dimension according to the modality to form a multimodal feature set. A variational autoencoder network containing a hybrid prior distribution of sharp peaks and heavy tails is constructed. During the training phase, the multimodal feature set is processed by random channel masking and then input into the encoder. The output is the feature representation of each modality. The feature representation of each modality is fused by a cross attention module to generate a fused feature vector. The fused feature vector is then input into the decoder for training. After training, a variational autoencoder network is used to perform multiple random channel mask perturbations on the forward propagation of each candidate feature. Based on the difference in the fire warning level probability vector of each candidate feature's corresponding input channel in the retained state and the zeroed state, the importance distribution of each candidate feature is determined. An adjustment factor is determined based on the divergence change of the input channel before and after perturbation. The baseline mean threshold and baseline variance threshold are adjusted using the adjustment factor. Features are selected based on the adjusted mean threshold and adjusted variance threshold to obtain the optimal feature subset. In the warning stage, the test data features are masked based on the optimal feature subset, and the fire warning level is output by the decoder.

[0007] Furthermore, this invention also relates to a fire early warning system for printing workshops based on multi-source data fusion, comprising: The mapping module is used to acquire multimodal raw data from the printing workshop, extract the gray-level co-occurrence matrix texture entropy of the smoke region and the scintillation spectrum centroid of the flame region in the video image, extract the area of ​​the high-temperature region and the average temperature gradient of the thermal infrared image, calculate the smoothing ratio of the normalized time derivative of the concentrations of volatile organic compounds and carbon monoxide, and perform temporal resampling on the extracted features, mapping them to a unified dimension according to the modality to form a multimodal feature set. The construction module is used to construct a variational autoencoder network containing a hybrid prior distribution of peaks and heavy tails. During the training phase, the multimodal feature set is processed by random channel masking and then input into the encoder to output the feature representation of each modality. The feature representation of each modality is fused by the cross attention module to generate a fused feature vector, and the fused feature vector is input into the decoder for training. The early warning module is used to perform multiple random channel mask perturbations on each candidate feature through a variational autoencoder network after training. Based on the difference in fire warning level probability vectors of each candidate feature's corresponding input channel in the retained state and the zeroed state, the importance distribution of each candidate feature is determined. An adjustment factor is determined based on the divergence change of the input channel before and after perturbation. The baseline mean threshold and baseline variance threshold are adjusted using the adjustment factor. Features are selected based on the adjusted mean threshold and adjusted variance threshold to obtain the optimal feature subset. In the early warning stage, the test data features are masked based on the optimal feature subset, and the fire warning level is output by the decoder.

[0008] This invention addresses the early warning scenario of fires in printing workshops. It extracts video smoke texture entropy, flame flicker spectrum centroid, thermal infrared high-temperature region features, and gas concentration change ratios, forming a multimodal feature set through temporal resampling and unified dimensional mapping. Further, a variational autoencoder network incorporating a hybrid prior distribution with sharp peaks and heavy tails is constructed. Combined with random channel masks and cross-attention modules, it learns the dependencies and fusion representations between different modalities. After training, the feature importance distribution is obtained through forward propagation with random channel mask perturbation. The feature selection threshold is adaptively adjusted based on the divergence change between the approximate posterior and prior distributions before and after input channel perturbation, retaining a subset of features that are more representative of the warning and satisfy stability constraints. During the warning stage, the tested data features are masked based on the optimal feature subset, and the decoder outputs the fire warning level, thereby reducing redundant feature interference and improving the accuracy and reliability of fire warnings in printing workshops. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the characteristic frequency domain distribution of the flame scintillation spectrum; Figure 2 This is a schematic diagram of the feature distribution of a thermal infrared image; Figure 3 This is a schematic diagram showing the gas characteristic sequence and ratio trend; Figure 4 This is a diagram showing the comparison of the effects of the multimodal model at different stages. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0011] This application discloses a fire early warning method for printing workshops based on multi-source data fusion, the method comprising: S1. Obtain multimodal raw data from the printing workshop, extract the gray-level co-occurrence matrix texture entropy of the smoke region and the scintillation spectrum centroid of the flame region from the video image, extract the area of ​​the high-temperature region and the average temperature gradient from the thermal infrared image, calculate the smoothing ratio of the normalized time derivative of the concentrations of volatile organic compounds and carbon monoxide, perform temporal resampling on the extracted features, and map them to a unified dimension according to the modality to form a multimodal feature set.

[0012] Connect to industrial high frame rate cameras and thermal infrared cameras to read real-time video streams and infrared image matrices. Read digital electrical signals output from volatile organic compound (VOC) and carbon monoxide (CO) sensors to complete raw data acquisition.

[0013] The Gaussian mixture background modeling algorithm is used to separate the foreground moving target in the video image. The suspected smoke region is extracted by HSV color space conversion and threshold segmentation. The gray-level co-occurrence matrix of the region in four directions is calculated. The texture entropy is calculated by multiplying the probability value by the negative of the probability logarithm.

[0014] The flame region is extracted using color thresholding and Otsu's method. The pixel brightness values ​​of the center point of the region for multiple consecutive frames are recorded to form a time series. A fast Fourier transform is performed to convert the time-domain signal into a spectral amplitude sequence. The centroid of the scintillation spectrum is obtained by calculating the sum of the products of the frequency components and the corresponding spectral amplitudes and dividing by the sum of the spectral amplitudes.

[0015] The thermal infrared image matrix is ​​read, and the temperature value matrix is ​​segmented globally using a threshold. The total number of pixels in connected components with values ​​exceeding a set threshold is counted, and the area of ​​the high-temperature region is calculated by combining this count with the actual area represented by each pixel. The Sobel operator is then used to calculate the approximate partial derivatives of the horizontal and vertical image coordinate temperature differences of this high-temperature region, and the average value is calculated to obtain the average temperature gradient.

[0016] The first discrete derivatives of the time series of volatile organic compounds (VOCs) and carbon monoxide (CO) concentrations are calculated separately. The two are mapped to the same scale by normalization of the maximum and minimum values. The normalized time derivative of VOC concentration is used as the numerator, and the sum of the normalized time derivative of CO concentration and the minimum anti-zero smoothing constant is used as the denominator. The smoothing ratio is obtained by dividing the two.

[0017] To address the issue of inconsistent sampling rates among different sensors, time-series indices are first established for video image features, thermal infrared features, and gas concentration features. A resampling function is then used to unify the feature sequences of each modality to a preset time interval. For missing time points after resampling, a cubic spline interpolation algorithm is used to fill in the corresponding feature values, ensuring that the features of each modality form a synchronized feature sequence at the same timestamp. Subsequently, a multi-layer fully connected linear mapping network is constructed to perform dimensional transformation on each modality feature. The multi-layer fully connected linear mapping network includes an input layer, at least one hidden layer, and an output layer. The number of nodes in the input layer is consistent with the original feature dimension of the corresponding modality. The hidden layer is used to perform nonlinear feature transformation, and the number of nodes in the output layer is set to a fixed dimension D, used to map different modality features to a vector space of the same length. Finally, according to a preset order of video image modality, thermal infrared modality, and gas modality, the mapped feature vectors of each modality are concatenated to form a multi-modal feature set. Among them, the video image mode includes the gray-level co-occurrence matrix texture entropy of the smoke region and the scintillation spectrum centroid of the flame region; the thermal infrared mode includes the area of ​​the high-temperature region and the average temperature gradient; and the gas mode includes the smoothed ratio of the normalized time derivative of the concentrations of volatile organic compounds and carbon monoxide.

[0018] As one possible implementation, the extraction of the gray-level co-occurrence matrix texture entropy of the smoke region and the scintillation spectral centroid of the flame region in the video image includes: The background subtraction method is used to extract the changing regions in the video image as a mask of interest; For the smoke region within the mask, the region is converted to grayscale space, a grayscale co-occurrence matrix is ​​generated by setting the spatial distance and direction, the joint probability distribution of each element in the matrix is ​​calculated, and the negative summation of the product of each probability value and its logarithm is taken as the texture entropy of the grayscale co-occurrence matrix. For the flame region within the mask, the time series of brightness values ​​of each pixel within a continuous time window is extracted. The time series is then converted to the frequency domain using a fast Fourier transform to obtain the spectral amplitude. The sum of the products of each frequency value and the corresponding spectral amplitude is divided by the sum of the spectral amplitudes to obtain the scintillation spectral centroid.

[0019] A video sequence of a printing workshop monitoring system with a frame rate preferably of 30fps is acquired. A Gaussian mixture model is used to model the background of the video sequence. The current video frame is then compared with the updated background model to obtain a foreground difference image. The foreground difference image is then thresholded and binarized to extract the foreground variation region as a mask of interest. For the smoke candidate region within the mask, the RGB color space is converted to a 256-level grayscale space. The spatial distance d is preferably set to 1 or 2 pixels. The grayscale co-occurrence matrices in the four directions of 0°, 45°, 90°, and 135° are calculated, and the mean matrix is ​​obtained.

[0020] The mean matrix is ​​normalized to obtain the joint probability distribution P(i,j) of each element. The texture entropy is then calculated by summing the negative products of each probability value and its base-2 logarithmic value, where P(i,j) is 0 and the corresponding summation term is treated as 0. For example, for an 8×8 pixel initial smoke region with a relatively uniform internal grayscale distribution, the calculated texture entropy is typically in the range of 3.0 to 4.5.

[0021] For the candidate flame region within the mask, a continuous time window N is set to 128 frames, corresponding to a time length of approximately 4.2 seconds. The grayscale values ​​of the flame pixels within the window are extracted to obtain a time series. A Fast Fourier Transform is applied to this time series to convert it to the frequency domain, obtaining the corresponding spectral amplitude A(f). The frequency resolution is calculated based on a set sampling rate of 30Hz and the window length. Each frequency value f is multiplied by its corresponding spectral amplitude A(f), and the product is accumulated. This sum is then divided by the sum of the spectral amplitudes to calculate the centroid of the scintillation spectrum. The frequency domain distribution characteristics of the flame scintillation signal are as follows: Figure 1 As shown.

[0022] If the centroid of the scintillation spectrum calculated from the spectral amplitude of a certain pixel is 8.5Hz, which is within the typical flame scintillation frequency range of 5Hz to 12Hz, it can be used as a reference for judging the actual flame scintillation characteristics.

[0023] As one possible implementation, the extraction of the high-temperature region area and average temperature gradient from the thermal infrared image includes: Set a preset temperature threshold, perform threshold segmentation on the thermal infrared image to extract high-temperature connected regions, count the total number of pixels in the connected regions, and multiply by the actual area represented by a single pixel to obtain the area of ​​the high-temperature region. Within the high-temperature connected region, the approximate partial derivatives of the image coordinate temperature difference for each pixel in the horizontal and vertical directions are calculated. The square root of the sum of the squares of the horizontal and vertical partial derivatives is taken as the image coordinate temperature gradient. The arithmetic mean of the image coordinate temperature gradients of all pixels in the high-temperature connected region is calculated to obtain the average temperature gradient.

[0024] A 640×512 resolution thermal infrared image was acquired. Due to the heat generated by the printing equipment during normal operation, a preset temperature threshold of 100℃ was set to filter out background heat source interference. This threshold was used to perform binarization segmentation of the thermal infrared image, and an 8-connected-component labeling algorithm was used to extract high-temperature connected regions exceeding this threshold. The total number of pixels M within these connected regions was counted, assuming M = 150 pixels. Combining this with the spatial resolution parameters of the thermal infrared camera at a detection distance of 10m, the actual area of ​​the instantaneous field of view corresponding to a single pixel was calculated, for example, 0.0004 square meters. Multiplying the total number of pixels M by the area of ​​a single pixel yielded a total area of ​​0.06 square meters for the current high-temperature region.

[0025] Within the extracted high-temperature connected region, the approximate partial derivatives of the image coordinate temperature difference at the corresponding pixel coordinates are calculated using the central difference method. The horizontal and vertical partial derivatives are calculated separately. The square root of the sum of the squares of the horizontal and vertical partial derivatives is taken as the image coordinate temperature gradient value at that center point. For example, if the horizontal partial derivative of a pixel is 5 degrees Celsius per pixel and the vertical partial derivative is 12 degrees Celsius per pixel, then the image coordinate temperature gradient at that point is 13 degrees Celsius per pixel.

[0026] The image coordinate temperature gradient is calculated and accumulated by traversing all pixels within the connected region. Dividing this sum by the total number of pixels (150) yields the average temperature gradient of the high-temperature region. In the initial stage of a fire, heat diffusion is relatively rapid, and the calculated average temperature gradient typically ranges from 10 to 25 degrees Celsius per pixel. This numerical characteristic can be used to quantify the heat spread trend. The thermal infrared characteristic distribution of normally operating equipment and the initial stage of a fire is shown below. Figure 2 As shown.

[0027] As one possible implementation, the calculation of the smoothed ratio of the normalized time derivatives of volatile organic compounds and carbon monoxide concentrations includes: The time derivatives of volatile organic compound (VOC) concentration and carbon monoxide (CO) concentration are obtained by dividing the difference between the VOC and CO concentration data at adjacent sampling times by the sampling time interval. The time derivatives of the volatile organic compound concentration and the carbon monoxide concentration are normalized to their maximum and minimum values ​​according to a preset time sliding window to obtain the normalized time derivatives of the volatile organic compound concentration and the carbon monoxide concentration. The ratio is obtained by dividing the normalized time derivative of the volatile organic compound concentration by the sum of the normalized time derivative of the carbon monoxide concentration and a preset minimum normal smoothing term.

[0028] Electrochemical and photoionization gas sensors deployed in the workshop are used to acquire the concentration time series of volatile organic compounds (VOCs) and carbon monoxide (CO) at fixed sampling intervals of 1 second. The time derivative of the VOC concentration is obtained by subtracting the concentration value from the previous moment from the current VOC concentration value and dividing by 1 second. For example, if the VOC concentration was 5.0 ppm one second and 5.2 ppm the current second, the time derivative is 0.2 ppm per second. The time derivative of CO is calculated similarly. A preset time sliding window, preferably 60 seconds in length, is set, and the maximum and minimum values ​​of the derivative sequence within the window are updated and recorded in real time.

[0029] The linear extremum mapping method is used to subtract the minimum value of the window from the derivative at the current time, and then divide by the difference between the maximum and minimum values ​​to complete the maximum-minimum normalization process. Assuming that the maximum value of the time derivative of volatile organic compounds is 0.5 ppm per second and the minimum value is 0.0 ppm per second within a 60-second window, and the current derivative is 0.2 ppm per second, then the calculated normalized time derivative is 0.4. Similarly, the normalized time derivative of carbon monoxide is calculated to be 0.1.

[0030] Using the normalized time derivative of volatile organic compounds (VOCs) as the numerator, a preset minimum positive constant smoothing term (range 1e-4 to 1e-2, preferably set to 0.001) is added to the normalized time derivative of carbon monoxide as the denominator. The calculated ratio, 3.96, represents the difference in gas release rates caused by smoldering paper or combustion of organic solvents in ink within the workshop. The trend of the gas characteristic ratio changes within a 60-second monitoring period is as follows: Figure 3 As shown.

[0031] S2, construct a variational autoencoder network containing a hybrid prior distribution of peaks and heavy tails. During the training phase, the multimodal feature set is processed by random channel masking and then input into the encoder. The output is the feature representation of each modality. The feature representation of each modality is fused by the cross attention module to generate a fused feature vector. The fused feature vector is then input into the decoder for training.

[0032] A variational autoencoder network is constructed, and its prior distribution is set as a peak-heavy-tailed hybrid prior distribution. The peak-heavy-tailed hybrid prior distribution is composed of a Laplace distribution with a small scale parameter as the peak part and a Gaussian distribution with a wide variance as the heavy tail part, and is weighted according to a preset or learnable hybrid weight.

[0033] During the forward propagation of training, a binary channel mask matrix is ​​generated using a Bernoulli distribution. This binary channel mask matrix is ​​then multiplied element-wise with the input multimodal feature set, randomly setting some input channels to zero, thus achieving random channel masking. The masked multimodal feature set is organized into a modal feature matrix according to the modal dimension. This modal feature matrix includes video image modal vectors, thermal infrared modal vectors, and gas modal vectors. The modal feature matrix is ​​input into a modal-encoded encoder network. This encoder network includes modal coding branches corresponding to the three modalities mentioned above. Each modal coding branch receives the corresponding modal vector and performs feature encoding through multiple fully connected layers. Each modal coding branch has two parallel fully connected output branches, which output the mean vector and log-variance vector of the modality in the latent space, respectively. Subsequently, based on the reparameterization technique, the latent space feature representations of the video image modality, thermal infrared modality, and gas modality are sampled according to the mean vector and log-variance vector corresponding to each modality.

[0034] The latent space feature representations of each modality are input into the cross-attention module, which generates query, key, and value matrices through linear transformations. The dot product of the query matrix and the transpose of the key matrix is ​​calculated, divided by a scaling factor, and then Softmax normalized to obtain the intermodal dependency weight matrix. This intermodal dependency weight matrix is ​​then multiplied by the value matrix, and the outputs of different attention heads are concatenated and fused through a feedforward neural network to generate a fused feature vector containing multimodal global context information.

[0035] The fused feature vector is input into a decoder network, which consists of a multilayer perceptron and is used to output the fire warning level prediction result and reconstruct the input multimodal feature set. During training, the mean square error between the decoder reconstruction output and the original input before masking is calculated as the reconstruction loss. The Monte Carlo sampling approximation method or numerical integration method is used to calculate the KL divergence between the approximate posterior distribution of the latent space of each modality and the peak-heavy-tailed mixed prior distribution. The sum of the KL divergences of each modality is used as the KL divergence loss. The reconstruction loss, KL divergence loss and warning level prediction loss are weighted and summed to obtain the total loss function. Finally, the AdamW optimizer is used to update the weight parameters of the variational autoencoder network.

[0036] As one possible implementation, the step of fusing the feature representations of each modality through a cross-attention module to generate a fused feature vector includes: The feature representations of each modality output by the encoder are transformed into corresponding query vectors, key vectors, and value vectors through three independent linear transformation matrices. Calculate the dot product of the query vector of a certain modality with the key vectors of all other modalities, divide it by the square root of the feature dimension, and then normalize it using the Softmax function to obtain the attention weight matrix between modalities. The attention weight matrix is ​​multiplied by the corresponding value vector, and then residually concatenated with the original feature representation. The resulting output is obtained by concatenating the vectors processed by all modalities.

[0037] Assume the multimodal features of the input encoder include three modalities: video image mode, thermal infrared mode, and gas mode. The video image mode is obtained by mapping the gray-level co-occurrence matrix texture entropy of the smoke region and the centroid of the scintillation spectrum of the flame region; the thermal infrared mode is obtained by mapping the area of ​​the high-temperature region and the average temperature gradient; and the gas mode is obtained by mapping the smoothed ratio of the normalized time derivatives of volatile organic compounds and carbon monoxide concentrations. The feature representation dimension after mapping for each mode is D=64.

[0038] Taking the video image modality as an example, its query vector is sequentially multiplied by the key vectors of all three modalities, including itself. If the dot product with the thermal infrared modality key vector is 80, it is divided by the square root of the feature dimension as a scaling factor, resulting in a scaled dot product score of 10. The scores calculated for all modalities are then input into the Softmax function for exponential normalization, outputting an attention weight matrix with a sum of 1, for example, a vector composed of 0.2, 0.5, and 0.3. This matrix represents the degree of dependence between the current video image modality and other modalities.

[0039] The weight matrix is ​​used to perform a weighted summation of the corresponding three value vectors to obtain the attention output vector. Then, a skip connection mechanism is used to add the vector element-wise to the original feature representation of the video image modality for residual connection.

[0040] After completing all modal processing, the three 1×64-dimensional enhancement vectors are concatenated along the feature dimensions to generate a 1×192-dimensional fused feature vector, which contains global contextual information from multiple sources.

[0041] As one possible implementation, the construction of the variational autoencoder network comprising a hybrid prior distribution with sharp peaks and heavy tails includes: The peak-heavy-tailed mixed prior distribution is composed of a Laplace distribution with a very small normal variance as the peak part and a Gaussian distribution with a variance greater than a preset variance threshold as the heavy-tailed part, which are weighted mixtures. The encoder of the variational autoencoder network includes multiple modality coding branches. Each modality coding branch receives the feature vector of the corresponding modality and includes multiple fully connected layers and two parallel output branches. The two parallel output branches are used to output the mean vector and log-variance vector of the modality in the Gaussian distribution of the latent space, respectively. The reparameterization technique is used to randomly sample from the Gaussian distribution of the latent space corresponding to each modality to obtain the feature representation of each modality. The decoder of the variational autoencoder network receives the fused feature vector, and on the one hand, it outputs the prediction probability vector corresponding to the fire warning level through multilayer perceptron mapping, and on the other hand, it reconstructs the input multimodal feature set to calculate the reconstruction loss.

[0042] The input to the variational autoencoder network model is a multimodal feature set consisting of video image mode, thermal infrared mode, and gas mode. The video image mode includes the texture entropy of the gray-level co-occurrence matrix of the smoke region and the centroid of the scintillation spectrum of the flame region. The thermal infrared mode includes the area of ​​the high-temperature region and the average temperature gradient. The gas mode includes the smoothed ratio of the normalized time derivatives of the concentrations of volatile organic compounds and carbon monoxide. The output of the network model is a predicted probability vector containing three fire warning levels: "normal," "alarm," and "danger," as well as a reconstructed multimodal feature matrix.

[0043] The encoder includes video image modal coding branches, thermal infrared modal coding branches, and gas modal coding branches. A cross-attention module fuses multimodal features through a linear transformation matrix and an attention mechanism. The decoder contains two parallel multilayer perceptron branches, used for prediction and reconstruction, respectively. A peak-heavy-tailed hybrid prior distribution is set to achieve sparsity penalty for redundant features and preservation of fire features. The peak portion uses a Laplace distribution with a very small scale parameter, preferably set to b=0.01, corresponding to a very small variance, to encourage the latent variables of irrelevant noise features to approach 0. The heavy-tailed portion uses a zero-mean Gaussian distribution with a variance greater than a preset variance threshold, preferably with a variance of... The value is set to 1.0 or 4.0 to provide a smooth and tolerant probabilistic modeling space for fire features with salient information. The peak and heavy tail portions are weighted and mixed using a mixing weight coefficient learned in the range of 0.1 to 0.3.

[0044] The variational autoencoder network's encoder performs encoding processing on each mode vector separately. The encoding branch for each mode consists of three fully connected layers with 256, 128, and 64 nodes respectively, using LeakyReLU as the activation function between layers. After feature extraction, it branches into two parallel fully connected linear branches, outputting the 64-dimensional latent variable mean vector μ and the 64-dimensional log-variance vector for the corresponding mode, respectively. Using auxiliary noise variables that follow a standard normal distribution. Through formula Reparameterized random sampling is performed to ensure that the network sampling process remains differentiable throughout. This yields the latent variables for video image modes, thermal infrared modes, and gas modes, which are then used as input to the cross-attention module.

[0045] After receiving the 1×192-dimensional fused feature vector, the network's decoder uses a multilayer perceptron with two hidden layers to output a prediction vector, for example, outputting prediction probabilities of 0.1, 0.8, and 0.1, corresponding to the three fire warning levels of "normal", "alarm", and "danger", respectively. At the same time, it reconstructs the multimodal feature matrix at the input through another symmetrically expanded multilayer perceptron branch. The mean square error is calculated by comparing the input and output as the reconstruction loss, and it is jointly optimized end-to-end with the classification cross-entropy loss.

[0046] S3. After training, the variational autoencoder network is used to perform multiple random channel mask perturbations on the forward propagation of each candidate feature. Based on the difference in the probability vector of the fire warning level of each candidate feature in the retained state and the zeroed state of the corresponding input channel, the importance distribution of each candidate feature is determined. The adjustment factor is determined based on the divergence change before and after the input channel perturbation. The baseline mean threshold and the baseline variance threshold are adjusted using the adjustment factor. Based on the adjusted mean threshold and the adjusted variance threshold, features are screened to obtain the optimal feature subset. In the warning stage, the features of the test data are masked based on the optimal feature subset, and the fire warning level is output by the decoder.

[0047] After the network model training converges, the weight parameters of the variational autoencoder network are locked, and each input dimension after unified dimension mapping is determined as a candidate feature channel. For any candidate feature channel, two types of input masks are constructed: the first type of input mask retains the original input value of the candidate feature channel, and the second type of input mask sets the candidate feature channel to zero; except for the candidate feature channel, the processing of other input channels is consistent in both types of input masks.

[0048] The multimodal features processed by the two types of input masks are input into the trained variational autoencoder network to obtain the fire warning level probability vector when the candidate feature channel is retained and the fire warning level probability vector when the candidate feature channel is set to zero. The difference between the two fire warning level probability vectors is calculated, and the difference is used as the single importance score of the candidate feature channel. The difference is preferably the L1 distance between the two probability vectors, that is, the sum of the absolute values ​​of the probability differences corresponding to each fire warning level. The forward propagation with random channel mask perturbation is repeated multiple times, and the empirical mean and sample variance of the importance score of each candidate feature channel are calculated to obtain the importance distribution of each candidate feature.

[0049] For each candidate feature channel, the KL divergence between the latent space approximate posterior distribution and the peak-heavy-tailed mixed prior distribution output by the encoder is calculated when the candidate feature channel is retained and when it is set to zero. Since the encoder outputs multiple modal approximate posterior distributions, the KL divergences between each modal approximate posterior distribution and the peak-heavy-tailed mixed prior distribution are summed as the overall KL divergence under that input state. The non-negative difference between the KL divergences under the two states is used as the adjustment factor corresponding to the candidate feature channel. A baseline mean threshold and a baseline variance threshold are set. The baseline mean threshold is divided by a factor plus the product of the adjustment factor and the first proportional coefficient to obtain the adjusted mean threshold. The baseline variance threshold is multiplied by a factor plus the product of the adjustment factor and the second proportional coefficient to obtain the adjusted variance threshold.

[0050] Iterate through each candidate feature channel in the multimodal feature set, determining whether the empirical mean of its importance score distribution is greater than the adjusted mean threshold, and whether the sample variance of its importance score distribution is less than the adjusted variance threshold. If both conditions are met, retain the dimension index of the candidate feature channel. Form the optimal feature subset from all retained dimension indices.

[0051] During the real-time warning phase, the same steps are followed to acquire the test data and extract multimodal features. A fixed inference period input mask matrix is ​​generated based on the dimensional index of the optimal feature subset. The input channels not selected for the optimal feature subset are set to zero using this inference period input mask matrix, while the input values ​​of the selected channels remain unchanged. Subsequently, the masked multimodal features are input into the trained variational autoencoder network, fused through a cross-attention module, and then fed into the decoder. The classification head network at the decoder outputs the probability values ​​of three fire warning levels—normal, alarm, and danger—using the Softmax function. The category with the highest probability value is determined as the current fire warning level, thus completing the real-time warning.

[0052] As one possible implementation, determining the adjustment factor based on the divergence change before and after the input channel perturbation, and adjusting the benchmark mean threshold and benchmark variance threshold using the adjustment factor, includes: For each candidate feature, calculate the KL divergence between the latent space approximate posterior distribution and the peak-heavy-tailed mixed prior distribution of the encoder output when its corresponding input channel is retained and set to zero, respectively. Determine the adjustment factor corresponding to the candidate feature based on the non-negative difference of the KL divergence in the two states. Divide the baseline mean threshold by (1 plus the product of the adjustment factor and the first proportional coefficient) to obtain the adjusted mean threshold. Multiply the baseline variance threshold by (1 plus the product of the adjustment factor and the second scaling factor) to obtain the adjusted variance threshold.

[0053] After the variational autoencoder model has completed its forward propagation, the KL divergence between the Gaussian approximate posterior distributions of each mode inferred by the encoder and the preset peak-heavy-tailed mixed prior distribution is quantified using the Monte Carlo sampling approximation method or numerical integration method. The sum of the KL divergences of each mode is then used as the overall KL divergence under the current input state. This KL divergence represents the degree of deviation of the information carried in the input features from the prior assumptions. For example, for fire features such as a sudden increase in the ratio of gas concentration derivatives or a significant increase in the area of ​​the thermal infrared high-temperature region, the posterior distribution differs from the prior, and the calculated KL divergence value is 2.5. For background white noise features, the divergence value is usually less than 0.1. The non-negative difference in KL divergence between the selected feature channel and the state of being retained and set to zero is extracted as the adjustment factor α for the corresponding feature channel, for example, α=2.5.

[0054] An initial baseline mean threshold of 0.5 and a baseline variance threshold of 0.1 are set, with the first and second proportionality coefficients preferably set to 0.4 and 0.5, respectively. During reverse adjustment, the adjusted mean threshold is calculated using the formula: baseline mean threshold / (1 + α × first proportionality coefficient). Substituting α = 2.5 and the first proportionality coefficient 0.4, the denominator is corrected to 2.0, and the adjusted mean threshold decreases to 0.25. During forward adjustment, the adjusted variance threshold is relaxed to 0.225 using the formula: baseline variance threshold × (1 + α × second proportionality coefficient). This mechanism means that when a feature contains a large amount of fire early warning information, the lower limit of the mean requirement for its selection into the optimal subset is reduced from 0.5 to 0.25, and the upper limit of the variance tolerance is increased from 0.1 to 0.225.

[0055] The importance scores of each candidate feature obtained from multiple forward propagations of perturbations are statistically analyzed, and their empirical mean and sample variance are calculated. These empirical mean and sample variance are then compared with adjusted mean thresholds and adjusted variance thresholds, respectively. If the empirical mean of the importance score of a candidate feature is greater than the adjusted mean threshold and the sample variance is less than the adjusted variance threshold, the candidate feature is retained; otherwise, it is discarded. The adjusted variance threshold provides limited tolerance for fluctuations in the importance of early-stage features in sudden fires, while still serving as an upper limit constraint on feature stability. Optionally, a maximum cutoff value can be further set to prevent high-noise features from being mistakenly selected due to excessive fluctuations.

[0056] The dataset used was real monitoring and sensor data collected from a printing workshop, containing 50,000 frames of multimodal synchronous data, divided into training, validation, and test sets in a 7:2:1 ratio. The model training hardware platform was configured with a single NVIDIA RTX 3090 computing card, with an initial learning rate of 0.001 and a batch size of 64. End-to-end optimization was performed using the AdamW optimizer. Performance evaluation metrics included early warning accuracy, false alarm rate, and total time spent on single-sample feature inference and classification.

[0057] The baseline model using only basic feature concatenation and fully connected layer classification achieved an early warning accuracy of 82.5%, a false positive rate of 11.4%, and an inference time of 42 ms. Adding a cross-attention module to the baseline model improved the early warning accuracy to 88.7%, reduced the false positive rate to 7.2%, and increased the inference time to 48 ms. Continuing to use a conventional variational autoencoder network instead of a regular classifier, the early warning accuracy reached 92.1%, the false positive rate was 4.5%, and the inference time was 55 ms. Using the complete scheme of this invention, which includes a peaked-fat-tailed mixed prior distribution and a KL divergence threshold adjustment mechanism, the early warning accuracy was approximately 96.8%, the false positive rate was approximately 1.2%, and the inference time was approximately 52 ms. A comparison of the model's early warning performance and inference efficiency at each optimization stage is provided below. Figure 4As shown.

[0058] The cross-attention module achieves the fusion of video images, infrared thermal images, and gas temporal modes by calculating the dependency weights of query key values, thus overcoming the discrimination limitations of single-source features. In comparison, the peak-heavy-tailed hybrid prior distribution in this application compresses the latent space dimension of redundant background noise. Combined with an adjustment factor determined based on the divergence change before and after input channel perturbation, it inversely adjusts the mean threshold and forward adjusts the variance threshold, enabling more lenient inclusion conditions for sudden, high-information-content fire features. While maintaining acceptable inference time, it reduces false alarms caused by normal heating of workshop equipment, thereby improving the accuracy of early fire warnings.

[0059] This application discloses a fire early warning system for a printing workshop based on multi-source data fusion, the system comprising: The mapping module is used to acquire multimodal raw data from the printing workshop, extract the gray-level co-occurrence matrix texture entropy of the smoke region and the scintillation spectrum centroid of the flame region in the video image, extract the area of ​​the high-temperature region and the average temperature gradient of the thermal infrared image, calculate the smoothing ratio of the normalized time derivative of the concentrations of volatile organic compounds and carbon monoxide, and perform temporal resampling on the extracted features, mapping them to a unified dimension according to the modality to form a multimodal feature set. The construction module is used to construct a variational autoencoder network containing a hybrid prior distribution of peaks and heavy tails. During the training phase, the multimodal feature set is processed by random channel masking and then input into the encoder to output the feature representation of each modality. The feature representation of each modality is fused by the cross attention module to generate a fused feature vector, and the fused feature vector is input into the decoder for training. The early warning module is used to perform multiple random channel mask perturbations on each candidate feature through a variational autoencoder network after training. Based on the difference in fire warning level probability vectors of each candidate feature's corresponding input channel in the retained state and the zeroed state, the importance distribution of each candidate feature is determined. An adjustment factor is determined based on the divergence change of the input channel before and after perturbation. The baseline mean threshold and baseline variance threshold are adjusted using the adjustment factor. Features are selected based on the adjusted mean threshold and adjusted variance threshold to obtain the optimal feature subset. In the early warning stage, the test data features are masked based on the optimal feature subset, and the fire warning level is output by the decoder.

[0060] As one possible implementation, the extraction of the gray-level co-occurrence matrix texture entropy of the smoke region and the scintillation spectral centroid of the flame region in the video image includes: The background subtraction method is used to extract the changing regions in the video image as a mask of interest; For the smoke region within the mask, the region is converted to grayscale space, a grayscale co-occurrence matrix is ​​generated by setting the spatial distance and direction, the joint probability distribution of each element in the matrix is ​​calculated, and the negative summation of the product of each probability value and its logarithm is taken as the texture entropy of the grayscale co-occurrence matrix. For the flame region within the mask, the time series of brightness values ​​of each pixel within a continuous time window is extracted. The time series is then converted to the frequency domain using a fast Fourier transform to obtain the spectral amplitude. The sum of the products of each frequency value and the corresponding spectral amplitude is divided by the sum of the spectral amplitudes to obtain the scintillation spectral centroid.

[0061] As one possible implementation, the extraction of the high-temperature region area and average temperature gradient from the thermal infrared image includes: Set a preset temperature threshold, perform threshold segmentation on the thermal infrared image to extract high-temperature connected regions, count the total number of pixels in the connected regions, and multiply by the actual area represented by a single pixel to obtain the area of ​​the high-temperature region. Within the high-temperature connected region, the approximate partial derivatives of the image coordinate temperature difference for each pixel in the horizontal and vertical directions are calculated. The square root of the sum of the squares of the horizontal and vertical partial derivatives is taken as the image coordinate temperature gradient. The arithmetic mean of the image coordinate temperature gradients of all pixels in the high-temperature connected region is calculated to obtain the average temperature gradient.

[0062] As one possible implementation, the calculation of the smoothed ratio of the normalized time derivatives of volatile organic compounds and carbon monoxide concentrations includes: The time derivatives of volatile organic compound (VOC) concentration and carbon monoxide (CO) concentration are obtained by dividing the difference between the VOC and CO concentration data at adjacent sampling times by the sampling time interval. The time derivatives of the volatile organic compound concentration and the carbon monoxide concentration are normalized to their maximum and minimum values ​​according to a preset time sliding window to obtain the normalized time derivatives of the volatile organic compound concentration and the carbon monoxide concentration. The ratio is obtained by dividing the normalized time derivative of the volatile organic compound concentration by the sum of the normalized time derivative of the carbon monoxide concentration and a preset minimum normal smoothing term.

[0063] As one possible implementation, the step of fusing the feature representations of each modality through a cross-attention module to generate a fused feature vector includes: The feature representations of each modality output by the encoder are transformed into corresponding query vectors, key vectors, and value vectors through three independent linear transformation matrices. Calculate the dot product of the query vector of a certain modality with the key vectors of all other modalities, divide it by the square root of the feature dimension, and then normalize it using the Softmax function to obtain the attention weight matrix between modalities. The attention weight matrix is ​​multiplied by the corresponding value vector, and then residually concatenated with the original feature representation. The resulting output is obtained by concatenating the vectors processed by all modalities.

[0064] As one possible implementation, the construction of the variational autoencoder network comprising a hybrid prior distribution with sharp peaks and heavy tails includes: The peak-heavy-tailed mixed prior distribution is composed of a Laplace distribution with a very small normal variance as the peak part and a Gaussian distribution with a variance greater than a preset variance threshold as the heavy-tailed part, which are weighted mixtures. The encoder of the variational autoencoder network includes multiple modality coding branches. Each modality coding branch receives the feature vector of the corresponding modality and includes multiple fully connected layers and two parallel output branches. The two parallel output branches are used to output the mean vector and log-variance vector of the modality in the Gaussian distribution of the latent space, respectively. The reparameterization technique is used to randomly sample from the Gaussian distribution of the latent space corresponding to each modality to obtain the feature representation of each modality. The decoder of the variational autoencoder network receives the fused feature vector, and on the one hand, it outputs the prediction probability vector corresponding to the fire warning level through multilayer perceptron mapping, and on the other hand, it reconstructs the input multimodal feature set to calculate the reconstruction loss.

[0065] As one possible implementation, determining the adjustment factor based on the divergence change before and after the input channel perturbation, and adjusting the benchmark mean threshold and benchmark variance threshold using the adjustment factor, includes: For each candidate feature, calculate the KL divergence between the latent space approximate posterior distribution and the peak-heavy-tailed mixed prior distribution of the encoder output when its corresponding input channel is retained and set to zero, respectively. Determine the adjustment factor corresponding to the candidate feature based on the non-negative difference of the KL divergence in the two states. Divide the baseline mean threshold by (1 plus the product of the adjustment factor and the first proportional coefficient) to obtain the adjusted mean threshold. Multiply the baseline variance threshold by (1 plus the product of the adjustment factor and the second scaling factor) to obtain the adjusted variance threshold.

[0066] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0067] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A fire early warning method for printing workshops based on multi-source data fusion, characterized in that, include: The process involves acquiring multimodal raw data from the printing workshop, extracting the gray-level co-occurrence matrix texture entropy of the smoke region and the scintillation spectrum centroid of the flame region from the video image, extracting the area of ​​the high-temperature region and the average temperature gradient from the thermal infrared image, calculating the smoothing ratio of the normalized time derivative of the concentrations of volatile organic compounds and carbon monoxide, resampling the extracted features in time series, and mapping them to a unified dimension according to the modality to form a multimodal feature set. A variational autoencoder network containing a hybrid prior distribution of sharp peaks and heavy tails is constructed. During the training phase, the multimodal feature set is processed by random channel masking and then input into the encoder. The output is the feature representation of each modality. The feature representation of each modality is fused by a cross attention module to generate a fused feature vector. The fused feature vector is then input into the decoder for training. After training, a variational autoencoder network is used to perform multiple random channel mask perturbations on the forward propagation of each candidate feature. Based on the difference in the fire warning level probability vector of each candidate feature's corresponding input channel in the retained state and the zeroed state, the importance distribution of each candidate feature is determined. An adjustment factor is determined based on the divergence change of the input channel before and after perturbation. The baseline mean threshold and baseline variance threshold are adjusted using the adjustment factor. Features are selected based on the adjusted mean threshold and adjusted variance threshold to obtain the optimal feature subset. In the warning stage, the test data features are masked based on the optimal feature subset, and the fire warning level is output by the decoder.

2. The method according to claim 1, characterized in that, The extraction of the gray-level co-occurrence matrix texture entropy of the smoke region and the scintillation spectral centroid of the flame region in the video image includes: The background subtraction method is used to extract the changing regions in the video image as a mask of interest; For the smoke region within the mask, the region is converted to grayscale space, a grayscale co-occurrence matrix is ​​generated by setting the spatial distance and direction, the joint probability distribution of each element in the matrix is ​​calculated, and the negative summation of the product of each probability value and its logarithm is taken as the texture entropy of the grayscale co-occurrence matrix. For the flame region within the mask, the time series of brightness values ​​of each pixel within a continuous time window is extracted. The time series is then converted to the frequency domain using a fast Fourier transform to obtain the spectral amplitude. The sum of the products of each frequency value and the corresponding spectral amplitude is divided by the sum of the spectral amplitudes to obtain the scintillation spectral centroid.

3. The method according to claim 2, characterized in that, The extraction of the high-temperature region area and average temperature gradient from the thermal infrared image includes: Set a preset temperature threshold, perform threshold segmentation on the thermal infrared image to extract high-temperature connected regions, count the total number of pixels in the connected regions, and multiply by the actual area represented by a single pixel to obtain the area of ​​the high-temperature region. Within the high-temperature connected region, the approximate partial derivatives of the image coordinate temperature difference for each pixel in the horizontal and vertical directions are calculated. The square root of the sum of the squares of the horizontal and vertical partial derivatives is taken as the image coordinate temperature gradient. The arithmetic mean of the image coordinate temperature gradients of all pixels in the high-temperature connected region is calculated to obtain the average temperature gradient.

4. The method according to claim 1, characterized in that, The calculation of the smoothed ratio of the normalized time derivatives of volatile organic compounds and carbon monoxide concentrations includes: The time derivatives of volatile organic compound (VOC) concentration and carbon monoxide (CO) concentration are obtained by dividing the difference between the VOC and CO concentration data at adjacent sampling times by the sampling time interval. The time derivatives of the volatile organic compound concentration and the carbon monoxide concentration are normalized to their maximum and minimum values ​​according to a preset time sliding window to obtain the normalized time derivatives of the volatile organic compound concentration and the carbon monoxide concentration. The ratio is obtained by dividing the normalized time derivative of the volatile organic compound concentration by the sum of the normalized time derivative of the carbon monoxide concentration and a preset minimum normal smoothing term.

5. The method according to claim 1, characterized in that, The step of fusing the feature representations of each modality through a cross-attention module to generate a fused feature vector includes: The feature representations of each modality output by the encoder are transformed into corresponding query vectors, key vectors, and value vectors through three independent linear transformation matrices. Calculate the dot product of the query vector of a certain modality with the key vectors of all other modalities, divide it by the square root of the feature dimension, and then normalize it using the Softmax function to obtain the attention weight matrix between modalities. The attention weight matrix is ​​multiplied by the corresponding value vector, and then residually concatenated with the original feature representation. The resulting output is obtained by concatenating the vectors processed by all modalities.

6. The method according to claim 4, characterized in that, The construction of the variational autoencoder network, which includes a hybrid prior distribution with sharp peaks and heavy tails, includes: The peak-heavy-tailed mixed prior distribution is composed of a Laplace distribution with a very small normal variance as the peak part and a Gaussian distribution with a variance greater than a preset variance threshold as the heavy-tailed part, which are weighted mixtures. The encoder of the variational autoencoder network includes multiple modality coding branches. Each modality coding branch receives the feature vector of the corresponding modality and includes multiple fully connected layers and two parallel output branches. The two parallel output branches are used to output the mean vector and log-variance vector of the modality in the Gaussian distribution of the latent space, respectively. The reparameterization technique is used to randomly sample from the Gaussian distribution of the latent space corresponding to each modality to obtain the feature representation of each modality. The decoder of the variational autoencoder network receives the fused feature vector, and on the one hand, it outputs the prediction probability vector corresponding to the fire warning level through multilayer perceptron mapping, and on the other hand, it reconstructs the input multimodal feature set to calculate the reconstruction loss.

7. The method according to claim 1, characterized in that, The step of determining the adjustment factor based on the divergence change before and after the input channel perturbation, and adjusting the benchmark mean threshold and benchmark variance threshold using the adjustment factor, includes: For each candidate feature, calculate the KL divergence between the latent space approximate posterior distribution and the peak-heavy-tailed mixed prior distribution of the encoder output when its corresponding input channel is retained and set to zero, respectively. Determine the adjustment factor corresponding to the candidate feature based on the non-negative difference of the KL divergence in the two states. Divide the baseline mean threshold by (1 plus the product of the adjustment factor and the first proportional coefficient) to obtain the adjusted mean threshold. Multiply the baseline variance threshold by (1 plus the product of the adjustment factor and the second scaling factor) to obtain the adjusted variance threshold.

8. A fire early warning system for a printing workshop based on multi-source data fusion, characterized in that, include: The mapping module is used to acquire multimodal raw data from the printing workshop, extract the gray-level co-occurrence matrix texture entropy of the smoke region and the scintillation spectrum centroid of the flame region in the video image, extract the area of ​​the high-temperature region and the average temperature gradient of the thermal infrared image, calculate the smoothing ratio of the normalized time derivative of the concentrations of volatile organic compounds and carbon monoxide, and perform temporal resampling on the extracted features, mapping them to a unified dimension according to the modality to form a multimodal feature set. The construction module is used to construct a variational autoencoder network containing a hybrid prior distribution of peaks and heavy tails. During the training phase, the multimodal feature set is processed by random channel masking and then input into the encoder to output the feature representation of each modality. The feature representation of each modality is fused by the cross attention module to generate a fused feature vector, and the fused feature vector is input into the decoder for training. The early warning module is used to perform multiple random channel mask perturbations on each candidate feature through a variational autoencoder network after training. Based on the difference in fire warning level probability vectors of each candidate feature's corresponding input channel in the retained state and the zeroed state, the importance distribution of each candidate feature is determined. An adjustment factor is determined based on the divergence change of the input channel before and after perturbation. The baseline mean threshold and baseline variance threshold are adjusted using the adjustment factor. Features are selected based on the adjusted mean threshold and adjusted variance threshold to obtain the optimal feature subset. In the early warning stage, the test data features are masked based on the optimal feature subset, and the fire warning level is output by the decoder.

9. The system according to claim 8, characterized in that, The extraction of the gray-level co-occurrence matrix texture entropy of the smoke region and the scintillation spectral centroid of the flame region in the video image includes: The background subtraction method is used to extract the changing regions in the video image as a mask of interest; For the smoke region within the mask, the region is converted to grayscale space, a grayscale co-occurrence matrix is ​​generated by setting the spatial distance and direction, the joint probability distribution of each element in the matrix is ​​calculated, and the negative summation of the product of each probability value and its logarithm is taken as the texture entropy of the grayscale co-occurrence matrix. For the flame region within the mask, the time series of brightness values ​​of each pixel within a continuous time window is extracted. The time series is then converted to the frequency domain using a fast Fourier transform to obtain the spectral amplitude. The sum of the products of each frequency value and the corresponding spectral amplitude is divided by the sum of the spectral amplitudes to obtain the scintillation spectral centroid.

10. The system according to claim 8, characterized in that, The extraction of the high-temperature region area and average temperature gradient from the thermal infrared image includes: Set a preset temperature threshold, perform threshold segmentation on the thermal infrared image to extract high-temperature connected regions, count the total number of pixels in the connected regions, and multiply by the actual area represented by a single pixel to obtain the area of ​​the high-temperature region. Within the high-temperature connected region, the approximate partial derivatives of the image coordinate temperature difference for each pixel in the horizontal and vertical directions are calculated. The square root of the sum of the squares of the horizontal and vertical partial derivatives is taken as the image coordinate temperature gradient. The arithmetic mean of the image coordinate temperature gradients of all pixels in the high-temperature connected region is calculated to obtain the average temperature gradient.

Images