A method for predicting the fire situation in the early stage of a building fire

Abstract

The present invention discloses a method for predicting the fire situation in the early stage of a fire in a building, including obtaining a fire detection model by training a fire detection data set, extracting fire detection parameters and combustible categories after detecting the fire, and then fusing the features. Two prediction approaches are provided to adapt to different application scenarios. One is to adapt to the use of embedded devices and improve the system response speed, and use the fused flame parameters as input to train the flame prediction network to predict the flame height, flame projection area, and flame spread position. The other is to deploy the algorithm in the server to run, use the fire detection parameters and combustible distribution characteristics as input to train the ConvLSTM fire prediction network, and output the flame position and size at the next moment. The present invention can not only predict the state of the flame at the next moment, but also help the fire department and other relevant departments to analyze the cause of the fire and formulate a fire prevention plan after the fire occurs.

Description

Technical Field

[0001] The invention belongs to the field of fire prediction, and in particular relates to a method for predicting the fire situation at the initial stage of a fire in a building. Background Art

[0002] In recent years, fires in urban buildings have occurred frequently, causing heavy casualties and property losses. Indoor fires are generally characterized by dense crowds that are difficult to evacuate, concentrated combustibles, and complex building designs that make escape routes difficult. These characteristics lead to rapid fire spread before firefighters arrive at the fire scene, causing inestimable losses. The development stages of indoor fires generally include four stages: initial fire growth, fire explosion, full fire development, and fire decay. When an indoor fire reaches the conditions of fire explosion, all surfaces of indoor combustibles enter a state of full combustion. At this time, the temperature rises sharply, and the fire development enters an uncontrollable state. Therefore, before the fire explosion occurs, the situation of the indoor fire flame is predicted in the initial growth stage of the fire, which is of great significance for personnel escape and fire fighting and rescue by fire departments. The initial stage of a fire is the golden moment for personnel escape and fire fighting and rescue, so it is necessary to predict the trend of fire spread at this stage.

[0003] At present, most of the research in the field of fire vision focuses on fire detection, and there is little research on fire situation prediction. In addition, the development of indoor fire follows the theory of fire dynamics, and the laws of fire spread are different at different stages of fire. There is little research in this area. At present, there are mainly three problems in fire situation prediction:

[0004] (1) Currently, there are relatively few studies combining detection and prediction in the field of fire vision. Most of them focus on the early detection of fires, while ignoring the extraction of fire parameters that reflect the development of fire from the fire image when the fire is detected, and using these parameters to predict the development of the fire, or predicting the location of the flame spread at the next moment based on the fire detection image.

[0005] (2) In the early stages of an indoor fire, the distribution, quantity, and properties of combustibles are the main factors affecting the spread of the fire. Existing related studies only predict the fire situation based on the dynamic or static characteristics of the flame itself, resulting in poor generalization of the prediction model for different data, making it difficult to achieve accurate predictions.

[0006] (3) Currently, data from fire scenes is generally obtained through surveillance cameras with fixed viewing angles. The surveillance camera images contain a large number of redundant frames. Sending real-time images to the model for detection and alarm frame by frame will cause a computational burden on the model, resulting in the real-time performance of fire detection and prediction failing to meet the expected goals. Summary of the invention

[0007] The early stage of an indoor fire is an important moment for firefighting, rescue and fire analysis. This stage is also the stage when the fire is predictable. Therefore, solutions are proposed mainly to the above-mentioned problems that there is currently little integration of indoor fire detection and prediction, the fire situation is predicted only by the characteristics of the flame itself, and the real-time performance of fire detection and prediction cannot meet the actual engineering requirements. In response to the problems existing in the prior art, the present invention provides a method for predicting the fire situation in the early stage of a fire in a building, which can better solve the above-mentioned problems and accurately predict the fire. The present invention provides the following technical solutions: A method for predicting the fire situation in the early stage of a fire in a building, comprising the following steps:

[0008] S1. Based on the fire scene video, extract the key frames of the video and construct a fire detection dataset, including the key frames of the fire video and the fire detection parameter labels and combustible category labels corresponding to the key frames;

[0009] S2. Build an improved dual-stream fusion fire detection network, and use the fire detection data set to train the improved dual-stream fusion fire detection network to obtain a fire detection model;

[0010] S3, input the fire image into the fire detection model to obtain fire detection parameters and combustible material categories, and then perform data preprocessing, fuse the fire detection parameters and combustible material distribution characteristics obtained after preprocessing, and obtain fused flame parameters;

[0011] S4. Build an LSTM flame prediction network, use the fused flame parameters as input, and the corresponding flame height, flame projection area, and flame spread position as output to train the LSTM flame prediction network to obtain a flame prediction model.

[0012] S5. Construct a ConvLSTM fire prediction network, take the fire detection parameters and the distribution characteristics of combustibles as input, and the corresponding flame position and size at the next moment as output, train the ConvLSTM fire prediction network, and obtain a fire prediction model;

[0013] S6. Based on the fire detection model, the flame prediction model and the fire prediction model, a fire situation prediction model is constructed, and the collected fire video is input into the fire situation prediction model to obtain the flame height, flame projection area, flame spread position, and the position and size of the flame at the next moment.

[0014] Furthermore, in the aforementioned step S1, extracting the key frames of the video includes the following sub-steps:

[0015] S1.1. Read the video frame and convert it into a grayscale image. Calculate the difference between the current frame and the two previous and next frames using the three-frame difference method. The calculation formula of the three-frame difference method is as follows:

[0016] ,

[0017] in, Represents the image grayscale values ​​of the three consecutive frames before and after, and the three frames are subtracted from each other to obtain the difference image, and the difference image is obtained by performing the AND operation ;

[0018] S1.2. Use the entropy extreme value of the differential image to determine whether the current frame is a key frame. Count the number of gray levels of pixels in the differential image. Use the gray histogram to calculate the probability of each gray level. Divide the number of pixels at each gray level by the total number of pixels in the image frame to calculate the probability distribution of each gray level, which is the probability density function. ;

[0019] S1.3, performing median filtering and smoothing processing on the obtained probability density function;

[0020] S1.4, normalize the probability density function so that the sum of the probability density is equal to 1, which is used to represent the relative weight of each gray level;

[0021] S1.5. Probability density function of each frame of the fire video obtained in step S1.2 , the entropy value of each frame is obtained by the entropy value calculation formula, the calculation formula is as follows: ;

[0022] S1.6. The calculated entropy value Mapped to the two-dimensional coordinate axis x with the image frame number as the axis, and finally taking the image number with the extreme value of the differential image entropy in the segment as the key frame in the segment n ;

[0023] S1.7. Return to step S1.1 until all video frames are traversed and all key frames are obtained.

[0024] Furthermore, in the aforementioned step S2, the improved dual-stream fusion fire detection network is constructed based on the SlowFast network, including a fast channel and a slow channel; the fast channel is used to extract the time features of the time series data; and the slow channel is used to extract the spatial features of the time series data.

[0025] Furthermore, the aforementioned construction of an improved dual-stream fusion fire detection network includes the following steps:

[0026] S2.1, set the frame extraction ratio of the fast channel and the slow channel, that is, the number of frames extracted per second by the fast channel is a preset multiple of the slow channel, and input the extracted key frame T into the fast channel, and the image size of the frame is 224×224×T;

[0027] S2.2, the slow channel uses the ResNet-18 network as the backbone network to extract deep features. The slow channel Resnet-18 network includes input layer, convolution layer, BN layer, activation function RELU, residual block, pooling layer, and fully connected layer; the first layer is a 7×7 convolution layer. The network contains 4 residual blocks, each residual block has two residual units, and each residual unit contains two 3×3 convolution layers, followed by a global average pooling layer and a fully connected layer;

[0028] S2.3, when the slow channel performs deep feature extraction, the convolution in the time dimension is added to the deep layer of the network to fuse with the fast channel data stream. In the ResNet-18 network, the traditional two-dimensional convolution is replaced by three-dimensional convolution in the Res3 and Res4 layers;

[0029] S2.4, the fast channel uses a 3D ResNet-18 network. In the first convolution layer of each residual unit of Res2, Res3, Res4, and Res5, the two-dimensional convolution is replaced by a three-dimensional convolution to extract the temporal features between sequence data;

[0030] S2.5. Input the features of the fast channel and the slow channel output into the CMDA module. The CMDA module includes: a channel attention mechanism module and a spatiotemporal attention mechanism module. The CMDA module concatenates the feature vectors of the fast channel and the slow channel to detect fire and combustibles.

[0031] Furthermore, in the aforementioned step S3, data preprocessing includes the following sub-steps:

[0032] S3.1. Extract and store the fire detection parameters: flame height, flame projection area, and flame spread position; the flame spread position is the flame centroid position;

[0033] S3.2. Use a filtering method based on category information to filter and obtain the distribution characteristics of combustibles;

[0034] S3.3. Smoothing, denoising and regularization of fire detection parameters.

[0035] Furthermore, in the aforementioned step S3, the fire detection parameters and the combustible distribution features are fused using a cross attention mechanism to obtain fused flame parameters, which specifically includes the following sub-steps:

[0036] Step S301: Calculate the Q of the combustible distribution characteristics i , K, V values, as follows:

[0037] ,in ， is the linear transformation matrix, yi Represents fire detection parameters, i=1, 2, 3, x Indicates the distribution characteristics of combustibles;

[0038] S302, respectively calculate the similarity between the fire detection parameter Query value and the combustible distribution feature Key to obtain the attention score, and then normalize the attention score through the function; multiply the value corresponding to the Key by the normalized attention score to obtain the attention output of 1 parameter, as shown in the following formula: Where, i=1,2,3;

[0039] ; Used to scale the dot product result. d yes k The dimension of T Represents the transpose of a matrix.

[0040] Furthermore, the aforementioned step S4 is specifically as follows: the LSTM network adds a unit state C throughout the network, and adds three control gates on the basis of the RNN: a forget gate, an input gate, and an output gate;

[0041] The forget gate is expressed as follows: , the cell state is initially set to the 0 vector; is the sigmoid function, the output of this function is [0,1], which represents a probability; the past unit state To update; W f represents the weight matrix of the forget gate, x t represents the input of the current time step, b f Represents the bias vector of the forget gate; the expression of the input gate is expressed as:

[0042] ;

[0043] ;

[0044] The new unit state C t for: ;

[0045] The output gate is represented as: ;

[0046] in, Indicates the flame height at the last moment, b i represents the bias vector of the input gate, x t represents the input of the current time step,b c The bias vector representing the candidate cell state, Indicates the candidate unit status, b o represents the bias vector of the output gate, h t represents the hidden state of the current time step, o t represents the output of the output gate, W i represents the weight matrix of the input gate, W c The weight matrix representing the candidate unit state, i t represents the output of the input gate, W o represents the weight matrix of the output gate, f t represents the output of the forget gate, Indicates the state of the unit after activation.

[0047] Furthermore, the aforementioned step S5 includes the following sub-steps:

[0048] S5.1. Set the network parameters of the ConvLSTM encoding layer, including the number of ConvLSTM layers, input parameters, and output parameters; ConvLSTM performs convolution calculations, inputs and updates the unit state, forgets, and outputs according to the gate structure, as follows:

[0049] ;

[0050] in represents convolution, Represents the inner product of corresponding elements in the same position;

[0051] S5.2, the spatiotemporal features extracted by deep ConvLSTM are used as the input of the spatiotemporal attention mechanism to focus on important feature information;

[0052] S5.3, the features processed by the attention mechanism are sent to the prediction network. The prediction network is composed of ConvLSTM cells with a depth of 3 layers. The output is the prediction of the next 5 frames of flame images for every T frames of flame images.

[0053] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:

[0054] 1. The present invention starts with the characteristics of the indoor fire development stage, combines fire dynamics theory, combines fire detection with fire prediction, and provides two prediction schemes. The first scheme predicts the state of the flame at the next moment by extracting parameters that can characterize the state of fire spread. The second scheme directly predicts the position and size of the flame spread at the next moment.

[0055] 2. The fire prediction of the present invention takes into account the distribution and quantity of combustibles, which are important factors affecting the early development of indoor fires, and integrates them with fire detection parameters through a cross-attention mechanism module to capture the connection between them, or uses a spatiotemporal attention mechanism module to directly capture the relationship between combustibles and flame spread, making flame prediction more accurate.

[0056] 3. In order to solve the above-mentioned problem that the current fire prediction cannot achieve real-time performance, the present invention adds an adaptive key frame extraction module in data preprocessing, and uses a lightweight network ResNet-18 to replace the original feature extraction network of SlowFast in the improved dual-stream fusion fire detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 1 is an overall flow chart of the present invention, wherein (a) is a flow chart of flame prediction, and (b) is a flow chart of fire prediction.

[0058] Figure 2 It is a structural diagram of the flame prediction model of the present invention.

[0059] Figure 3 It is a structural diagram of the fire prediction model of the present invention. DETAILED DESCRIPTION

[0060] In order to better understand the technical content of the present invention, specific embodiments are given and described as follows in conjunction with the accompanying drawings.

[0061] Various aspects of the invention are described herein with reference to the accompanying drawings, in which many illustrative embodiments are shown. The embodiments of the invention are not limited to those described in the accompanying drawings. It should be understood that the invention is implemented by any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed in the invention are not limited to any implementation. In addition, some aspects disclosed in the invention may be used alone or in any appropriate combination with other aspects disclosed in the invention.

[0062] refer to Figure 1, (a) is a flow chart of flame prediction, and (b) is a flow chart of fire prediction. A method for predicting the fire situation in the early stage of a fire in a building of the present invention comprises adaptive key frame extraction, improved dual-stream fusion fire detection, fire detection parameter processing, fusion of combustible distribution characteristics and fire detection parameters, LSTMLSTM flame prediction network, and improved ConvLSTM fire prediction network. The present invention comprises the following steps:

[0063] S1. Based on the fire scene video, extract the key frames of the video and construct a fire detection dataset, including the key frames of the fire video and the fire detection parameter labels and combustible category labels corresponding to the key frames. The datasets used in the present invention are the public datasets VisiFire dataset and FLAME dataset. Some data are collected through the network. The equipment used to collect data at the fire scene is a surveillance camera with a fixed viewing angle. Therefore, when the camera continuously monitors the controlled area, the background in the indoor environment remains basically unchanged. If the video of the fire scene is fed into the fire detection model frame by frame, the calculation time of the model will be increased, which is not conducive to the on-site fire alarm and fire prediction. The key frames in the on-site fire video need to be extracted to improve the fire detection speed. Extracting the key frames of the video includes the following sub-steps:

[0064] S1.1. Read the video frame and convert it into a grayscale image. Calculate the difference between the current frame and the two previous and next frames using the three-frame difference method. The calculation formula of the three-frame difference method is as follows:

[0065] ,

[0066] in, Represents the image grayscale values ​​of the three consecutive frames before and after, and the three frames are subtracted from each other to obtain the difference image, and the difference image is obtained by performing the AND operation ;

[0067] S1.2. Use the entropy extreme value of the differential image to determine whether the current frame is a key frame. Count the number of gray levels of pixels in the differential image. Use the gray histogram to calculate the probability of each gray level. Divide the number of pixels at each gray level by the total number of pixels in the image frame to calculate the probability distribution of each gray level, which is the probability density function. ;

[0068] S1.3, performing median filtering and smoothing processing on the obtained probability density function;

[0069] S1.4, normalize the probability density function so that the sum of the probability density is equal to 1, which is used to represent the relative weight of each gray level;

[0070] S1.5. Probability density function of each frame of the fire video obtained in step S1.2 , the entropy value of each frame is obtained by the entropy value calculation formula, the calculation formula is as follows: ;

[0071] S1.6. The calculated entropy value Mapped to the two-dimensional coordinate axis x with the image frame number as the axis, and finally taking the image number with the extreme value of the differential image entropy in the segment as the key frame in the segment n ;

[0072] S1.7. Return to step S1.1 until all video frames are traversed and all key frames are obtained.

[0073] S2. Build an improved dual-stream fusion fire detection network, train the improved dual-stream fusion fire detection network with the fire detection dataset, and obtain a fire detection model.

[0074] like Figure 2 ,The improved dual-stream fusion fire detection network is based on the SlowFast model. The SlowFast network extracts features from the time and space perspectives of time series data. The fast channel aims to extract the time features of time series data. Generally, a recurrent neural network (RNN) or a 3D convolution block is used as the backbone network. The slow channel aims to extract the spatial features of time series data, so general CNN can be used to extract features. And the fast and slow channels set different frame rates. Although the frame rate of the fast channel is set to a higher frame rate, its processing speed is only 20% of the overall calculation. Because the fast channel extraction network does not focus on extracting the spatial features of the data, that is, the channel dimension of the feature extraction is low, and the number of extracted channels is only 1 / 6 of the slow channel. With the characteristics of high temporal resolution, the fast path not only has high input resolution, but also pursues high-resolution features throughout the network hierarchy. No temporal downsampling layer (neither temporal pooling nor time-span convolution) is used in the entire fast path until the global pooling layer before classification. Therefore, the fast channel feature tensor always has frames in the time dimension to maintain temporal fidelity as much as possible. Specifically, it includes the following sub-steps:

[0075] S2.1, set the frame extraction ratio of the fast channel and the slow channel, which is set to 6 in the embodiment, that is, the number of frames extracted per second by the fast channel is 6 times that of the slow channel, and the extracted key frame T is input to the fast channel, and the image size of the frame is 224×224×T;

[0076] S2.2. Feature extraction is performed on the slow channel and the fast channel. The slow channel uses the ResNet-18 network as the backbone network to extract deep features. The Resnet-18 network of the slow channel includes an input layer, a convolutional layer, a BN layer, an activation function RELU, a residual block, a pooling layer, and a fully connected layer. The first layer is a 7×7 convolutional layer. The network contains a total of 4 residual blocks, each residual block has two residual units, and each residual unit contains two 3×3 convolutional layers, followed by a global average pooling layer and a fully connected layer.

[0077] S2.3, when the slow channel is performing deep feature extraction, adding convolution in the time dimension to the deep layer of the network to fuse with the fast channel data stream helps to effectively fuse with the fast channel data stream. In the ResNet-18 network, the Res3 and Res4 layers replace the traditional two-dimensional convolution with a three-dimensional convolution;

[0078] S2.4 and the fast channel use a 3D ResNet-18 network to improve the efficiency of processing high frame rate time series data. The first convolution layer of each residual unit in Res2, Res3, Res4, and Res5 replaces the two-dimensional convolution with a three-dimensional convolution to extract the temporal features between sequence data. The number of channels is set to 1 / 8 of that of the slow channel. The modification of the fast channel mainly replaces the two-dimensional convolution block of each layer of Resnet-18 with a three-dimensional convolution block to extract the temporal features between sequence data.

[0079] S2.5. Input the features of the fast channel and slow channel output into the CMDA module. The CMDA module includes: a channel attention mechanism module and a spatial-temporal attention mechanism module. The CMDA module concatenates the feature vectors of the fast channel and the slow channel to detect fire and combustibles. With only one convolutional layer transformation, the information fused by the fast path may be oblique or indirect to the slow path. The CMDA module for fast and slow channel fusion is mainly composed of two attention mechanism modules, namely the channel attention mechanism module (channel attention) and the spatial-temporal attention mechanism module (spatial-temporalattention), and includes upsampling and downsampling operations.

[0080] Each delayed frame is downsampled along the time dimension of the fast path without any parameters and computational overhead. This simple operation has good interpretability, that is, sequential frame extraction, maintaining high temporal resolution throughout the Fast method. Equally spaced sampling may miss some key frames and cannot guarantee that key frames can be obtained every time. 3D temporal maximum pooling is used, which only considers the time dimension, not the spatial features, and does not destroy the interpretability of temporal downsampling. Afterwards, the fused features are connected to the slow channel.

[0081] The slow channel is fused to the fast channel using a single 1*1*1 Conv layer to reduce and match the channel size. Then, the spatiotemporal attention map is calculated before further operations, that is, by optimizing the execution order and then upsampling by temporal interpolation. The fused features are connected to the original fast channel to detect fire and combustibles.

[0082] S3. Input the fire image into the fire detection model to obtain fire detection parameters and combustible material categories, and then perform data preprocessing to fuse the fire detection parameters and combustible material distribution characteristics obtained after preprocessing to obtain fused flame parameters.

[0083] Data preprocessing includes the following sub-steps:

[0084] S3.1. Extract and store the fire detection parameters: flame height, flame projection area, and flame spread position; the flame spread position is the flame centroid position;

[0085] S3.2. Use a filtering method based on category information to filter and obtain the distribution characteristics of combustibles; specifically, it includes:

[0086] S3.2.1. Traverse the list of detection results and, for each detected target, check whether its category label is a combustible category.

[0087] S3.2.2 For the remaining combustible identification results, choose to set a confidence threshold A. Only when the confidence score of the target is higher than the threshold A, it will be retained; otherwise, it will be filtered out.

[0088] S3.3, smoothing, denoising and regularizing the fire detection parameters. Use Loess, i.e. Locally Weighted Scatterplot Smoothing, which is a non-parametric smoothing method used to fit the smooth curve of the data. The main steps are window width, weighted fitting, and smooth output.

[0089] S3.4, when complex models are used to characterize complex data, overfitting is prone to occur. Regularizing the data can reduce the risk of overfitting. Robust Scaling is a data regularization method that is used to scale data to a range while being robust to outliers. Since the dynamic changes of flames are more obvious at certain moments, the Robust Scaling method uses the median and interquartile range (IQR) of the data for scaling instead of the mean and standard deviation of the data, so it has less impact on outliers. The calculation formula is: .

[0090] The interquartile range (IQR) is the upper quartile (Q 3 ) minus the lower quartile (Q 1 ). Q 2 The middle position of the values ​​in the data set.

[0091] refer to Figure 3 In this embodiment, in order to improve the prediction accuracy of the three fire detection parameters, it is necessary to fuse the combustible distribution characteristics with them. Since the three fire detection parameters are one-dimensional data, the combustible distribution characteristics are two-dimensional data. This is a fusion of data of different modalities. The present invention fuses the two through a cross attention mechanism module. Specifically, it includes the following sub-steps: S301, calculate the fire detection parameters and the combustible distribution characteristics Q i , K , V Value, as follows:

[0092] ,in ， is the linear transformation matrix, y i Represents fire detection parameters, i=1, 2, 3, x Indicates the distribution characteristics of combustibles;

[0093] S302, respectively calculate the similarity between the fire detection parameter Query value and the combustible distribution feature Key to obtain the attention score, and then normalize the attention score through the function; multiply the value corresponding to the Key by the normalized attention score to obtain the attention output of 1 parameter, as shown in the following formula:

[0094] Among them, i=1,2,3.

[0095] in, ; Used to scale the dot product result. d yes k The dimension of T Represents the transpose of a matrix.

[0096] S4. Construct an LSTM flame prediction network, use the fused flame parameters as input, and the corresponding flame height, flame projection area, and flame spread position as output to train the LSTM flame prediction network and obtain a flame prediction model. The preprocessed fire prediction parameters are divided into a training set and a test set at a ratio of 80% and 20%.

[0097] Step S4 is as follows: add a unit state C to the LSTM network throughout the network, build a 2-layer LSTM model, set the hidden layer to 128, and use RMSprop as the optimization algorithm, which uses moving average to adjust the learning rate, so that it is more robust when the update amplitude of different parameters changes greatly. This is to take into account the possibility of rapid changes in flames. Use trial and error to select the learning step size of LSTM, the number of samples put in this time, and the number of iterations of the algorithm.

[0098] Three control gates are added on the basis of RNN: forget gate, input gate, and output gate; the loss function uses the mean square error loss function (MSE), and the evaluation index uses the mean absolute error (MAE) function. The calculation formula of MSE is as follows:

[0099] Where n is the sample of the data set, x m is the output value of the model, x a is the baseline value.

[0100] The forget gate is expressed as follows: , the cell state is initially set to the 0 vector; is the sigmoid function, the output of this function is [0,1], which represents a probability; the past unit state To update; among them, W f represents the weight matrix of the forget gate, x t represents the input of the current time step, b f Represents the bias vector of the forget gate;

[0101] The expression of the input gate is expressed as:

[0102] ;

[0103] ;

[0104] The new unit state C t for: ;

[0105] The output gate is represented as: ;

[0106] in, Indicates the flame height at the last moment, b i represents the bias vector of the input gate, x t represents the input of the current time step, bc The bias vector representing the candidate cell state, Indicates the candidate unit status, b o represents the bias vector of the output gate, h t represents the hidden state of the current time step, o t represents the output of the output gate, W i represents the weight matrix of the input gate, W c The weight matrix representing the candidate unit state, i t represents the output of the input gate, W o represents the weight matrix of the output gate, f t represents the output of the forget gate, Indicates the state of the unit after activation.

[0107] Use the cyclic selection method to select the best model. Select the memory duration, the number of fully connected layers and neural units in each layer for model testing. You can choose 64 combinations of model parameters. Test the 64 combination models one by one, calculate the function value of MAE, determine the model parameters of the minimum value of the MAE function, and save the parameters. MAE calculation formula: , where n is the number of samples in the data set, a i and b i are the true value and predicted value of the prediction task at the moment.

[0108] S5. Construct a ConvLSTM fire prediction network, take the fire detection parameters and the distribution characteristics of combustibles as input, and the corresponding flame position and size at the next moment as output, train the ConvLSTM fire prediction network, and obtain a fire prediction model; different from the flame prediction model, this embodiment mainly considers running it in the server, which is conducive to the fire department and other departments to analyze the cause of the fire in the background. It includes the following sub-steps:

[0109] S5.1. Set the network parameters of the ConvLSTM encoding layer, including the number of ConvLSTM layers, input parameters, and output parameters; ConvLSTM performs convolution calculations, inputs and updates the unit state, forgets, and outputs according to the gate structure, as follows:

[0110] ;

[0111] in represents convolution, Represents the inner product of corresponding elements in the same position; it is the input gate at time t, f t is the forget gate at time t, C t is the cell state at time t, o t The output gate at time t, h t is the output at time t, x t is the input at time t, and each matrix W and vector b are the weights and bias values ​​to be learned during model training. and tanh is the activation function, and * is the convolution operation.

[0112] S5.2. The spatiotemporal features extracted by the deep ConvLSTM are used as the input of the spatiotemporal attention mechanism to focus on important feature information. The ConvLSTM features in the encoding stage are advanced, and a convolution layer is added with a convolution kernel size of 1×1×3. Through the setting of the convolution kernel, the image has only one channel during the processing process, which can reduce the calculation complexity of the model.

[0113] S5.3, the features processed by the attention mechanism are sent to the prediction network. The prediction network consists of ConvLSTM cells with a depth of 3 layers. T frames of images are sent at a time, i.e., sequence data. The convolution kernel size of each ConvLSTM cell is 3×3, stride=2, and padding is the same. The output is that every T frames of flame image predicts the next 5 frames of flame image. The number of predicted future frames here can be set according to the experimental results. The specific number of frames that can be predicted depends on the frame number setting of model training and the error accumulation of recursive prediction. The present invention is set to 5 frames.

[0114] In order to consider the relationship between the distribution of combustibles and the spread of fire, the spatiotemporal feature information of the extracted fire images is weighted by importance through the spatiotemporal attention mechanism after encoding. Through training, the model can pay more attention to the important time and space points where the fire is affected by the distribution of combustibles.

[0115] The ConvLSTM settings in the decoding stage are similar to those in the encoding stage. A convolution layer is added at the end of the decoding stage to adjust the image size to output the prediction result. The prediction stage is to extract the fire picture in real time from the first T frames after the encoding stage and spatiotemporal attention calculation, and predict it as the fire spread picture of the next frame, that is, asynchronous prediction. Asynchronous prediction allows the model to process the input separately at each time step and generate the corresponding output, which can reflect the current fire status in real time and is more suitable for application scenarios that require timely adjustment and feedback.

[0116] This embodiment predicts 5 frames of fire images for every T frames of fire images, that is, every T+5 frames of data are a group, the first T frames are the input of the model, and the last 5 frames are data labels. The data is divided into a training set and a validation set in an 8:2 ratio, and the number of iterations is 1000.

[0117] S6. Based on the fire detection model, the flame prediction model and the fire prediction model, a fire situation prediction model is constructed, and the collected fire video is input into the fire situation prediction model to obtain the flame height, flame projection area, flame spread position, and the position and size of the flame at the next moment.

[0118] Although the present invention has been described above with preferred embodiments, it is not intended to limit the present invention. A person skilled in the art of the present invention may make various modifications and improvements without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the definition of the claims.

Claims

1. A method for predicting the fire situation at the initial stage of a fire in a building, characterized in that: The following steps are involved: S1. Based on the fire scene video, extract the key frames of the video and construct a fire detection dataset, including the key frames of the fire video and the fire detection parameter labels and combustible category labels corresponding to the key frames; S2. Build an improved dual-stream fusion fire detection network, and use the fire detection data set to train the improved dual-stream fusion fire detection network to obtain a fire detection model; S3, input the fire image into the fire detection model to obtain fire detection parameters and combustible material categories, and then perform data preprocessing, fuse the fire detection parameters and combustible material distribution characteristics obtained after preprocessing, and obtain fused flame parameters; S4. Construct an LSTM flame prediction network, use the fused flame parameters as input, and the corresponding flame height, flame projection area, and flame spread position as output to train the LSTM flame prediction network to obtain a flame prediction model. S5. Construct a ConvLSTM fire prediction network, take the fire detection parameters and the distribution characteristics of combustibles as input, and the corresponding flame position and size at the next moment as output, train the ConvLSTM fire prediction network, and obtain a fire prediction model; S6. Based on the fire detection model, the flame prediction model and the fire prediction model, a fire situation prediction model is constructed, and the collected fire video is input into the fire situation prediction model to obtain the flame height, flame projection area, flame spread position, and the position and size of the flame at the next moment.

2. A method for predicting the initial fire situation in a building according to claim 1, characterized in that: In step S1, extracting the key frames of the video includes the following sub-steps: S1.

1. Read the video frame and convert it into a grayscale image. Calculate the difference between the current frame and the two previous and next frames using the three-frame difference method. The calculation formula of the three-frame difference method is as follows: , in, Represents the image grayscale values ​​of the three consecutive frames before and after, and the three frames are subtracted from each other to obtain the difference image, and the difference image is obtained by performing the AND operation ; S1.

2. Use the entropy extreme value of the differential image to determine whether the current frame is a key frame. Count the number of gray levels of pixels in the differential image. Use the gray histogram to calculate the probability of each gray level. Divide the number of pixels at each gray level by the total number of pixels in the image frame to calculate the probability distribution of each gray level, which is the probability density function. ; S1.3, performing median filtering and smoothing processing on the obtained probability density function; S1.4, normalize the probability density function so that the sum of the probability density is equal to 1, which is used to represent the relative weight of each gray level; S1.

5. Probability density function of each frame of the fire video obtained in step S1.2 , the entropy value of each frame is obtained by the entropy value calculation formula, the calculation formula is as follows: ; S1.

6. The calculated entropy value Mapped to the two-dimensional coordinate axis x with the image frame number as the axis, and finally taking the image number with the extreme value of the differential image entropy in the segment as the key frame in the segment n ; S1.

7. Return to step S1.1 until all video frames are traversed and all key frames are obtained.

3. A method for predicting the initial fire situation in a building according to claim 1, characterized in that: In step S2, the improved dual-stream fusion fire detection network is constructed based on the SlowFast network, including a fast channel and a slow channel; the fast channel is used to extract the time characteristics of the time series data; The slow channel is used to extract the spatial features of time series data.

4. A method for predicting the initial fire situation in a building according to claim 3, characterized in that: Building an improved two-stream fusion fire detection network includes the following steps: S2.1, set the frame extraction ratio of the fast channel and the slow channel, that is, the number of frames extracted per second by the fast channel is a preset multiple of the slow channel, and input the extracted key frame T into the fast channel, and the image size of the frame is 224×224×T; S2.2, the slow channel uses the ResNet-18 network as the backbone network to extract deep features. The slow channel Resnet-18 network includes input layer, convolution layer, BN layer, activation function RELU, residual block, pooling layer, and fully connected layer; the first layer is a 7×7 convolution layer. The network contains 4 residual blocks, each residual block has two residual units, and each residual unit contains two 3×3 convolution layers, followed by a global average pooling layer and a fully connected layer; S2.3, when the slow channel performs deep feature extraction, the convolution in the time dimension is added to the deep layer of the network to fuse with the fast channel data stream. In the ResNet-18 network, the traditional two-dimensional convolution is replaced by three-dimensional convolution in the Res3 and Res4 layers; S2.4, the fast channel uses a 3D ResNet-18 network. In the first convolution layer of each residual unit of Res2, Res3, Res4, and Res5, the two-dimensional convolution is replaced by a three-dimensional convolution to extract the temporal features between sequence data; S2.

5. Input the features of the fast channel and the slow channel output into the CMDA module. The CMDA module includes: a channel attention mechanism module and a spatiotemporal attention mechanism module. The CMDA module concatenates the feature vectors of the fast channel and the slow channel to detect fire and combustibles.

5. The method for predicting the initial fire situation in a building according to claim 1, characterized in that: In step S3, data preprocessing includes the following sub-steps: S3.

1. Extract and store the fire detection parameters: flame height, flame projection area, and flame spread position; the flame spread position is the flame centroid position; S3.

2. Use a filtering method based on category information to filter and obtain the distribution characteristics of combustibles; S3.

3. Smoothing, denoising and regularization of fire detection parameters.

6. A method for predicting the initial stage of a fire in a building according to claim 1, characterized in that: In step S3, the fire detection parameters and combustible distribution features are fused using a cross-attention mechanism to obtain fused flame parameters, which specifically includes the following sub-steps: S301, calculate the Q of combustible distribution characteristics i , K, V values, as follows: ,in ， is the linear transformation matrix, y i Represents fire detection parameters, i=1, 2, 3, x Indicates the distribution characteristics of combustibles; S302, respectively calculate the similarity between the fire detection parameter Query value and the combustible distribution feature Key to obtain the attention score, and then normalize the attention score through the function; multiply the value corresponding to the Key by the normalized attention score to obtain the attention output of 1 parameter, as shown in the following formula: Where, i=1,2,3; ; Used to scale the dot product result. d yes k The dimension of T Represents the transpose of a matrix.

7. A method for predicting the initial stage of a fire in a building according to claim 1, characterized in that: Step S4 is as follows: the LSTM network adds a unit state C throughout the network, and adds three control gates based on the RNN: forget gate, input gate, and output gate; The forget gate is expressed as follows: , the cell state is initially set to the 0 vector; is the sigmoid function, the output of this function is [0,1], which represents a probability; the past unit state To update; W f represents the weight matrix of the forget gate, x t represents the input of the current time step, b f Represents the bias vector of the forget gate; the expression of the input gate is expressed as: ； ； The new unit state C t for: ; The output gate is represented as: ; in, Indicates the flame height at the last moment, b i represents the bias vector of the input gate, x t represents the input of the current time step, b c The bias vector representing the candidate cell state, Indicates the candidate unit status, b o represents the bias vector of the output gate, h t represents the hidden state of the current time step, o t represents the output of the output gate, W i represents the weight matrix of the input gate, W c The weight matrix representing the candidate unit state, i t represents the output of the input gate, W o represents the weight matrix of the output gate, f t represents the output of the forget gate, Indicates the state of the unit after activation.

8. A method for predicting the initial stage of a fire in a building according to claim 7, characterized in that: Step S5 includes the following sub-steps: S5.

1. Set the network parameters of the ConvLSTM encoding layer, including the number of ConvLSTM layers, input parameters, and output parameters; ConvLSTM performs convolution calculations, inputs and updates the unit state, forgets, and outputs according to the gate structure, as follows: ； in represents convolution, Represents the inner product of corresponding elements in the same position; S5.2, the spatiotemporal features extracted by deep ConvLSTM are used as the input of the spatiotemporal attention mechanism to focus on important feature information; S5.3, the features processed by the attention mechanism are sent to the prediction network. The prediction network consists of ConvLSTM cells with a depth of 3 layers. The output is that every T frames of flame images predict the next 5 frames of flame images.

Images