Fire-fighting video fireworks intelligent identification and alarm method

Abstract

The application relates to the technical field of image data processing, and discloses a fire-fighting video smoke and fire intelligent recognition and alarm method. The method acquires a continuous video frame sequence, extracts multi-scale basic feature maps through a feature pyramid network, applies a deformable time convolution module on each scale feature map, predicts the current frame sampling point position according to the offset of the previous frame smoke area, and aggregates the micro smoke movement features of adjacent multiple frames, inputs the aggregated features into a time consistency verification module, extracts the long-term time sequence dependence relationship of smoke area expansion and gray gradient by using a long short-term memory network, outputs a smoke confidence according to the dependence relationship, and generates an alarm signal when the confidence exceeds a preset threshold and the time sequence gradient presents a positive growth trend. The scheme overcomes the problem that a fixed geometric structure convolution is insufficient for extracting the edge features of micro irregular smoke, enhances the capturing capacity for early weak smoke movement tracks, and reduces the false negative rate of early fires.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology and discloses a method for intelligent identification and alarm of smoke and fire in fire-fighting videos. Background Technology

[0002] Current conventional smoke recognition solutions in the field of fire protection video surveillance mainly rely on deep learning networks. The closest existing technical solutions typically input continuous video frames into a feature extraction network, extracting basic feature layers of different spatial resolutions through multiple convolutional layers from the bottom up, and then fusing the features from different layers. When processing video temporal information, this approach generally uses standard convolutional kernels or fixed-stride 3D convolutions to perform feature extraction and fusion within a fixed time window, and determines the presence of smoke targets in the video frame based on the fused features.

[0003] The aforementioned conventional technical solutions have specific limitations when dealing with early-stage fire scenarios. The fixed geometry of standard convolutions can only extract features according to a set grid stride. Early-stage, minute smoke particles, driven by ambient airflow, exhibit random and irregular trajectories, and the fixed grid stride cannot accurately reflect this microscopic deformation process. Fixed-stride 3D convolutions, limited by their fixed receptive field, cannot establish smoke gradation dependencies spanning longer periods. This results in the network's inability to extract motion trajectory features that adapt to irregular deformations and long-period gradation features when processing early, weak smoke, leading to missed detections of early-stage, minute smoke. Summary of the Invention

[0004] The purpose of this invention is to provide a method for intelligent identification and alarm of smoke and fire in fire protection video, which can solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for intelligent identification and alarm of smoke and fire in fire-fighting video includes: acquiring a continuous video frame sequence of a fire monitoring area to be detected, inputting the continuous video frame sequence into a preset feature pyramid network, and extracting multi-scale basic feature maps corresponding to the continuous video frame sequence;

[0007] A deformable temporal convolution module is applied to the multi-scale basic feature map at each scale to predict the sampling point position of the current frame based on the offset of the smoke region in the previous frame in the continuous video frame sequence, and to aggregate the small smoke motion features of multiple adjacent frames based on the sampling point position of the current frame.

[0008] The aggregated features from each scale are input into the temporal consistency verification module, and the long short-term memory network is used to extract the long-term temporal dependency between smoke area expansion and grayscale gradation.

[0009] Smoke confidence is output based on the long-term temporal dependency relationship. When the smoke confidence exceeds a preset alarm threshold and the temporal gradient of the smoke confidence shows a positive increasing trend, a fire alarm signal is generated.

[0010] Preferably, the step of inputting the continuous video frame sequence into a preset feature pyramid network to extract the multi-scale basic feature map corresponding to the continuous video frame sequence includes: inputting the current frame in the continuous video frame sequence into multiple convolutional layers from bottom to top to extract basic feature layers with different spatial resolutions.

[0011] Upsampling and dimensionality reduction are performed on the basic feature layers of adjacent levels. The upsampled basic feature layers are then added and fused element by element with the adjacent shallow basic feature layers to generate the multi-scale basic feature map.

[0012] In the process of dimensionality reduction of the basic feature layer, depthwise separable convolution is used instead of standard convolution to preserve the spatial distribution details of tiny smoke particles at different scales.

[0013] Preferably, the step of predicting the sampling point position of the current frame based on the offset of the smoke region in the previous frame in the continuous video frame sequence includes: concatenating the feature map of the smoke region at the scale corresponding to the previous frame with the feature map of the current frame at the scale corresponding to the current frame, and inputting it into the offset prediction convolutional layer.

[0014] The offset prediction convolutional layer outputs a three-dimensional offset field centered on the coordinates of each feature point in the current frame. The three-dimensional offset field includes horizontal offset, vertical offset, and time dimension offset.

[0015] The geometric sampling position of the feature points in the current frame is adjusted according to the three-dimensional offset field, so that the sampling position is adaptively offset along the motion trajectory of the tiny smoke particles driven by airflow.

[0016] Preferably, the step of aggregating the micro smoke motion features of adjacent multiple frames based on the sampling point position of the current frame includes: extracting multi-frame sampling feature vectors from the feature maps of the corresponding scales of the adjacent multiple frames based on the sampling point position of the current frame;

[0017] The multi-frame sampled feature vectors are input into the temporal attention calculation layer to calculate the relevance score of each frame sampled feature vector in the adjacent multi-frames relative to the current frame sampled feature vector.

[0018] The correlation score is used to perform a weighted summation of the sampled feature vectors of each of the adjacent frames, suppressing background static features in the adjacent frames that are not related to the smoke motion of the current frame, and outputting the aggregated micro smoke motion features.

[0019] Preferably, the step of extracting the long-term temporal dependency relationship between smoke area expansion and grayscale gradient using a long short-term memory network includes: performing feature decoding on the aggregated micro smoke motion features to extract the smoke bounding box coordinates and the average grayscale value of the smoke region in the current frame;

[0020] The difference sequence between the area change of the smoke bounding box coordinates and the average gray value of the smoke region is concatenated to form a temporal state vector;

[0021] The temporal state vector is input into the long short-term memory network in time steps. Through the cell state update mechanism of the long short-term memory network, the area increasing feature and grayscale decay feature across multiple time steps are preserved.

[0022] Preferably, the step of generating a fire alarm signal when the smoke confidence exceeds a preset alarm threshold and the temporal gradient of the smoke confidence shows a positive increasing trend includes: constructing the smoke confidence sequence within the current time window into a confidence time curve;

[0023] Perform a first-order difference operation on the confidence time curve to extract the confidence difference value between adjacent time steps;

[0024] When the duration for which the confidence difference value remains in a positive range exceeds a preset duration threshold, and the current smoke confidence exceeds the preset alarm threshold, it is determined that the time-series gradient shows a positive increasing trend, triggering the generation of the fire alarm signal.

[0025] Preferably, before fusing the upsampled basic feature layer with the adjacent shallow basic feature layer element by element, the method further includes: inputting the upsampled basic feature layer and the adjacent shallow basic feature layer into a spatial attention guidance network;

[0026] The spatial attention guidance network performs max pooling and average pooling in the channel dimension, and after concatenation, generates a spatial weight matrix.

[0027] The spatial weight matrix is ​​used to enhance the tiny smoke edge pixels in the adjacent shallow basic feature layers with high weights and suppress the background pixels with low weights. The enhanced adjacent shallow basic feature layers are then fused with the upsampled basic feature layers element by element.

[0028] Preferably, after adjusting the geometric sampling position of the current frame feature points according to the three-dimensional offset field, the method further includes: applying a smoothness constraint to the adjusted geometric sampling position;

[0029] Extract the initial optical flow field of the smoke region in the previous frame, and calculate the optical flow deviation matrix between the three-dimensional offset field and the initial optical flow field;

[0030] The optical flow bias matrix is ​​introduced as a regularization penalty term into the loss function of the offset prediction convolutional layer.

[0031] Preferably, in the process of using the correlation score to perform a weighted summation of the sampled feature vectors of each of the adjacent multi-frames, the method further includes: extracting the area of ​​the smoke bounding box corresponding to each of the adjacent multi-frames.

[0032] Calculate the absolute value of the rate of change of the smoke bounding box area of ​​each of the adjacent frames relative to the smoke bounding box area of ​​the current frame;

[0033] The absolute value of the rate of change is multiplied into the relevance score as a decay factor to generate the dynamically adjusted final attention weight.

[0034] The final attention weight is used to reduce the contribution of historical frames in which the smoke area shrinks in adjacent frames to the aggregated features of the current frame.

[0035] Preferably, in the process of retaining the area increase feature and grayscale decay feature across multiple time steps through the cell state update mechanism of the long short-term memory network, the method further includes: introducing a background suppression bias term in the forgetting gating calculation of the long short-term memory network.

[0036] Extract the initial background frame of the fire monitoring area to be detected, and calculate the static background difference feature map between the current frame and the initial background frame;

[0037] The static background difference feature map is mapped to the background suppression bias term.

[0038] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0039] 1. This invention applies a deformable temporal convolution module to multi-scale basic feature maps at various scales. Based on the offset of the smoke region in the previous frame, it predicts the sampling point position in the current frame, causing the sampling position to adaptively shift along the motion trajectory of tiny smoke particles driven by airflow. By combining a long short-term memory network to extract the long-term temporal dependency between smoke area expansion and grayscale gradation, it overcomes the problems of fixed-geometric convolution failing to match the irregular deformation of tiny smoke particles and fixed-stride 3D convolution struggling to establish long-term temporal gradation dependencies. This enhances the ability to capture the motion trajectory of early, weak smoke particles and reduces the false negative rate of early fires.

[0040] 2. In the feature extraction stage, depthwise separable convolution is used for dimensionality reduction, preserving the spatial distribution details of tiny smoke particles at different scales. In the temporal aggregation stage, a temporal attention layer is introduced to calculate relevance scores, suppressing interference from static background features. An optical flow deviation matrix is ​​constructed using an initial optical flow field as a regularization penalty, limiting the variation of the 3D offset field between spatially adjacent sampling points and filtering out anomalous jumps in offset caused by random turbulence. The absolute value of the area change rate is used as a decay factor to adjust the attention weights, reducing the contribution of historical frames to smoke area shrinkage. In the forgetting gate calculation of the Long Short-Term Memory network, a background suppression bias term based on the background difference feature map is introduced, blocking the transmission of static background interference features to cell states in subsequent time steps and eliminating the influence of background interference on the accumulation of temporal features. Attached Figure Description

[0041] Figure 1 This is the main flowchart of the present invention;

[0042] Figure 2 This is a flowchart of the multi-scale basic feature map extraction process of the present invention;

[0043] Figure 3 This is a flowchart of the deformable temporal convolution offset prediction process of the present invention;

[0044] Figure 4 This is a flowchart illustrating the aggregation of minute smoke motion features according to the present invention.

[0045] Figure 5 This is a flowchart of the long-term temporal dependency extraction process of the present invention;

[0046] Figure 6 This is a flowchart of the alarm signal generation and background suppression process of the present invention. Detailed Implementation

[0047] Please refer to the attached document. Figure 1 This embodiment provides a method to input a sequence of decoded, continuous video frames into a preset feature pyramid network to extract multi-scale basic feature maps corresponding to the continuous video frame sequence. The feature pyramid network includes a bottom-up feature extraction path and a top-down feature fusion path. The bottom-up feature extraction path includes multiple cascaded convolutional units. Each convolutional unit includes at least one convolutional layer and one non-linear activation layer. The kernel size of the convolutional layer is a fixed value, and the non-linear activation layer uses the ReLU activation function.

[0048] The current frame image data in a continuous video frame sequence is input to the first convolutional unit. After convolution and non-linear activation in the first convolutional unit, a first-level basic feature layer is generated. The spatial resolution of the first-level basic feature layer is consistent with the spatial resolution of the current frame image data. The first-level basic feature layer is input to the next convolutional unit. After convolution and non-linear activation, a second-level basic feature layer is generated. The spatial resolution of the second-level basic feature layer is half that of the first-level basic feature layer. This process continues, with the bottom-up feature extraction path outputting multiple basic feature layers with different spatial resolutions. The spatial resolution of each subsequent basic feature layer is half that of the previous layer, and the number of channels in each subsequent basic feature layer is twice that of the previous layer. Different levels of basic feature layers correspond to different receptive fields. Lower-level basic feature layers contain rich spatial detail features, while higher-level basic feature layers contain rich semantic features.

[0049] The top-down feature fusion path upsamples the highest-level basic feature layer using bilinear interpolation to ensure the spatial resolution of the upsampled feature map matches that of the adjacent lower-level basic feature layer. The upsampled feature map is then element-wise added to the adjacent lower-level basic feature layer to generate a fused feature map. This fused feature map is then convolved to eliminate aliasing during upsampling, generating a multi-scale basic feature map of the corresponding scale. This process of upsampling, feature addition, and convolution is repeated for each level of the fused feature map, generating multiple multi-scale basic feature maps with different spatial resolutions. Each multi-scale basic feature map corresponds to a feature representation at one scale, and all multi-scale basic feature maps constitute a multi-scale basic feature map set.

[0050] In this embodiment, the hierarchical correspondence and feature attributes of the multi-scale basic feature maps are shown in the table below:

[0051] Table 1. Hierarchical correspondence and feature attribute table of multi-scale basic feature maps

[0052] Feature level Spatial resolution relative proportion Relative proportion of channel number Core Feature Attributes Corresponding smoke target detection scenarios Level 1 1:1 1:1 Pixel-level spatial detail features and edge texture features Early extraction of edge contours and spatial distribution features of micro-smoke Level 2 1:2 2:1 Local region texture features, small target semantic features Extraction of local motion features of small-scale diffuse smoke Level 3 1:4 4:1 Regional semantic features and shape features Extraction of overall morphological features of smoke with medium diffusion range Level 4 1:8 8:1 Global semantic features, scene context features Global scene feature recognition for large-scale smoke diffusion Level 5 1:16 16:1 High-level abstract semantic features and category distinguishing features Classification and semantic recognition of smoke and interfering targets

[0053] The table above clarifies the feature attributes and applicable scenarios corresponding to different levels of multi-scale basic feature maps. Low-level features correspond to the extraction of spatial detail features of small smoke particles, while high-level features correspond to the semantic feature recognition of smoke targets, avoiding the omission of features of small smoke particles or large-scale smoke particles by single-scale features.

[0054] Deformable temporal convolutional modules are applied to multi-scale basic feature maps at various scales. Based on the offset of the smoke region in the previous frame within a continuous video frame sequence, the sampling point position of the current frame is predicted. Then, based on the sampling point position of the current frame, minute smoke motion features from adjacent frames are aggregated. The deformable temporal convolutional modules are configured separately for each scale's multi-scale basic feature map. The input to each deformable temporal convolutional module is a multi-frame feature map within a continuous time window at the corresponding scale. This multi-frame feature map within the continuous time window includes the multi-scale basic feature map at the scale corresponding to the current frame, as well as the multi-scale basic feature maps at the corresponding scales from the preceding consecutive frames.

[0055] The smoke region in the previous frame is the region containing smoke features identified through feature recognition in the previous frame image. The feature map at the scale corresponding to the smoke region in the previous frame is the feature region in the multi-scale basic feature map at the same scale as the smoke region in the previous frame, whose spatial position matches that of the smoke region in the previous frame. The feature map at the scale corresponding to the smoke region in the previous frame is concatenated with the multi-scale basic feature map at the same scale in the current frame along the channel dimension to generate a concatenated feature map. The concatenated feature map is then input into the offset prediction convolutional layer. The kernel size of the offset prediction convolutional layer is consistent with the standard kernel size of the deformable temporal convolution module. The offset prediction convolutional layer outputs a three-dimensional offset field centered on the coordinates of each feature point in the current frame. The three-dimensional offset field includes horizontal offset, vertical offset, and temporal offset. Each sampling point corresponds to a set of three-dimensional offsets, and the values ​​of the three-dimensional offsets are continuous real numbers used to adjust the position of the sampling point in the spatial and temporal dimensions.

[0056] The geometric sampling positions of the feature points in the current frame are adjusted based on the 3D offset field, so that the sampling positions are adaptively offset along the motion trajectory of the tiny smoke particles driven by airflow. The standard sampling grid corresponding to each output feature point in the current frame feature map consists of multiple uniformly distributed sampling points centered on the coordinates of that feature point, and the coordinates of the standard sampling points are preset fixed coordinates. The horizontal and vertical offsets corresponding to each standard sampling point in the 3D offset field are superimposed on the x and y coordinates of that standard sampling point, respectively, to generate the adjusted spatial sampling position; the temporal offset is superimposed on the time step of the current frame to generate the adjusted temporal sampling position, which corresponds to adjacent frames within a continuous time window.

[0057] Based on the adjusted spatial and temporal sampling positions, feature vectors for the corresponding sampling positions are extracted from the feature maps of adjacent frames at the corresponding scales within a continuous time window using bilinear interpolation. Each sampling position corresponds to a sampling feature vector, and the sampling feature vectors of all sampling positions constitute a set of multi-frame sampling feature vectors.

[0058] A set of multi-frame sampled feature vectors is input into the temporal attention computation layer. The layer performs a dot product operation between the sampled feature vector of each historical frame and the sampled feature vector of the current frame, calculating the relevance score of each historical frame's sampled feature vector relative to the current frame's sampled feature vector. The relevance scores of all historical frames are normalized to generate normalized relevance weights. These normalized relevance weights are then used to weight and sum the sampled feature vectors of the corresponding historical frames, generating a weighted feature vector. This weighted feature vector is then concatenated with the sampled feature vector of the current frame to generate aggregated micro-smoke motion features. This aggregation process suppresses background static features in adjacent frames that are irrelevant to the smoke motion of the current frame, highlighting dynamic features related to the smoke motion.

[0059] The aggregated features from each scale are input into the temporal consistency verification module, which utilizes a Long Short-Term Memory (LSTM) network to extract the long-term temporal dependency between smoke area expansion and grayscale gradation. The input to the temporal consistency verification module is the aggregated minute smoke motion features from each scale. This module comprises a feature decoding submodule and an LSM network. The feature decoding submodule decodes the aggregated minute smoke motion features from each scale, including convolution operations, bounding box regression, and classification operations, outputting the smoke bounding box coordinates and the classification confidence score of the smoke region in the current frame. The smoke bounding box coordinates include the x-coordinate of the top-left corner, the y-coordinate of the top-left corner, the x-coordinate of the bottom-right corner, and the y-coordinate of the bottom-right corner of the smoke region. The smoke region area in the current frame is calculated based on these coordinates. The average grayscale value of the smoke region is the arithmetic mean of the grayscale values ​​of all pixels within the region matching the smoke bounding box coordinates in the current frame image.

[0060] The smoke region area and average gray value of the smoke region at multiple consecutive time steps are processed to generate a temporal state vector. The change in smoke region area between adjacent time steps is calculated; this change is the difference between the smoke region area at the current time step and the area at the previous time step. The difference in average gray value of the smoke region between adjacent time steps is also calculated; this difference is the difference between the average gray value of the smoke region at the current time step and the average gray value at the previous time step. The area change and gray value difference values ​​from multiple consecutive time steps are concatenated in chronological order to generate the temporal state vector, with the dimension matching the number of time steps.

[0061] The temporal state vector is input into a Long Short-Term Memory (LSTM) network step by step. The LSTM network contains multiple cascaded LSTM units, each corresponding to the temporal state vector input for one time step. Each LSTM unit includes a forget gate, an input gate, a cell state gate, and an output gate. The forget gate receives the temporal state vector of the current time step and the hidden state of the previous time step, outputting forget gate weights to control the information to be retained in the cell state of the previous time step. The input gate receives the temporal state vector of the current time step and the hidden state of the previous time step, outputting input gate weights to control the content to be updated to the cell state in the input information of the current time step. The cell state is updated according to the forget gate weights and the input gate weights, retaining feature information spanning multiple time steps. The output gate receives the temporal state vector of the current time step and the hidden state of the previous time step, outputting output gate weights to control the output of the hidden state of the current time step. Through the cell state update mechanism of the LSTM network, the area increase and grayscale decay features spanning multiple time steps are retained, extracting the long-term temporal dependency between smoke area expansion and grayscale gradation.

[0062] Smoke confidence is output based on long-term temporal dependencies. When the smoke confidence exceeds a preset alarm threshold and the temporal gradient of the smoke confidence shows a positive increasing trend, a fire alarm signal is generated. The final hidden state of the Long Short-Term Memory (LSTM) network is output to a fully connected layer. The fully connected layer performs a linear transformation and non-linear activation on the final hidden state, outputting the smoke confidence at the current time step. The smoke confidence value ranges from 0 to 1; a higher smoke confidence value indicates a higher probability that the current frame contains a real smoke target. The preset alarm threshold is a fixed value set in advance, ranging from 0 to 1. The smoke confidence of multiple consecutive time steps within the current time window is obtained and arranged in chronological order to construct a confidence-time curve.

[0063] A first-order difference operation is performed on the confidence-time curve to calculate the confidence difference value between adjacent time steps. The confidence difference value is the difference between the smoke confidence level at the current time step and the smoke confidence level at the previous time step. When the duration for which the confidence difference value remains continuously in the positive range exceeds a preset duration threshold, and the smoke confidence level at the current time step exceeds a preset alarm threshold, the time-series gradient is determined to be showing a positive increasing trend, triggering the generation of a fire alarm signal. The fire alarm signal is transmitted to the fire monitoring platform through the communication unit, simultaneously triggering the audible and visual alarm devices at the monitoring site.

[0064] In this embodiment, please refer to the appendix. Figure 2By extracting multi-scale basic feature maps of continuous video frame sequences through a feature pyramid network, feature coverage of smoke targets of different sizes is achieved; by predicting the sampling point position of the current frame through a deformable temporal convolution module, the sampling position is adapted to the motion trajectory of the smoke, and the small smoke motion features of adjacent frames are aggregated; by extracting the long temporal dependency relationship between smoke area expansion and grayscale gradation through a long short-term memory network, the long-period gradation feature of the smoke target is modeled; and by making dual judgments on smoke confidence and temporal gradient, a fire alarm signal is generated.

[0065] In one optional embodiment, the current frame from a continuous video frame sequence is sequentially input into multiple bottom-up convolutional layers to extract basic feature layers with different spatial resolutions. Upsampling and dimensionality reduction are performed on adjacent basic feature layers. The upsampled basic feature layers are then fused element-wise with adjacent shallow basic feature layers to generate multi-scale basic feature maps. When performing dimensionality reduction on the basic feature layers, depthwise separable convolution is used instead of standard convolution. Depthwise separable convolution includes two steps: depthwise convolution and pointwise convolution. Depthwise convolution uses an independent convolution kernel for each input channel, while pointwise convolution uses a 1×1 convolution kernel to transform the channel dimension of the output feature map from the depthwise convolution, achieving dimensionality reduction. During the operation of depthwise separable convolution, the spatial feature information of each input channel is transmitted independently, avoiding the loss of spatial distribution details of micro-smoke during channel fusion caused by standard convolution, thus preserving the spatial distribution details of micro-smoke at different scales.

[0066] Before element-wise addition and fusion of the upsampled base feature layer with the adjacent shallow base feature layer, the upsampled base feature layer and the adjacent shallow base feature layer are input into the spatial attention guidance network. The spatial attention guidance network performs max pooling and average pooling operations on the input feature map along the channel dimension. The max pooling operation extracts the maximum feature value of all channels at each spatial location to generate a max pooled feature map; the average pooling operation calculates the average feature value of all channels at each spatial location to generate an average pooled feature map.

[0067] The max-pooling and average-pooling feature maps have the same spatial resolution as the input feature map, and each map has one channel. The max-pooling and average-pooling feature maps are concatenated along the channel dimension to generate a concatenated pooling feature map with two channels. This concatenated pooling feature map is then input into a convolutional layer and processed by a sigmoid activation function to generate a spatial weight matrix. Each element of the spatial weight matrix ranges from 0 to 1, corresponding to the weight coefficient at each spatial location in the input feature map. The spatial weight matrix is ​​then element-wise multiplied with adjacent shallow basic feature layers. This results in high-weight enhancement of small smoke edge pixels in the adjacent shallow basic feature layers and low-weight suppression of background pixels, generating enhanced adjacent shallow basic feature layers. Finally, these enhanced adjacent shallow basic feature layers are element-wise added to and fused with the upsampled basic feature layers to generate a fused multi-scale basic feature map.

[0068] The computational process of the spatial attention-guided network described above is quantitatively described by the following mathematical formula:

[0069]

[0070]

[0071] in, The input consists of adjacent shallow basic feature layers. This represents the number of channels in the feature map. The spatial coordinates of the feature map For max pooling feature maps, This is the average pooling feature map.

[0072]

[0073] in, This is a splicing operation at the channel dimension. The kernel size is Convolution operation, It is the sigmoid activation function. This is the spatial weight matrix.

[0074]

[0075] in, This is an element-wise multiplication operation. This is the enhanced adjacent shallow basic feature layer.

[0076] Please refer to the attached document. Figure 3The feature map at the scale corresponding to the smoke region in the previous frame is concatenated with the feature map at the scale corresponding to the current frame, and then input into the offset prediction convolutional layer. The feature map at the scale corresponding to the smoke region in the previous frame is a feature region in the multi-scale basic feature map at the scale corresponding to the previous frame that matches the spatial position of the smoke bounding box identified in the previous frame. The spatial coordinates of this feature region correspond one-to-one with the coordinates of the smoke bounding box in the previous frame, ensuring that the input features for offset prediction are concentrated in the area of ​​smoke movement and avoiding interference from background features.

[0077] The input to the offset prediction convolutional layer is a concatenated feature map, the number of channels of which is the sum of the number of channels in the two input feature maps. The kernel size of the offset prediction convolutional layer is 33, with a stride of 1 and padding of 1, ensuring that the spatial resolution of the output offset field feature map is consistent with the spatial resolution of the input feature map. The output of the offset prediction convolutional layer has 3K×K channels, where K is the kernel size of the deformable temporal convolutional module. Each sampling point corresponds to an offset in three dimensions: horizontal offset, vertical offset, and temporal offset, forming a three-dimensional offset field.

[0078] The prediction process of the above three-dimensional migration field is quantitatively described by the following mathematical formula:

[0079]

[0080] in, The coordinates of the feature points in the feature map of the current frame. For feature points The corresponding three-dimensional offset field, The operation of the convolutional layer is to predict the offset. This is a splicing operation at the channel dimension. The previous frame Corresponding scale feature map The feature vector of the location, For the current frame Corresponding scale feature map The feature vector of the location.

[0081] The geometric sampling positions of the feature points in the current frame are adjusted based on the 3D offset field, so that the sampling positions are adaptively offset along the motion trajectory of the tiny smoke particles driven by airflow. The adjustment process is quantitatively described by the following mathematical formula:

[0082]

[0083] in, For the adjusted number The location of each sampling point To output the center coordinates of the feature points, For the first The spatial offset of each sampling point, including both horizontal and vertical offsets. For the first The time dimension offset of each sampling point corresponds to the time step of the adjacent frame.

[0084] After adjusting the geometric sampling positions of the feature points in the current frame based on the 3D offset field, a smoothness constraint is applied to the adjusted geometric sampling positions. The initial optical flow field of the smoke region in the previous frame is extracted. The initial optical flow field is calculated using the Lucas-Kanade optical flow algorithm. The initial optical flow field describes the motion vectors of the pixels in the smoke region between the previous frame and the current frame. Each pixel corresponds to an optical flow vector in the horizontal and vertical directions.

[0085] The optical flow deviation matrix is ​​calculated between the spatial offset in the 3D offset field and the optical flow vector of the initial optical flow field. Each element in the optical flow deviation matrix is ​​the Euclidean distance between the spatial offset and the optical flow vector at the corresponding location. This optical flow deviation matrix is ​​introduced as a regularization penalty term into the loss function of the offset prediction convolutional layer. The loss function includes a cross-entropy loss term and a regularization penalty term, with the weights of the regularization penalty term being preset fixed coefficients. This regularization penalty term constrains the variation amplitude of the 3D offset field between spatially adjacent sampling points, keeping the offsets of adjacent sampling points smooth. It filters out anomalous offset jumps caused by random turbulence interference from minute smoke particles, avoiding feature aggregation errors caused by drastic offset changes.

[0086] The process of constructing the above loss function is quantitatively described by the following mathematical formula:

[0087]

[0088] in, For the total loss function, For cross-entropy loss term, These are the regularization weight coefficients. and These represent the height and width of the feature map, respectively. These are the spatial offset components in the three-dimensional offset field. Let be the optical flow vector of the initial optical flow field. This refers to L2 norm operations.

[0089] In this embodiment, the parameter definitions and constraint rules of the three-dimensional offset field are shown in the following table:

[0090] Table 2. Parameter Definitions and Constraint Rules for the 3D Migration Field

[0091] Offset Dimension Parameter symbol Physical meaning Range of values Regularization constraint rules Horizontal offset Δx Offset distance of sampling points in the horizontal axis of the feature map [-5,5] pixels The deviation from the horizontal component of the initial optical flow field does not exceed 2 pixel units, and the difference in offset between adjacent sampling points does not exceed 1 pixel unit. Vertical offset Δy Offset distance of sampling points along the vertical axis of the feature map [-5,5] pixels The deviation from the vertical component of the initial optical flow field does not exceed 2 pixel units, and the offset difference between adjacent sampling points does not exceed 1 pixel unit. Time dimension offset Δt The offset step of the sampling points in the time dimension [-4,0] time step Offsets are only allowed in the direction of historical frames, and the difference in time offset step size between adjacent sampling points shall not exceed one time step.

[0092] The table above clarifies the physical meaning, value range, and regularization constraint rules of the parameters in each dimension of the three-dimensional offset field. By constraining the values ​​and smoothness of the offsets in each dimension, it ensures that the offset of the sampling points can match the real movement trajectory of the smoke, while avoiding abnormal offsets caused by random turbulence, thus improving the accuracy of feature aggregation.

[0093] Please refer to the attached document. Figure 4 Based on the sampling point position of the current frame, multi-frame sampling feature vectors are extracted from the feature maps of corresponding scales in adjacent frames. These multi-frame sampling feature vectors include the sampling feature vector of the current frame and the sampling feature vectors of multiple consecutive historical frames. These multi-frame sampling feature vectors are then input into a temporal attention layer. The temporal attention layer uses the sampling feature vector of the current frame as the query vector and the sampling feature vector of each historical frame as the key and value vectors. A dot product attention mechanism is used to calculate the relevance score between the key vector and the query vector of each historical frame. Softmax normalization is applied to the relevance scores of all historical frames to generate initial relevance weights.

[0094] Extract the area of ​​the smoke bounding box corresponding to each of the adjacent frames, and calculate the absolute value of the rate of change of the smoke bounding box area of ​​each of the adjacent frames relative to the smoke bounding box area of ​​the current frame. The calculation process of the absolute value of the rate of change is quantitatively described by the following mathematical formula:

[0095]

[0096] in, For the first The absolute value of the area change rate of each historical frame. For the first The area of ​​the smoke bounding box in each historical frame. This represents the area of ​​the smoke bounding box in the current frame.

[0097] The absolute value of the rate of change is multiplied into the initial relevance score as a decay factor to generate the dynamically adjusted final attention weight. The calculation process of the final attention weight is quantitatively described by the following mathematical formula:

[0098]

[0099] in, For the first The final attention weights for each historical frame, For the first The initial relevance score of each historical frame, This represents the number of historical frames.

[0100] The sampled feature vectors of each frame in multiple adjacent frames are weighted and summed using the final attention weight to generate an aggregated feature vector. By attenuating the absolute value of the area change rate, the contribution of historical frames where the smoke area shrinks in multiple adjacent frames to the aggregated features of the current frame is reduced. Real fire smoke shows a continuous expansion trend, and frames with shrinking area are likely to be interfering targets or misidentified areas. Through this attenuation mechanism, interfering features can be further suppressed, and the effectiveness of aggregated features can be improved.

[0101] In this embodiment, dimensionality reduction is performed using depthwise separable convolution to preserve the spatial distribution details of the micro-smoke. A spatial attention-guided network enhances and suppresses shallow features, improving the saliency of micro-smoke edge features. Prediction and adjustment of the 3D offset field adapt the sampling position to the irregular motion trajectory of the smoke. Optical flow regularization constraints filter out anomalous shifts in offset, ensuring the smoothness of the sampling position. Dynamic adjustment of the temporal attention mechanism and area decay factor suppresses the feature contributions of static background features and interfering frames, improving the effectiveness of micro-smoke motion feature aggregation.

[0102] In another optional embodiment, the aggregated features from each scale are input into a temporal consistency verification module. The feature decoding submodule of this module decodes the aggregated minute smoke motion features, extracting the smoke bounding box coordinates and the average grayscale value of the smoke region in the current frame. The feature decoding submodule includes a bounding box regression branch and a grayscale value calculation branch. The bounding box regression branch performs a convolution operation on the aggregated feature map, outputting the smoke bounding box coordinate offset corresponding to each feature point. Combined with preset anchor box coordinates, the final smoke bounding box coordinates are calculated. The smoke bounding box coordinates include the upper left x-coordinate. Top left ordinate The horizontal coordinate of the lower right corner bottom right ordinate Calculate the area of ​​the smoke region in the current frame based on the coordinates of the smoke bounding box. The calculation process is quantitatively described using the following mathematical formula:

[0103]

[0104] in, > , > , The value of is a positive real number.

[0105] The grayscale calculation branch maps the smoke bounding box coordinates to the original image data of the current frame, extracts the grayscale values ​​of all pixels within the region that matches the smoke bounding box coordinates in the original image data, and calculates the arithmetic mean of the grayscale values ​​of all pixels within that region to obtain the average grayscale value of the smoke region. Calculate the area change of the smoke region and the grayscale value difference over multiple consecutive time steps, and the area change... For the current time step Area of ​​the smoke zone Compared with the previous time step Area of ​​the smoke zone The difference, grayscale value difference For the current time step Average gray value of the smoke area Compared with the previous time step Average gray value of the smoke area The difference is calculated by concatenating the area change and grayscale difference values ​​from multiple consecutive time steps in chronological order to generate a temporal state vector. Temporal state vector The dimensions match the number of time steps.

[0106] Please refer to the attached document. Figure 5 , the time-series state vector The input is fed into the Long Short-Term Memory (LSTM) network at time steps. Each LSTM unit in the LSM network corresponds to one time step of input. A background suppression bias term is introduced into the forgetting gating calculation of the LSM network. The initial background frame is extracted from the fire monitoring area to be detected. The initial background frame is a static image frame of the monitoring area without smoke or dynamic interference targets. The initial background frame is obtained through a background modeling algorithm. The background modeling algorithm uses a Gaussian mixture model to model multiple consecutive interference-free images in the initial stage of monitoring area startup, generating the initial background frame. The static background difference feature map between the current frame and the initial background frame is calculated. The static background difference feature map is a matrix composed of the absolute values ​​of the differences in pixel grayscale values ​​between the current frame image data and the initial background frame image data. In the static background difference feature map, areas with values ​​close to 0 are static background areas, while areas with larger values ​​are dynamically changing areas.

[0107] The static background difference feature map is linearly transformed through a fully connected layer and mapped to a background suppression bias term that matches the forgetting gate input dimension. Background suppression bias term The dimension of the input is consistent with the output dimension of the forgetting gate. In the calculation of the forgetting gate, the background suppression bias term is... When linear transformations are incorporated into the computation, the forgetting gating process is quantitatively described by the following mathematical formula:

[0108]

[0109] in, For the current time step Forgetting gate output weights, It is the sigmoid activation function. Here is the weight matrix for forgetting gating. For the previous time step The hidden state, This is the concatenation operation between the hidden state and the temporal state vector. The initial bias term for the forgetting gate, This is a background suppression bias term.

[0110] When the sequence state vector When the background suppression bias term contains interfering features that match the static background difference feature map, i.e., when the dynamic change region of the current frame is a pixel perturbation of the static background region, the background suppression bias term... When the value is negative, the output weight of the forgetting gate is reduced by the background suppression bias term. This disables the forgetting gate, preventing the transmission of the interfering feature to the cell state in subsequent time steps, thus avoiding the accumulation of interfering features in the cell state from the static background and improving the accuracy of temporal dependency extraction.

[0111] The computational processes of input gating, cell state updates, and output gating in Long Short-Term Memory (LSTM) networks are quantitatively described using the following mathematical formulas:

[0112]

[0113]

[0114]

[0115]

[0116]

[0117] in, Input gating output weights, For the input gate weight matrix, For input gating bias terms; This represents the candidate cell state at the current time step. This is the candidate cell state weight matrix. This refers to the candidate cell state bias term; This represents the cell state at the current time step. This represents the cell state at the previous time step; To output gating output weights, To output the gated weight matrix, This is the output gated bias term; The hidden state at the current time step.

[0118] Through the aforementioned cell state update mechanism, the Long Short-Term Memory Network can retain the area increase feature and gray value decay feature across multiple time steps, extracting the long-term temporal dependency relationship between smoke area expansion and gray value change. The area of ​​real fire smoke continuously expands over time, and the average gray value continuously decays over time. This long-term temporal feature is the core basis for distinguishing real smoke from static and dynamic interference targets.

[0119] In this embodiment, the inputs, outputs, and functions of each gate in the Long Short-Term Memory network are defined as shown in the table below:

[0120] Table 3. Input, Output, and Function Definitions of Each Gating Gate in the Long Short-Term Memory Network

[0121] Gating name Input signal Core operational logic Output signal Core Function Definition Forgotten Gate Previous time step hidden state Current time step timing state vector Background suppression bias term Initial bias term The sigmoid linear transformation outputs gate weights between 0 and 1. Forgetting Gating Weights By controlling the retention and discarding of information from the cell state in the previous time step, the transmission of static background interference features is blocked through a background suppression bias term. Input gate Previous time step hidden state Current time step timing state vector Bias terms The sigmoid linear transformation outputs gate weights between 0 and 1. Input gating weights Control the content in the current time step input information that needs to be updated to the cell state. Cell state update Forgetting Gating Weights Input gating weights Previous time step cell state Candidate cell status The forgetting weight is multiplied by the preceding cell state, and the input weight is multiplied by the candidate state, then added element by element. Current time step cell state It stores long-term temporal characteristics spanning multiple time steps, preserving the core features of gradual increase in smoke area and grayscale decay. Output gate Previous time step hidden state Current time step timing state vector Bias terms Current time step cell state The sigmoid linear transformation generates the gate weights, and... Multiplication of activated cell states Current time step hidden state The information that needs to be output from the cell state at the current time step is controlled for subsequent calculation of smoke confidence.

[0122] The table above clarifies the input signals, operational logic, output signals, and core functions of each gating component of the Long Short-Term Memory (LSTM) network. It clearly defines the role of each gating component in the long-term temporal dependency extraction process, especially the introduction method and function of the background suppression bias term in the forgetting gate, ensuring that those skilled in the art can fully reproduce the temporal feature extraction process of the LSM network.

[0123] The final hidden state of the Long Short-Term Memory (LSTM) network is input into a fully connected layer. The fully connected layer performs a linear transformation on the final hidden state, processes it through a sigmoid activation function, and outputs the smoke confidence score at the current time step. Smoke confidence The value range is 0 to 1. Obtain the smoke confidence score sequence for multiple consecutive time steps within the current time window, arrange the smoke confidence score sequence in chronological order, and construct a confidence score-time curve. Perform a first-order difference operation on the confidence score-time curve to extract the confidence score difference value between adjacent time steps. The calculation process of the confidence score difference value is quantitatively described by the following mathematical formula:

[0124]

[0125] in, This refers to the time step number within the time window. For the first The confidence difference at each time step.

[0126] Please refer to the attached document. Figure 6 The duration for which the statistical confidence difference remains in a continuously positive range, i.e., across multiple consecutive time steps. The number of time steps that are all greater than 0. When the duration exceeds a preset duration threshold, and the smoke confidence level of the current time step... When the preset alarm threshold is exceeded, the timing gradient is determined to show a positive increasing trend, triggering the generation of a fire alarm signal. The fire alarm signal includes the monitored area number, alarm time, smoke confidence data, and corresponding video frame data, which is transmitted to the fire monitoring center platform via wired or wireless communication network, simultaneously triggering the audible and visual alarm devices at the monitored site.

[0127] In this embodiment, the coordinates of the smoke bounding box and the average gray value are extracted by feature decoding to construct a temporal state vector representing the temporal changes of smoke. By introducing a background suppression bias term into the forget gate of the Long Short-Term Memory Network, the transmission of static background interference features is blocked, avoiding the accumulation of interference features in the cell state. Through the cell state update mechanism of the Long Short-Term Memory Network, the long-term temporal dependency relationship between smoke area expansion and gray level gradient is effectively extracted. By judging the threshold of smoke confidence and judging the positive growth trend of temporal gradient, accurate identification of real smoke targets is achieved, reducing the false alarm rate and the false negative rate.

Claims

1. A method for intelligent identification and alarm of smoke and fire in fire protection video, characterized in that, include: A continuous video frame sequence of the fire monitoring area to be detected is obtained, and the continuous video frame sequence is input into a preset feature pyramid network to extract the multi-scale basic feature map corresponding to the continuous video frame sequence. A deformable temporal convolution module is applied to the multi-scale basic feature map at each scale to predict the sampling point position of the current frame based on the offset of the smoke region in the previous frame in the continuous video frame sequence, and to aggregate the small smoke motion features of multiple adjacent frames based on the sampling point position of the current frame. The aggregated features from each scale are input into the temporal consistency verification module, and the long short-term memory network is used to extract the long-term temporal dependency between smoke area expansion and grayscale gradation. Smoke confidence is output based on the long-term temporal dependency relationship. When the smoke confidence exceeds a preset alarm threshold and the temporal gradient of the smoke confidence shows a positive increasing trend, a fire alarm signal is generated.

2. The fire video smoke and fire intelligent recognition and alarm method according to claim 1, characterized in that, The step of inputting the continuous video frame sequence into a preset feature pyramid network to extract the multi-scale basic feature map corresponding to the continuous video frame sequence includes: inputting the current frame in the continuous video frame sequence into multiple convolutional layers from bottom to top to extract basic feature layers with different spatial resolutions. Upsampling and dimensionality reduction are performed on the basic feature layers of adjacent levels. The upsampled basic feature layers are then added and fused element by element with the adjacent shallow basic feature layers to generate the multi-scale basic feature map. In the process of dimensionality reduction of the basic feature layer, depthwise separable convolution is used instead of standard convolution to preserve the spatial distribution details of tiny smoke particles at different scales.

3. The fire video smoke and fire intelligent recognition and alarm method according to claim 1, characterized in that, The step of predicting the sampling point position of the current frame based on the offset of the smoke region in the previous frame in the continuous video frame sequence includes: concatenating the feature map of the smoke region at the scale corresponding to the previous frame with the feature map of the current frame at the scale corresponding to the current frame, and inputting it into the offset prediction convolutional layer. The offset prediction convolutional layer outputs a three-dimensional offset field centered on the coordinates of each feature point in the current frame. The three-dimensional offset field includes horizontal offset, vertical offset, and time dimension offset. The geometric sampling position of the feature points in the current frame is adjusted according to the three-dimensional offset field, so that the sampling position is adaptively offset along the motion trajectory of the tiny smoke particles driven by airflow.

4. The fire video smoke and fire intelligent recognition and alarm method according to claim 1, characterized in that, The step of aggregating the micro smoke motion features of adjacent multiple frames based on the sampling point position of the current frame includes: extracting multi-frame sampling feature vectors from the feature maps of the corresponding scales of the adjacent multiple frames based on the sampling point position of the current frame. The multi-frame sampled feature vectors are input into the temporal attention calculation layer to calculate the relevance score of each frame sampled feature vector in the adjacent multi-frames relative to the current frame sampled feature vector. The correlation score is used to perform a weighted summation of the sampled feature vectors of each of the adjacent frames, suppressing background static features in the adjacent frames that are not related to the smoke motion of the current frame, and outputting the aggregated micro smoke motion features.

5. The fire protection video smoke and fire intelligent recognition and alarm method according to claim 1, characterized in that, The step of extracting the long-term temporal dependency relationship between smoke area expansion and grayscale gradient using a long short-term memory network includes: performing feature decoding on the aggregated micro smoke motion features, and extracting the smoke bounding box coordinates and the average grayscale value of the smoke region in the current frame. The difference sequence between the area change of the smoke bounding box coordinates and the average gray value of the smoke region is concatenated to form a temporal state vector; The temporal state vector is input into the long short-term memory network in time steps. Through the cell state update mechanism of the long short-term memory network, the area increasing feature and grayscale decay feature across multiple time steps are preserved.

6. The fire video smoke and fire intelligent recognition and alarm method according to claim 1, characterized in that, The step of generating a fire alarm signal when the smoke confidence exceeds a preset alarm threshold and the temporal gradient of the smoke confidence shows a positive increasing trend includes: constructing the smoke confidence sequence within the current time window into a confidence time curve; Perform a first-order difference operation on the confidence time curve to extract the confidence difference value between adjacent time steps; When the duration for which the confidence difference value remains in a positive range exceeds a preset duration threshold, and the current smoke confidence exceeds the preset alarm threshold, it is determined that the time-series gradient shows a positive increasing trend, triggering the generation of the fire alarm signal.

7. The fire protection video smoke and fire intelligent recognition and alarm method according to claim 2, characterized in that, Before fusing the upsampled basic feature layer with the adjacent shallow basic feature layer element by element, the method further includes: inputting the upsampled basic feature layer and the adjacent shallow basic feature layer into a spatial attention guidance network; The spatial attention guidance network performs max pooling and average pooling in the channel dimension, and after concatenation, generates a spatial weight matrix. The spatial weight matrix is ​​used to enhance the tiny smoke edge pixels in the adjacent shallow basic feature layers with high weights and suppress the background pixels with low weights. The enhanced adjacent shallow basic feature layers are then fused with the upsampled basic feature layers element by element.

8. The fire protection video smoke and fire intelligent recognition and alarm method according to claim 3, characterized in that, After adjusting the geometric sampling position of the current frame feature points according to the three-dimensional offset field, the method further includes: applying a smoothness constraint to the adjusted geometric sampling position; Extract the initial optical flow field of the smoke region in the previous frame, and calculate the optical flow deviation matrix between the three-dimensional offset field and the initial optical flow field; The optical flow bias matrix is ​​introduced as a regularization penalty term into the loss function of the offset prediction convolutional layer.

9. The fire video smoke and fire intelligent recognition and alarm method according to claim 4, characterized in that, In the process of using the correlation score to perform a weighted summation of the sampled feature vectors of each of the adjacent multi-frames, the method further includes: extracting the area of ​​the smoke bounding box corresponding to each of the adjacent multi-frames. Calculate the absolute value of the rate of change of the smoke bounding box area of ​​each of the adjacent frames relative to the smoke bounding box area of ​​the current frame; The absolute value of the rate of change is multiplied into the relevance score as a decay factor to generate the dynamically adjusted final attention weight. The final attention weight is used to reduce the contribution of historical frames in which the smoke area shrinks in adjacent frames to the aggregated features of the current frame.

10. The fire protection video smoke and fire intelligent recognition and alarm method according to claim 5, characterized in that, In the process of preserving the area increase feature and grayscale decay feature across multiple time steps through the cell state update mechanism of the Long Short-Term Memory Network, the method also includes: introducing a background suppression bias term in the forgetting gating calculation of the Long Short-Term Memory Network. Extract the initial background frame of the fire monitoring area to be detected, and calculate the static background difference feature map between the current frame and the initial background frame; The static background difference feature map is mapped to the background suppression bias term.

Images