Fire detection method based on space-time double-flow network
Through the space-time dual-stream network combining feature extraction and adaptive feature fusion modules of spatial and temporal flow, the problem of insufficient feature recognition of flame motion state in the prior art is solved, the accuracy and robustness of fire detection are improved, and the Detection effect.
Patent Information
- Application Number
- CN202410948895.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-16
- Publication Date
- 2025-01-03
AI Technical Summary
The existing deep learning-based fire detection methods mainly learn the appearance characteristics of static images, ignore the appearance characteristics in the flame motion state, and cannot effectively identify the flame characteristics in different motion states, resulting in poor detection results.
The space-time dual-stream network is adopted to collect fire video data, and the space-time dual-stream network model is constructed. The pre-trained stream estimation network, spatial stream and temporal stream are used to extract the motion and appearance characteristics of the flame, and the space-time features are integrated through the adaptive feature fusion module to perform fire. Test.
It improves the accuracy and robustness of fire detection, reduces the calculation cost, effectively recognizes the flame characteristics under different motion states, reduces the possibility of false alarms and misjudgments, and shows high competitiveness.
Smart Images
Figure CN119251808A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of fire detection, and in particular to a fire detection method based on a spatiotemporal dual-stream network. Background Art
[0002] Fire is a common natural disaster that seriously threatens human life and property. Fires are affected by factors such as the type of combustibles, terrain and weather, and can spread rapidly to neighboring areas and are difficult to control. At the same time, unlike other natural disasters, fires are highly sporadic and there are many reasons for fires, most of which are related to human factors. The earlier a fire is discovered, the less damage it causes. Therefore, it is very necessary to detect and warn in the early stages of a fire to reduce or avoid casualties and property losses.
[0003] With the development of computer vision technology and deep learning, the application of fire detection using surveillance cameras has become increasingly widespread. Vision-based fire detection methods can cope with different scenarios and have strong robustness and high accuracy. In recent years, deep learning methods have made major breakthroughs in the field of target detection. Fire detection methods using deep learning have higher accuracy, stronger adaptability and robustness.
[0004] Most of the existing fire detection methods based on deep learning only learn the appearance features of static images, but ignore the appearance features of the flame in motion, and cannot meet the requirements of identifying the appearance features of flames in different motion states; therefore, they do not meet the existing needs. In this regard, we propose a fire detection method based on a spatiotemporal dual-stream network. Summary of the invention
[0005] The purpose of the present invention is to provide a fire detection method based on a spatiotemporal dual-stream network to solve the problem that the existing fire detection methods based on deep learning proposed in the above background technology mostly only learn the appearance features of static images, but ignore the appearance features of flames in motion, and cannot meet the needs of identifying the appearance features of flames in different motion states.
[0006] To achieve the above object, the present invention provides the following technical solution: a fire detection method based on a spatiotemporal dual-stream network, comprising the following steps:
[0007] (A): Collect fire videos, label them using the LabelImg tool, and build a fire detection video dataset;
[0008] (B): Based on the technology of spatiotemporal dual-stream network, a spatiotemporal dual-stream network model is constructed for fire detection;
[0009] (C): Use the constructed fire detection video dataset to train the spatiotemporal two-stream network model and obtain the optimal weights;
[0010] (D): The fire video to be detected is input into the trained spatiotemporal dual-stream network model to obtain the fire detection result.
[0011] Preferably, the fire detection video dataset in (A) includes dataset 1 and dataset 2, wherein dataset 1 is used to train the spatiotemporal dual-stream network model, and dataset 1 is divided into a training set and a validation set, wherein the training set includes 71 video scenes, each of which has 50 to 100 continuous video frames, and the 50 to 100 continuous video frames have a total of 5139 images, and the validation set includes 16 video scenes, each of which has 50 video frames, and the 50 video frames have a total of 800 images;
[0012] Dataset 2 is the Furg Fire Dataset, in which 8 representative fire videos are selected, and in Dataset 2, 4 fire videos containing interference instances are added.
[0013] Preferably, the spatiotemporal dual-stream network model consists of a pre-trained flow estimation network, a spatial stream, a temporal stream and a detection layer. The flow estimation network uses LiteFlowNet3 to extract the optical flow images of the current video frame and the previous frame as the motion information of the flame. The spatial stream uses the RGB image of the current video frame as input to obtain the appearance and texture features of the fire. The temporal stream uses the optical flow image as input to obtain the motion features of the fire. Both the spatial stream and the temporal stream use ShuffleNetV2 as feature extraction networks. The feature extraction network prunes the network of the temporal stream and fuses it into spatiotemporal features that are input into the detection layer based on YOLOv5s to detect fire.
[0014] Preferably, the spatiotemporal features are channel-weighted by an additional adaptive feature fusion module.
[0015] Preferably, the spatiotemporal dual-stream network model is trained through a multi-task processing process and the LiteFlowNet3 optical flow network is initialized using pre-trained weights. After the LiteFlowNet3 optical flow network is initialized, it is frozen during training, and then the spatial stream and temporal stream are trained separately. Finally, the spatiotemporal dual-stream network is fine-tuned. Such a training strategy helps to make full use of the characteristics of the network and improve fire detection performance.
[0016] Preferably, the ShuffleNetV2 network consists of two modules: a basic unit and a spatial downsampling unit. The basic unit does not change the output channel and spatial size, and the spatial downsampling unit reduces the spatial size of the output feature map by half and doubles the number of channels.
[0017] Preferably, adjacent pixels of the optical flow image should be smooth or continuous, the optical flow image does not contain as many high-level semantics and complex textures as RGB images, the temporal flow prunes the number of basic units of Stage 3 in the deep layer of the network from 7 to 3 and prunes the number of basic units of Stage 4 from 3 to 1, and the spatial downsampling unit remains unchanged.
[0018] Preferably, the detection layer consists of two parts, Neck and Head, the Neck fuses multi-scale features, the Head detects flame targets, and the detection layer extracts features from network layers with resolutions of 1 / 8, 1 / 16 and 1 / 32 of the original size and inputs the features into the detection layer for multi-scale detection.
[0019] Preferably, the Neck includes a CBS module and a C3 module, the CBS module is a convolutional layer, the convolutional layer includes a 3×3 convolution, batch normalization and SiLU activation function, the C3 module is the main unit for feature extraction in the network, and the C3 module includes 3 convolutional layers and multiple BottleNecks.
[0020] Preferably, the adaptive feature fusion module comprises the following steps:
[0021] S1: Preliminary fusion of temporal and spatial features using pixel-level addition
[0022]
[0023] S∈R C×H×W Represents spatial features, T∈R C×H×W Represents time characteristics, represents pixel-level addition, and F is the initial fusion feature;
[0024] S2: Aggregate the information of each channel using global average pooling:
[0025]
[0026] Among them, M g ∈R C×1×1 is the aggregated channel information;
[0027] S3: Use a dimension reduction layer and a dimension increase layer to capture cross-channel information interactions:
[0028] M c =β(W2δ(β(W1(M g ))))
[0029] in, is a dimension reduction layer, is a dimension-increasing layer, r is the channel reduction rate, δ represents the ReLU activation function, and β represents batch normalization;
[0030] S4: Use the Sigmoid activation function to obtain a weight between 0 and 1:
[0031] M=Sigmoid(M c );
[0032] S5: Assign weights to temporal and spatial features:
[0033]
[0034] in, represents element-wise multiplication, Z∈R C×H×W Represents the features after adaptive fusion.
[0035] Compared with the prior art, the present invention has the following beneficial effects:
[0036] 1. The present invention constructs a spatiotemporal dual-stream network. The spatial stream extracts the appearance features of the current frame, and the temporal stream extracts the motion features between frames. The spatiotemporal features are used for fire detection to improve the accuracy of fire detection.
[0037] 2. The present invention uses lightweight ShuffleNetV2 as the spatiotemporal feature extraction network to reduce the computational cost of two parallel branches, and at the same time prunes the depth of the time stream to prevent overfitting;
[0038] 3. In order to effectively integrate spatiotemporal features, the present invention uses an adaptive feature fusion module to adaptively adjust the weights of spatiotemporal flows through the network to further improve the accuracy of fire detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 is a flow chart of the fire detection method of the present invention;
[0040] Figure 2 It is a structural diagram of the spatiotemporal dual-stream network of the present invention;
[0041] Figure 3 is a network structure diagram of the detection layer of the present invention;
[0042] Figure 4 This is a network structure diagram of the adaptive feature fusion module of the present invention. DETAILED DESCRIPTION
[0043] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0044] See also Figure 1 , an embodiment of the present invention provides: a fire detection method based on a spatiotemporal dual-stream network, comprising the following steps:
[0045] (A): Collect fire videos, label them using the LabelImg tool, and build a fire detection video dataset;
[0046] The fire detection video dataset includes dataset 1 and dataset 2. Dataset 1 is used to train the spatiotemporal dual-stream network model. Dataset 1 is divided into a training set and a validation set. The training set contains 71 video scenes. Each video scene in the training set has 50 to 100 continuous video frames, and 50 to 100 continuous video frames total 5139 images. The validation set contains 16 video scenes. Each video scene in the validation set has 50 video frames, and 50 video frames total 800 images.
[0047] Dataset 2 is the Furg Fire Dataset. Eight representative fire videos are selected from the Furg Fire Dataset. Four fire videos containing interference instances are added to Dataset 2. The LabelImg tool is used to label the flames in the dataset, and the annotation information is saved as a txt file.
[0048] (B): Based on the technology of spatiotemporal dual-stream network, a spatiotemporal dual-stream network model is constructed for fire detection;
[0049] See also Figures 2 to 4 The spatiotemporal dual-stream network model consists of a pre-trained flow estimation network, a spatial stream, a temporal stream, and a detection layer. The flow estimation network uses LiteFlowNet3 to extract the optical flow images of the current video frame and the previous frame as the motion information of the flame. The spatial stream uses the RGB image of the current video frame as input to obtain the appearance and texture features of the fire. The temporal stream uses the optical flow image as input to obtain the motion features of the fire. Both the spatial stream and the temporal stream use ShuffleNetV2 as feature extraction networks. The feature extraction network prunes the network of the temporal stream and fuses it into spatiotemporal features. The detection layer based on YOLOv5s is used to detect fire.
[0050] The ShuffleNetV2 network consists of two modules: the basic unit and the spatial downsampling unit. The basic unit does not change the output channel and spatial size. The spatial downsampling unit reduces the spatial size of the output feature map by half and doubles the number of channels to increase the network's receptive field.
[0051] Adjacent pixels of optical flow images should be smooth or continuous. Optical flow images do not contain as many high-level semantics and complex textures as RGB images. The temporal stream prunes the number of basic units of Stage 3 in the deep layer of the network from 7 to 3 and the number of basic units of Stage 4 from 3 to 1. The spatial downsampling unit remains unchanged to maintain the same feature resolution and dimension as the spatial stream in the output of the same layer;
[0052] The detection layer consists of two parts: Neck and Head. Neck fuses multi-scale features. Head detects flame targets. The detection layer extracts features from network layers with resolutions of 1 / 8, 1 / 16, and 1 / 32 of the original size and inputs the features into the detection layer for multi-scale detection. Multi-layer features can enhance the model's contextual perception of targets or scenes. Neck includes CBS module and C3 module. CBS module is a convolutional layer. The convolutional layer includes a 3×3 convolution, batch normalization, and SiLU activation function. C3 module is the main unit for feature extraction in the network. C3 module contains 3 convolutional layers and multiple BottleNecks, which are used to increase the depth and receptive field of the network and improve the ability of feature extraction.
[0053] The spatiotemporal features are channel-weighted through an additional adaptive feature fusion module to more effectively fuse the spatiotemporal information.
[0054] The adaptive feature fusion module includes the following steps:
[0055] S1: Preliminary fusion of temporal and spatial features using pixel-level addition
[0056]
[0057] S∈R C×H×W Represents spatial features, T∈R C×H×W Represents time characteristics, represents pixel-level addition, and F is the initial fusion feature;
[0058] S2: Aggregate the information of each channel using global average pooling:
[0059]
[0060] Among them, M g ∈R C×1×1 is the aggregated channel information;
[0061] S3: Use a dimension reduction layer and a dimension increase layer to capture cross-channel information interactions:
[0062] M c =β(W2δ(β(W1(M g ))))
[0063] in, is a dimension reduction layer, is a dimension-increasing layer, r is the channel reduction rate, δ represents the ReLU activation function, and β represents batch normalization;
[0064] S4: Use the Sigmoid activation function to obtain a weight between 0 and 1:
[0065] M=Sigmoid(M c );
[0066] S5: Assign weights to temporal and spatial features:
[0067]
[0068] in, represents element-wise multiplication, Z∈R C×H×W Represents the features after adaptive fusion.
[0069] (C): Use the constructed fire detection video dataset to train the spatiotemporal two-stream network model and obtain the optimal weights;
[0070] The spatiotemporal dual-stream network model is trained through a multi-task processing process and uses pre-trained weights to initialize the LiteFlowNet3 optical flow network. After the LiteFlowNet3 optical flow network is initialized, it is frozen during training, and then the spatial stream and temporal stream are trained separately. Finally, the spatiotemporal dual-stream network is fine-tuned. This training strategy helps to make full use of the characteristics of the network and improve fire detection performance.
[0071] (D): Input the fire video to be detected into the trained spatiotemporal dual-stream network model to obtain the fire detection result;
[0072] Five indicators are used to evaluate the fire detection results: AP50, AP50-95, precision (P), recall (R), and F1 score. The calculation formulas for precision, recall, average precision (AP), and F1 score are as follows:
[0073]
[0074]
[0075]
[0076]
[0077] Among them, TP represents true positive, FP represents false positive, and FN represents false negative.
[0078] Two experiments were conducted. The first one was an ablation experiment to verify the performance of the two-stream network, the depth of the pruned temporal stream, and the adaptive feature fusion module. The second one was a comparative experiment to further evaluate the performance of the method and compare the spatiotemporal two-stream network method with different deep learning fire detection methods.
[0079] Experiment 1:
[0080] In Experiment 1, the spatiotemporal two-stream network was trained and evaluated on Dataset 1, using AP50 and AP50-95 as evaluation indicators. AP50 represents the average accuracy when the intersection-over-union (IoU) threshold is greater than 0.5, and AP50-95 represents the average accuracy when the IoU threshold ranges from 0.5 to 0.95.
[0081] The empty two-stream network improves the detection performance of AP50, but AP50-95 decreases slightly. This may be because the optical flow image does not contain complex texture features. Too deep features may cause redundancy and affect the final detection results. Therefore, the network depth in the temporal stream is pruned to increase AP50-95 from 0.53 to 0.579. In order to effectively fuse spatial and temporal features, an adaptive feature fusion module is used to further improve the performance of the method. Experimental results show that the improved spatiotemporal two-stream network achieves the best detection effect.
[0082] Experiment 2:
[0083] To further evaluate the performance of the method, the spatiotemporal two-stream network method was compared with different deep learning fire detection methods, including YOLOv5s, YOLOv4, YOLOv4-tiny, and the method of Zhao et al.
[0084] The AP50 of the spatiotemporal two-stream network method reaches 0.932, which is 0.049 higher than the second-best method YOLOv4. Although the accuracy of the spatiotemporal two-stream network method is not the highest, its accuracy is very close to the optimal method YOLOv5, while its recall rate is only 0.682. In terms of recall rate, the spatiotemporal two-stream network method is significantly better than other methods. The spatiotemporal two-stream network method has good performance in both accuracy and recall rate, indicating that it can effectively distinguish between fire and non-fire objects and reduce the possibility of false alarms and misjudgments. The spatiotemporal two-stream network method also obtained the highest F1 score (0.935). Therefore, compared with other deep learning methods, the method in this paper is more competitive in fire detection.
[0085] In summary, compared with other deep learning fire detection methods, the spatiotemporal two-stream network method is more competitive in fire detection. It has better performance in detecting fire and fire-like objects and is a promising fire detection method.
[0086] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.
Claims
1. A fire detection method based on a spatiotemporal dual-stream network, characterized in that: The following steps are involved: (A): Collect fire videos, label them using the LabelImg tool, and build a fire detection video dataset; (B): Based on the technology of spatiotemporal dual-stream network, a spatiotemporal dual-stream network model is constructed for fire detection; (C): Use the constructed fire detection video dataset to train the spatiotemporal two-stream network model and obtain the optimal weights; (D): Input the fire video to be detected into the trained spatiotemporal dual-stream network model to obtain the fire detection result.
2. The fire detection method based on spatiotemporal dual-stream network according to claim 1 is characterized in that: The fire detection video dataset in (A) includes dataset 1 and dataset 2, wherein dataset 1 is used to train a spatiotemporal dual-stream network model, and dataset 1 is divided into a training set and a validation set, wherein the training set includes 71 video scenes, each of which has 50 to 100 continuous video frames, and the 50 to 100 continuous video frames have a total of 5139 images, and the validation set includes 16 video scenes, each of which has 50 video frames, and the 50 video frames have a total of 800 images; Dataset 2 is the Furg Fire Dataset, in which 8 representative fire videos are selected, and in Dataset 2, 4 fire videos containing interference instances are added.
3. The fire detection method based on spatiotemporal dual-stream network according to claim 2 is characterized in that: The spatiotemporal dual-stream network model consists of a pre-trained flow estimation network, a spatial stream, a temporal stream and a detection layer. The flow estimation network uses LiteFlowNet3 to extract the optical flow images of the current video frame and the previous frame as the motion information of the flame. The spatial stream uses the RGB image of the current video frame as input to obtain the appearance and texture features of the fire. The temporal stream uses the optical flow image as input to obtain the motion features of the fire. Both the spatial stream and the temporal stream use ShuffleNetV2 as feature extraction networks. The feature extraction network prunes the network of the temporal stream and fuses it into spatiotemporal features, which are input into the detection layer based on YOLOv5s to detect fire.
4. The fire detection method based on spatiotemporal dual-stream network according to claim 3 is characterized in that: The spatiotemporal features are channel-weighted through an additional adaptive feature fusion module.
5. The fire detection method based on spatiotemporal dual-stream network according to claim 3 is characterized in that: The spatiotemporal dual-stream network model is trained through a multi-task processing process and uses pre-trained weights to initialize the LiteFlowNet3 optical flow network. After the LiteFlowNet3 optical flow network is initialized, it is frozen during training, and then the spatial stream and the temporal stream are trained separately. Finally, the spatiotemporal dual-stream network is fine-tuned. Such a training strategy helps to make full use of the characteristics of the network and improve fire detection performance.
6. The fire detection method based on spatiotemporal dual-stream network according to claim 3 is characterized in that: The ShuffleNetV2 network consists of two modules: a basic unit and a spatial downsampling unit. The basic unit does not change the output channel and spatial size, and the spatial downsampling unit reduces the spatial size of the output feature map by half and doubles the number of channels.
7. The fire detection method based on spatiotemporal dual-stream network according to claim 6 is characterized in that: Adjacent pixels of the optical flow image should be smooth or continuous, and the optical flow image does not contain as many high-level semantics and complex textures as RGB images. The temporal flow prunes the number of basic units of Stage 3 in the deep layer of the network from 7 to 3 and prunes the number of basic units of Stage 4 from 3 to 1, and the spatial downsampling unit remains unchanged.
8. The fire detection method based on spatiotemporal dual-stream network according to claim 4 is characterized in that: The detection layer consists of two parts, Neck and Head. The Neck fuses multi-scale features, and the Head detects flame targets. The detection layer extracts features from network layers with resolutions of 1 / 8, 1 / 16, and 1 / 32 of the original size and inputs the features into the detection layer for multi-scale detection.
9. The fire detection method based on spatiotemporal dual-stream network according to claim 8, characterized in that: The Neck includes a CBS module and a C3 module. The CBS module is a convolutional layer, and the convolutional layer includes a 3×3 convolution, batch normalization and SiLU activation function. The C3 module is the main unit for feature extraction in the network, and the C3 module includes 3 convolutional layers and multiple BottleNecks.
10. A fire detection method based on a spatiotemporal dual-stream network according to claim 9, characterized in that: The adaptive feature fusion module comprises the following steps: S1: Preliminary fusion of temporal and spatial features using pixel-level addition F=S⊕T S∈R C×H×W Represents spatial features, T∈R C×H×W represents the temporal feature, ⊕ represents pixel-level addition, and F is the initial fusion feature; S2: Aggregate the information of each channel using global average pooling: Among them, M g ∈R C×1×1 is the aggregated channel information; S3: Use a dimension reduction layer and a dimension increase layer to capture cross-channel information interactions: M c =β(W2δ(β(W1(M g )))) in, is a dimension reduction layer, is a dimension-increasing layer, r is the channel reduction rate, δ represents the ReLU activation function, and β represents batch normalization; S4: Use the Sigmoid activation function to obtain a weight between 0 and 1: M=Sigmoid(M c ); S5: Assign weights to temporal and spatial features: in, represents element-wise multiplication, Z∈R C×H×W Represents the features after adaptive fusion.