A video fire smoke detection method and system based on spatiotemporal feature fusion

Abstract

The present invention relates to a video fire smoke detection method and system with spatiotemporal feature fusion, the method comprising: S1: collecting smoke and non-smoke video data to construct a sample set, and marking the sample set; S2: extracting the moving target area of ​​the video in the sample set by using an improved Gaussian mixture model to obtain a detection frame of the moving target; S3: based on the detection frame of the moving target, extracting the optical flow features between two frames by using an optical flow algorithm, and obtaining L horizontal and vertical dense optical flow maps respectively; cross-stacking the two in time sequence to obtain a stacked optical flow feature F τ ; Step S4: stack the optical flow features F τ and F τ The corresponding starting frame τ is input into the video fire smoke detection neural network based on the fusion of time and space features, and the fire smoke prediction result is output. The method disclosed in the present invention simultaneously utilizes the time domain and space domain features of the video for smoke recognition, obtains richer fire smoke video information, and realizes accurate and rapid automatic detection of fire smoke.

Description

Technical Field

[0001] The invention relates to the fields of fire detection, image-type fire detectors and novel fire detectors, and in particular to a video fire smoke detection method and system that integrates time and space features. Background Art

[0002] Fire is one of the most common and harmful disasters in nature and human society. It not only causes huge economic losses to society, but also has a bad impact on the natural environment and poses a great threat to human life and safety. Timely and accurate detection of fire is extremely important in fire warning and remediation. Most traditional smoke detection systems are based on the physical principles of smoke, such as smoke particle sensing and thermal sensing detectors. These solutions will only issue warnings when the cumulative change of physical quantities reaches a certain threshold, and their application in open space scenarios is very limited. With the development of hardware and signal processing technology, fire monitoring and detection methods based on video images have become an effective solution for large-space buildings and outdoor forest and grassland fires. Among them, the video smoke detection task is an early warning solution for fires. Compared with flame detection, it can detect fires more timely and buy more time for firefighting. Compared with traditional fire detection methods, video smoke detection mainly uses computer vision-related technologies and relies on visual features such as smoke color, transparency, texture and motion for target recognition. However, existing methods based on manually labeled features, such as wavelet features, turbulence features, image block statistical features, sparse optical flow features, etc., have limited representation capabilities and still have many problems in adaptability and generalization.

[0003] The patent application published in the journal "A Smoke Recognition Method and Recognition System for Image-Based Fire Detectors (CN112257523A)" provides a smoke recognition method and recognition system for image-based fire detectors to solve the technical problems of high algorithm complexity and low positioning accuracy in existing smoke recognition methods. The method obtains video images of the monitoring site and converts the video images into image sequences; sets reference points in the RGB color space; performs motion foreground area segmentation on the image sequence in the RGB color space to obtain suspected target areas; and determines the smoke movement patterns of the image sequence in the suspected target area to determine the smoke target area. The invention also solely relies on video images for fire smoke recognition.

[0004] The patent application published in the "A flame identification method for an image-based fire detector (CN111914689A)" obtains the video of the monitoring site, sets the frame rate of the image sequence in the video, performs spatial modeling on the target color in the visible light image and the target color in the infrared image, determines the dynamic target area, uses the frame difference method to lock the dynamic target area, calculates the number of displacements of the center point in the continuous frames of the dynamic target area, locks the suspected target, matches the suspected target detection results in the visible light image and the infrared image, and determines the flame target area. However, the invention relies on the displacement from the center point of the flame area to the edge position to identify the flame target, which cannot effectively eliminate interference such as sunlight and certain lighting interference, and is prone to false alarms. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a video fire smoke detection method and system with spatiotemporal feature fusion.

[0006] The technical solution of the present invention is: a video fire smoke detection method with spatiotemporal feature fusion, comprising:

[0007] Step S1: Collecting original smoke and non-smoke video data under different background environments to construct a sample set, and labeling the sample set, labeling the data without smoke as 0 and the data with smoke as 1;

[0008] Step S2: using an improved Gaussian mixture model T2 FGMM to extract the moving target area of ​​the video in the sample set to obtain a detection frame of the moving target;

[0009] Step S3: Based on the detection frame of the moving target, the optical flow algorithm is used to extract the optical flow features between two frames in L consecutive frames, and L horizontal dense optical flow maps F are obtained respectively. u and L vertical dense optical flow maps F v ; The optical flow features of the 2L frames are stacked crosswise in time order to obtain a stacked optical flow feature F of a 2L channel τ ;

[0010] Step S4: stack the optical flow features F τ and F τ The corresponding starting frame τ is input into the video fire smoke detection neural network based on the fusion of time and space features, wherein the stacked optical flow features are passed through the time convolutional neural network; the original color video image is processed through the spatial stream convolutional neural network to obtain deep features in the time domain and the spatial domain respectively, and the deep features in the time domain and the spatial domain are fused to obtain the fused deep features; the fused deep features are processed through the average pooling and Flatten layer to output the fire smoke prediction result.

[0011] Compared with the prior art, the present invention has the following advantages:

[0012] The present invention discloses a video fire smoke detection method with spatiotemporal feature fusion, which uses the airspace information and time domain information of the fire video for fire detection at the same time, and can more comprehensively describe the characteristics of the fire when it occurs. Compared with the existing fire detection method, the present invention uses richer description information to judge whether a fire has occurred, overcomes the deficiency of the existing technology that only relies on airspace features such as color and shape for fire analysis, and improves the reliability of fire monitoring. In addition, the method disclosed by the present invention uses a deep learning method to perform multi-data stream analysis, which can extract more sufficient fire features and more effectively distinguish fire targets from other interferences compared to the commonly used image analysis method based on video frames; it can alarm faster, buy more time for later fire warning, reduce fire losses, and effectively promote the intelligent development of firefighting equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of a video fire smoke detection method using spatiotemporal feature fusion in an embodiment of the present invention;

[0014] Figure 2 Schematic diagram of the optical flow feature stacking method in an embodiment of the present invention;

[0015] Figure 3 Schematic diagram of the neural network structure of video fire smoke detection based on spatiotemporal feature fusion in an embodiment of the present invention;

[0016] Figure 4 The structure block diagram of a video fire and smoke detection system with spatiotemporal feature fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The present invention provides a video fire smoke detection method with time-space feature fusion, which simultaneously utilizes the time domain and space domain features of the video to perform smoke recognition, obtains richer fire smoke video information, and realizes accurate and rapid automatic detection of fire smoke.

[0018] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0019] Embodiment 1

[0020] like Figure 1 As shown, a video fire smoke detection method using spatiotemporal feature fusion is provided in an embodiment of the present invention, comprising the following steps:

[0021] Step S1: Collect the original smoke and non-smoke video data under different background environments to construct a sample set, and annotate the sample set, annotating the data without smoke as 0 and the data with smoke as 1;

[0022] Step S2: using the improved Gaussian mixture model T2 FGMM to extract the moving target area of ​​the video in the sample set and obtain the detection frame of the moving target;

[0023] Step S3: Based on the detection frame of the moving target, the optical flow algorithm is used to extract the optical flow features between two frames in L consecutive frames, and L horizontal dense optical flow maps F are obtained respectively. u and L vertical dense optical flow maps F v ; Stack the optical flow features of the 2L frames crosswise in time order to obtain a stacked optical flow feature F of the 2L channel τ ;

[0024] Step S4: Stack the optical flow features F τ and F τ The corresponding starting frame τ is input into the video fire smoke detection neural network based on the fusion of time and space features, wherein the stacked optical flow features are passed through the temporal convolutional neural network; the original color video image is processed by the spatial stream convolutional neural network to obtain the deep features in the time domain and the spatial domain respectively, and the deep features in the time domain and the spatial domain are fused to obtain the fused deep features; the fused deep features are processed by the average pooling and Flatten layer to output the fire smoke prediction results.

[0025] In one embodiment, the above step S1: collects original smoke and non-smoke video data under different background environments to construct a sample set, and labels the sample set, where data without smoke is labeled as 0, and data with smoke is labeled as 1;

[0026] The embodiment of the present invention collects original smoke and non-smoke video data under different background environments. The fire smoke video data and non-smoke video data used are from multiple online public data sets. The video is processed into a video clip in mp4 format with a length of 10-25 seconds and 25 frames per second. Finally, 250 smoke and non-smoke videos are obtained to construct the sample set of the embodiment of the present invention. Among them, the smoke video covers the smoke conditions of various scenes, and the non-smoke video content is the monitoring scene and the sample of the interference movement. The video data of the sample set is annotated, and the data without smoke is annotated as 0, and the data with smoke is annotated as 1.

[0027] In one embodiment, the above step S2: using the improved Gaussian mixture model T2 FGMM to extract the moving target area of ​​the video in the sample set to obtain the detection frame of the moving target specifically includes:

[0028] The T2 FGMM model uses the type-two fuzzy set membership function in set theory to process the uncertain mean and covariance in the Gaussian mixture model, so that the observation probability in the T2 FGMM is within an interval, and then constrained by the generalized linear model GLM to perform subsequent tasks; after obtaining the binary image of the moving target through the background subtraction algorithm, the detection frame of the moving target is obtained by combining noise reduction and morphological processing; the observation value at time t is X t The probability of can be described by K Gaussian distribution modeling, as shown in formulas (1) to (2):

[0029]

[0030]

[0031] Among them, the parameter K represents the number of Gaussian distributions, η(X t ,μ i,t ,Σ i,t ) is the i-th Gaussian density function in the Gaussian mixture model GMM at time t, where μ i,t is the mean vector, σ i,t is the variance, Σ i,t is the covariance matrix, n is the number of color channels, ω i,t is its corresponding weight.

[0032] In one embodiment, the above step S3: based on the detection frame of the moving target, the optical flow algorithm is used to extract the optical flow features between two frames in L consecutive frames, and L horizontal dense optical flow maps F are obtained respectively. u and L vertical dense optical flow maps F v ; Stack the optical flow features of the 2L frames crosswise in time order to obtain a stacked optical flow feature F of the 2L channel τ , specifically including:

[0033] Step S31: Let the video sampling start frame be τ, use the TV-L1 optical flow algorithm, extract the optical flow features between two frames in L consecutive frames, and obtain L horizontal dense optical flow maps F respectively. u and L vertical dense optical flow maps F v ;

[0034] Step S32: stack the optical flow features of the 2L frames crosswise in time order to obtain a stacked optical flow feature F of a 2L channel τ .

[0035] like Figure 2As shown in FIG. 1 , let the starting frame of video sampling be τ, extract the optical flow features between two frames in L consecutive frames, obtain a total of 2L dense optical flow maps of the horizontal and vertical components, respectively, stack the L horizontal optical flow maps from τ+1 to τ+L and the L vertical optical flow maps from τ+1 to τ+L in time sequence to form a stacked feature of 2L channels, and cross-connect the dense optical flows between multiple frames of the video to describe the continuous motion of the video content.

[0036] In one embodiment, the above step S4: stacking the optical flow features F τ and F τ The corresponding starting frame τ is input into the video fire smoke detection neural network based on spatiotemporal feature fusion, where the starting frame τ passes through the spatial stream convolutional neural network and stacks the optical flow feature F τ After being processed by the time convolutional neural network, the spatial and temporal features are obtained respectively; the temporal and spatial features are fused to obtain the fused deep features; the fused deep features are processed by average pooling and Flatten layer to obtain the fire smoke prediction results, which include:

[0037] Step S41: The spatial stream convolutional neural network is used to extract the spatial domain features of the starting frame τ;

[0038] The spatial stream convolutional neural network in the embodiment of the present invention is used to process the static image of the sampled frame of the video, that is, the starting frame τ, to obtain the deep image features in the image. The spatial stream convolutional neural network uses ResNet50 with pre-trained weights on ImageNet, and its input is the starting frame image of size w×h×3, such as Figure 3 As shown, the spatial stream convolutional neural network in the embodiment of the present invention uses five convolutional layers to extract features to obtain spatial domain features.

[0039] For the sampling of the starting frame, the embodiment of the present invention adopts the TSN network input method to sample the video data, uniformly samples to obtain K starting frame images, obtains their classification scores through the spatial stream convolutional neural network, and obtains the video-level prediction results according to the classification score statistics of each frame.

[0040] Step S42: Temporal convolutional neural network is used to extract stacked optical flow features F τ The time domain characteristics of

[0041] The temporal convolutional neural network in the embodiment of the present invention is used to process the stacked optical flow features F of the video sampling frames. τThe temporal features of the convolutional neural network can explicitly express the continuous motion information of multiple frames, so that the temporal convolutional neural network does not need to implicitly estimate the motion. The recognition results are different from those obtained by static images, and complement the features obtained by the spatial convolutional neural network. The structure of the temporal convolutional neural network is consistent with that of the spatial convolutional neural network. The difference is that the input of the network is a stacked optical flow F of size w×h×2L. τ , the weights of each channel in the first layer of the spatial stream convolutional neural network are averaged and then copied to each channel in the first layer of the temporal convolutional neural network. The spatial stream convolutional neural network is used for feature extraction to obtain the time domain features.

[0042] Step S43: Fusing the time domain and space domain features to obtain fused deep features;

[0043] At the end of the two neural network convolution structures, the spatial and temporal features are added and fused to obtain the fused deep features. The fused deep features are used instead of the spatial and temporal features to complete subsequent calculations, which can reduce the number of parameters of the fully connected layer by half and obtain a classifier for the fused deep features.

[0044] Step S44: The fused deep features are processed by average pooling and Flatten layer to obtain the fire smoke prediction result.

[0045] The fused deep features are averaged and pooled and then Flattened to obtain a total feature vector, which is then discriminated and output to finally obtain the classification result of whether it is a fire.

[0046] After the initial training is completed, the video fire smoke detection neural network model based on spatiotemporal feature fusion is tested. If the accuracy is low, the network model performance is improved by increasing training data, correcting network model parameters, etc. until the preset standards are met. The training is then completed and a trained video fire smoke detection neural network model based on spatiotemporal feature fusion is obtained.

[0047] In actual video surveillance and fire detection, after real-time video data is collected, the video data is standardized and input into the trained video fire and smoke detection neural network based on spatiotemporal feature fusion for prediction. If a fire is predicted, an audible and visual alarm is issued to notify relevant personnel to handle it. If no fire is predicted, data collection for fire monitoring continues.

[0048] The method disclosed in the present invention needs to be implemented by software and run on a computer with a graphics processor, and the data capture device used is a high-definition color visible light camera. The camera and the computer are connected, and can be connected by wire or wirelessly. First, turn on the power, start the detector, test the detector, and check on the computer whether the detector is working normally. The present invention can also be installed on a computer in an existing video surveillance system in a monitoring location. When the monitoring video is transmitted back to the computer of the monitoring center, the method disclosed in the present invention is run to process the monitoring video to form an intelligent security system. The present invention is suitable for places where image-type fire detectors need to be installed, such as logistics warehouses, grain warehouses, large space buildings such as Bo Wuguang, or forest and grassland fire monitoring places.

[0049] The present invention discloses a video fire smoke detection method with spatiotemporal feature fusion, which uses the airspace information and time domain information of the fire video for fire detection at the same time, and can more comprehensively describe the characteristics of the fire when it occurs. Compared with the existing fire detection method, the present invention uses richer description information to judge whether a fire has occurred, overcomes the deficiency of the existing technology that only relies on airspace features such as color and shape for fire analysis, and improves the reliability of fire monitoring. In addition, the method disclosed by the present invention uses a deep learning method to perform multi-data stream analysis, which can extract more sufficient fire features and more effectively distinguish fire targets from other interferences compared to the commonly used image analysis method based on video frames; it can alarm faster, buy more time for later fire warning, reduce fire losses, and effectively promote the intelligent development of firefighting equipment.

[0050] Embodiment 2

[0051] like Figure 4 As shown, the embodiment of the present invention provides a video fire smoke detection system with spatiotemporal feature fusion, including the following modules:

[0052] A sample set acquisition module 51 is used to collect original smoke and non-smoke video data under different background environments to construct a sample set, and to mark the sample set, where data without smoke is marked as 0 and data with smoke is marked as 1;

[0053] A moving target detection frame acquisition module 52 is used to extract the moving target region of the video in the sample set by using the Gaussian mixture model improved by T2 to acquire the moving target detection frame;

[0054] The stacked optical flow feature acquisition module 53 is used to extract the optical flow features between two frames in L consecutive frames using the optical flow algorithm based on the detection frame of the moving target, and obtain L horizontal dense optical flow maps F respectively. u and L vertical dense optical flow maps F v; The optical flow features of the 2L frames are stacked crosswise in time order to obtain a stacked optical flow feature F of a 2L channel τ ;

[0055] The fire smoke prediction result module 54 is used to stack the optical flow features F τ and F τ The corresponding starting frame τ is input into the video fire smoke detection neural network based on the fusion of time and space features, wherein the stacked optical flow features are passed through the time convolutional neural network; the original color video image is processed through the spatial stream convolutional neural network to obtain deep features in the time domain and the spatial domain respectively, and the deep features in the time domain and the spatial domain are fused to obtain the fused deep features; the fused deep features are processed through the average pooling and Flatten layer to output the fire smoke prediction result.

[0056] The above embodiments are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention should all be included within the scope of the present invention.

Claims

1. A video fire smoke detection method based on spatiotemporal feature fusion, characterized in that: include: Step S1: Collecting original smoke and non-smoke video data under different background environments to construct a sample set, and labeling the sample set, labeling the data without smoke as 0 and the data with smoke as 1; Step S2: using the improved Gaussian mixture model T2 FGMM to extract the moving target area of ​​the video in the sample set to obtain the detection frame of the moving target, specifically including: The T2 FGMM model processes the uncertain mean and covariance in the Gaussian mixture model through the type-two fuzzy set membership function in set theory, so that the observation probability in the T2 FGMM is within an interval, and then constrained by the generalized linear model GLM to perform subsequent tasks; after obtaining the binary image of the moving target through the background subtraction algorithm, the detection frame of the moving target is obtained by combining noise reduction and morphological processing; wherein the observation value at time t is The probability of is described by K Gaussian distribution modeling, as shown in formulas (1) to (2): (1) (2) Among them, the parameter K represents the number of Gaussian distributions, is the i-th Gaussian density function in the Gaussian mixture model GMM at time t, where is the mean vector, is the covariance matrix, n is the number of color channels, is its corresponding weight; Step S3: Based on the detection frame of the moving target, the optical flow algorithm is used to extract the optical flow features between two frames in L consecutive frames, and L horizontal dense optical flow maps are obtained respectively. and L vertical dense optical flow maps ; Stack the optical flow features of the 2L frames crosswise in time order to obtain a stacked optical flow feature of a 2L channel , including: Step S31: Let the video sampling start frame be τ, use the TV-L1 optical flow algorithm, extract the optical flow features between two frames in L consecutive frames, and obtain L horizontal dense optical flow maps respectively and L vertical dense optical flow maps ; Step S32: stack the optical flow features of the 2L frames crosswise in time order to obtain a stacked optical flow feature of a 2L channel ; Step S4: stacking the optical flow features and The corresponding starting frame τ is input into the video fire smoke detection neural network based on the fusion of time and space features, wherein the stacked optical flow features are passed through the time convolutional neural network; the original color video image is processed through the spatial stream convolutional neural network to obtain deep features in the time domain and the spatial domain respectively, and the deep features in the time domain and the spatial domain are fused to obtain the fused deep features; the fused deep features are processed through the average pooling and Flatten layer to output the fire smoke prediction result.

2. The video fire smoke detection method based on spatiotemporal feature fusion according to claim 1 is characterized in that: The step S4: stacking the optical flow features and The corresponding starting frame τ is input into the video fire smoke detection neural network based on spatiotemporal feature fusion, wherein the starting frame τ passes through the spatial stream convolutional neural network, and the stacked optical flow feature After being processed by a time convolutional neural network, the spatial and temporal features are obtained respectively; the temporal and spatial features are fused to obtain fused deep features; the fused deep features are processed by average pooling and Flatten layers to obtain fire smoke prediction results, which specifically include: Step S41: the spatial stream convolutional neural network is used to extract the spatial domain features of the starting frame τ; Step S42: The temporal convolutional neural network is used to extract the stacked optical flow features The time domain characteristics of Step S43: fusing the time domain and space domain features to obtain fused deep features; Step S44: The fused deep features are processed by average pooling and Flatten layer to obtain a fire smoke prediction result.

3. A video fire smoke detection system with spatiotemporal feature fusion, characterized in that: Includes the following modules: A sample set acquisition module is used to collect original smoke and non-smoke video data under different background environments to construct a sample set, and annotate the sample set, where data without smoke is annotated as 0 and data with smoke is annotated as 1; The module for obtaining the detection frame of the moving target uses the Gaussian mixture model improved by T2 to extract the moving target area of ​​the video in the sample set to obtain the detection frame of the moving target, which specifically includes: The T2 FGMM model processes the uncertain mean and covariance in the Gaussian mixture model through the type-two fuzzy set membership function in set theory, so that the observation probability in the T2 FGMM is within an interval, and then constrained by the generalized linear model GLM to perform subsequent tasks; after obtaining the binary image of the moving target through the background subtraction algorithm, the detection frame of the moving target is obtained by combining noise reduction and morphological processing; wherein the observation value at time t is The probability of is described by K Gaussian distribution modeling, as shown in formulas (1) to (2): (1) (2) Among them, the parameter K represents the number of Gaussian distributions, is the i-th Gaussian density function in the Gaussian mixture model GMM at time t, where is the mean vector, is the covariance matrix, n is the number of color channels, is its corresponding weight; Obtain a stacked optical flow feature module, which is used to extract the optical flow features between two frames in L consecutive frames based on the detection frame of the moving target, and obtain L horizontal dense optical flow maps respectively. and L vertical dense optical flow maps ; Stack the optical flow features of the 2L frames crosswise in time order to obtain a stacked optical flow feature of a 2L channel , including: Step S31: Let the video sampling start frame be τ, use the TV-L1 optical flow algorithm, extract the optical flow features between two frames in L consecutive frames, and obtain L horizontal dense optical flow maps respectively and L vertical dense optical flow maps ; Step S32: stack the optical flow features of the 2L frames crosswise in time order to obtain a stacked optical flow feature of a 2L channel ; The fire smoke prediction result module is used to stack the optical flow features and The corresponding starting frame τ is input into the video fire smoke detection neural network based on the fusion of time and space features, wherein the stacked optical flow features are passed through the time convolutional neural network; the original color video image is processed through the spatial stream convolutional neural network to obtain deep features in the time domain and the spatial domain respectively, and the deep features in the time domain and the spatial domain are fused to obtain the fused deep features; the fused deep features are processed through the average pooling and Flatten layer to output the fire smoke prediction result.

Images