Fire detection system and method based on pulse neural network

Abstract

The invention provides a fire detection system and method based on a pulse neural network. An image acquisition module is used for reading image frames of a video; the pulse coding module forms image pulse sequence data; the fire feature extraction module extracts related features of the image pulse sequence data layer by layer to respectively obtain a shallow fire feature map and a deep fire feature map; the fire feature extraction and fusion module fuses the high-resolution features of the shallow fire feature map and the low-resolution features of the deep fire feature map; the pulse activation module triggers a fire characteristic time sequence pulse sequence; the fire target detection module optimizes the fire characteristic time sequence pulse sequence to obtain fire characteristic core data; the pulse decoding module obtains classification and bounding box parameters of a fire target; the fire result output module outputs a final fire detection result after screening; according to the invention, high-precision fire detection performance is realized, and the advantage of low power consumption of the pulse neural network is maintained.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and deep learning, and specifically relates to a fire detection method based on a spiking neural network, which is suitable for real-time monitoring of fires in multiple scenes such as forests, factories, high-rise buildings, and indoor residences, and can be deployed on resource-constrained devices such as unmanned aerial vehicles, embedded cameras, and mobile terminals to achieve low-power, high-precision flame and smoke detection. BACKGROUND

[0002] Fire is a major disaster that threatens human life and property safety and the integrity of the ecological system. In the process of industrialization and urbanization, forest, chemical park, high-rise building, and other scenes have a high incidence of fire accidents. The traditional fire monitoring system relies on temperature sensors and smoke sensors, which have the defects of long response delay and high false alarm rate. The sensors need to collect environmental parameters in a contact manner, which is difficult to capture early fire, and is easily triggered by environmental interference such as dust and humidity, which interferes with the allocation of emergency resources.

[0003] With the development of machine vision and deep learning technology, video stream-based fire detection algorithms have gradually replaced traditional sensor solutions. Deep learning models represented by Faster R-CNN and YOLO series significantly improve detection accuracy by automatically extracting features such as flame color, flicker frequency (2-20Hz), and smoke diffusion dynamics. However, existing deep learning models have a "precision-energy consumption" contradiction: the YOLOv3 model has a parameter size of 61.5M, a single inference memory usage of over 8GB, and an energy consumption of over 350mJ, making it difficult to deploy on resource-constrained devices such as unmanned aerial vehicles and edge cameras. Although YOLOv5 and YOLOv8 have reduced parameter size through lightweight design, they still follow the continuous value calculation paradigm of traditional artificial neural networks (ANN), and the energy consumption problem has not been fundamentally solved.

[0004] As a core architecture of brain-like computing, spiking neural networks (SNN) simulate the pulse firing mechanism of biological neurons, and have the advantages of low power consumption, event-driven, and high temporal resolution. Only when a pulse is triggered, calculation is performed, and the operation is mainly simple addition, with energy consumption only 1 / 20-1 / 5 of traditional ANN. Therefore, how to design a network structure that adapts to SNN to improve the multi-scale fire target detection accuracy while maintaining low power consumption has become a technical problem that needs to be solved. SUMMARY

[0005] The present application provides a fire detection algorithm based on a spiking neural network, which extracts features through an improved YOLOv2_R34 network structure and outputs fire detection information using a pulse frequency decoding method, significantly reducing computational power consumption while maintaining detection accuracy, To solve the problems of the prior art, the present application adopts the following technical solutions: A fire detection system based on a spiking neural network, the system comprising a video image acquisition module, a pulse coding module, a fire feature extraction module, a fire feature extraction fusion module, a pulse activation module, a fire target detection module, a pulse decoding module and a fire result output module; wherein: The image acquisition module is used to read the image frames of the camera video stream or stored video. The pulse coding module arranges the image frames in time dimension to form image pulse sequence data. The fire feature extraction module uses a convolution layer with a residual structure to extract relevant features layer by layer from the image pulse sequence data to obtain shallow fire feature maps and deep fire feature maps. The fire feature extraction fusion module fuses the high-resolution features of the shallow fire feature maps with the low-resolution features of the deep fire feature maps, and obtains multi-scale fire feature data by channel dimension splicing of the feature maps. The pulse activation module processes the multi-scale fire feature data to trigger fire feature time sequence pulses through the membrane dynamics model of SIF neurons. The fire target detection module obtains fire feature core data by optimizing the fire feature time sequence pulses through a detection head with deepened depth and dilated convolution. The pulse decoding module obtains the classification and bounding box parameters of the fire target by mapping the fire feature core data through frequency values. The fire result output module outputs the final fire detection result after filtering through a non-maximum suppression bounding box and a confidence threshold.

[0006] Further, the fire feature extraction module adopts a YOLOv2_ResNet34 network; the YOLOv2_ResNet34 network comprises: Layers 1, 2, 3 and 4 respectively contain 3, 4, 6 and 3 basic residual blocks, each basic residual block contains two 3x3 convolution layers, each convolution layer is connected with a BN layer in turn, the output channel numbers of layers 1, 2, 3 and 4 are 64, 128, 256 and 512 respectively, and the step of the first convolution layer of the first basic residual block of each layer is 1, 2, 2 and 2 respectively.

[0007] Further, the fire feature extraction fusion module adopts a neck feature fusion network of Passthrough of YOLOv2.

[0008] Further, the fire feature extraction fusion module fuses the high-resolution features of the shallow fire feature maps with the low-resolution features of the deep fire feature maps, and obtains multi-scale fire feature data by channel dimension splicing of the feature maps; the process comprises: The third layer in the fire feature extraction module extracts a first shallow fire feature map from the image pulse sequence data; The first shallow fire feature map is split into four sub-tensors by slicing operation, and each pixel contains a value. The W and H information is concentrated in the channel space, and the input channel is expanded by four times. Meanwhile, the second shallow fire feature map is obtained by concatenating along the channel dimension. The first deep feature map output by the fourth layer in the fire feature extraction module is calculated by a convolution layer to obtain a second deep feature map. The high-resolution features of the second shallow fire feature map and the low-resolution features of the second deep fire feature map are spliced and integrated by the Passthrough structure to obtain multi-scale fire feature data.

[0009] Further, the pulse activation module processes the multi-scale fire feature data through the membrane dynamics model of the SIF neuron to trigger the process of the fire feature time sequence pulse sequence; including: The membrane dynamics model of the SIF neuron is constructed by the following formula:

[0010] Wherein: represents the membrane potential of neuron i at the first layer at time step t; represents the total membrane charge of neuron i at the first layer at time step t; represents the number of triggered pulses of neuron i at the first layer at time step t; represents the triggered time sequence pulse at the first layer at time step t; are the positive threshold and the negative threshold respectively ， is a small negative value (-1e-3); The membrane potential threshold of the membrane dynamics model of the SIF neuron is set to is set to 1.0, the reset potential is 0, and when the membrane potential exceeds the threshold, a pulse with a value of 1 is emitted, and the membrane potential is reset to 0.

[0011] Further, the fire target detection module adopts the detection head network of Spiking Deep-Dilated-Dense YOLOv2; the detection head network includes four 3x3 convolution layers.

[0012] Further, the pulse decoding module maps the fire feature core data by frequency value to obtain the classification and bounding box parameters of the fire target, including: By accumulating the pulse cumulative amount of the neuron within T time steps, and then dividing by T, the continuous pulse frequency reflecting the activation intensity of the neuron is obtained; The frequency value is mapped to fire feature core data containing an offset, a confidence and a category probability, and conversion from the time sequence pulse signal to the spatial target parameter is completed.

[0013] To solve the prior art, the application can also adopt a fire detection method based on a pulse neural network, comprising the following steps: Image frames for reading camera video streams or stored videos; The image frames are arranged according to the time dimension to form image pulse sequence data; The image pulse sequence data is extracted layer by layer to obtain relevant features by using a convolution layer with a residual structure, and shallow fire feature maps and deep fire feature maps are obtained; The high-resolution features of the shallow fire feature maps and the low-resolution features of the deep fire feature maps are fused respectively, and multi-scale fire feature data is obtained by channel dimension splicing of the feature maps; The multi-scale fire feature data is processed to trigger a fire feature time sequence pulse sequence through a membrane dynamics model of a SIF neuron; wherein: The membrane dynamics model of the SIF neuron is constructed by the following formula:

[0014] Wherein: represents the membrane potential of neuron i in the first layer at time step t; represents the total membrane charge of neuron i in the first layer at time step t; represents the number of triggered pulses of neuron i in the first layer at time step t; represents the triggered time sequence pulse in the first layer at time step t; are the positive threshold and the negative threshold respectively ， is a small negative value (-1e-3); The membrane dynamics model of the SIF neuron is set to have a membrane potential threshold is set to 1.0, the reset potential is 0, when the membrane potential exceeds the threshold, a pulse with a value of 1 is emitted, and the membrane potential is reset to 0; The fire feature core data is obtained by detecting the head depth deepening and expanding the convolution to optimize the processing of the fire feature time sequence pulse sequence; The classification and the bounding box parameters of the fire target are obtained by mapping the fire feature core data through the frequency value; The redundant boxes of the bounding box are suppressed by non-maximum suppression, and the final fire detection result is output after confidence threshold screening.

[0015] Advantages This invention uses a spiking neural network (SNN) as the core architecture of brain-like computing, simulating the spiking mechanism of biological neurons. It has the advantages of low power consumption, event-driven operation, and high temporal resolution—computation is performed only when a spiking is triggered, and the operation is mainly simple addition, with energy consumption only 1 / 20 to 1 / 5 of that of traditional ANN.

[0016] This invention improves the accuracy of multi-scale fire target detection while maintaining the advantage of low power consumption, which is a technical problem that urgently needs to be solved.

[0017] The fire detection algorithm based on spiking neural networks proposed in this invention achieves high-precision fire detection performance through improved network structure design, while maintaining the low power consumption advantage of spiking neural networks, and has high practical application value. Attached Figure Description

[0018] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is the network structure diagram of the algorithm of this invention; Figure 3 This is a diagram showing the fire detection results of the algorithm of this invention. Detailed Implementation

[0019] To verify the effectiveness of the algorithm of this invention, and to make the technical problem to be solved, the technical solution, and the advantages of this invention clearer, the following will be discussed in conjunction with the appendix. Figure 1 Appendix Figure 2 The specific embodiments are described in detail below.

[0020] This invention provides a fire detection system based on a pulse neural network. The system includes a video image acquisition module, a pulse coding module, a fire feature extraction module, a fire feature extraction and fusion module, a pulse activation module, a fire target detection module, a pulse decoding module, and a fire result output module; wherein: The image acquisition module is used to read the camera video stream or store image frames of the video; The pulse coding module arranges the image frames according to the time dimension to form image pulse sequence data; The fire feature extraction module uses a convolutional layer with residual structure to extract relevant features from the image pulse sequence data layer by layer to obtain shallow fire feature maps and deep fire feature maps respectively; The fire feature extraction and fusion module fuses the high-resolution features of shallow fire feature maps with the low-resolution features of deep fire feature maps, and obtains multi-scale fire feature data by stitching the feature maps in the channel dimension. The pulse activation module triggers a fire feature temporal pulse sequence by processing multi-scale fire feature data through the membrane dynamics model of SIF neurons; The fire target detection module obtains core fire feature data by optimizing the fire feature time-series pulse sequence through detection head depth enhancement and dilated convolution. The pulse decoding module obtains the classification and bounding box parameters of fire targets by mapping the frequency values ​​to the core fire feature data; The fire result output module suppresses redundant bounding boxes using non-maximum suppression and outputs the final fire detection result after filtering by a confidence threshold.

[0021] This embodiment provides a fire detection algorithm based on a pulse neural network, the specific steps of which are as follows: Figure 1 As shown, it includes the following steps: (1) Video data input: Read image frames from the camera's real-time video stream or stored video; (2) The temporal spread coding method is adopted, and the specific implementation process is as follows: The input RGB image (size 3×H×W) is repeated T times along the time dimension to form a four-dimensional tensor of size T×3×H×W. Then, it is processed through a convolutional coding layer, which uses a 7×7 convolutional kernel, such as... Figure 2 As shown, the above process is the image encoding process; (3) Fire Feature Extraction Module: The fire feature extraction module extracts fire-related features layer by layer from the input image, and provides high semantic and low redundancy basic features for subsequent modules through downsampling and channel enhancement. The core is a residual block connection structure, which alleviates gradient vanishing and avoids the problem of "increased depth but degraded performance" in traditional networks, thus realizing deep extraction of fire features; wherein: The fire feature extraction module uses YOLOv2_R34, specifically a ResNet34 network, allowing a network depth of up to 34 layers. The core idea of ​​residual networks is to introduce "skip connections" so that the network learns the residual (i.e., the difference) between the input and output, rather than directly learning complex mapping relationships. Specifically, the residual structure adds the outputs F(x) after the input x undergoes several layers of nonlinear transformations to obtain the final output H(x) = F(x) + x. This design alleviates the gradient vanishing problem in deep networks and solves the network degradation phenomenon.

[0022] like Figure 2 As shown, the fire feature extraction module specifically includes four residual block layers. Layers 1, 2, 3, and 4 contain 3, 4, 6, and 3 basic residual blocks, respectively. Each basic residual block contains two 3×3 convolutional layers. Each convolutional layer is followed by a BN (Batch Normalization) layer. The number of output channels for layers 1, 2, 3, and 4 are 64, 128, 256, and 512, respectively. The stride of the first convolutional layer of the first basic residual block in each layer is 1, 2, 2, and 2, respectively.

[0023] (4) Fire feature extraction and fusion module: the neck feature fusion network adopts the Passthrough structure of YOLOv2, and the specific steps of Passthrough are as follows: ①The output feature map (26x26x256) of layer 3 of ResNet34 is output through a 1x1 convolution kernel, and the first fire shallow feature map of 26x26x128 is output; ②The output feature map of 26x26x128 is split into 4 sub-tensors (13x13x128) through slicing operation, and the specific operation of slicing is to take a value every other pixel, and the W and H information is concentrated in the channel space, and the input channel is expanded by 4 times; ③The 4 sub-tensors are spliced in the channel dimension to obtain the second shallow fire feature map (13x13x512); ④The first deep feature map (13x13x512) output by layer 4 of ResNet34 is first subjected to several convolution layers to obtain the second deep fire feature map (13x13x1024), and the second shallow fire feature map obtained in ③ is spliced in the channel dimension to form a multi-scale fire feature map of 13x13x1536.

[0024] The Passthrough structure fuses the shallow high-resolution feature (26x26x512) of ResNet34 and the deep low-resolution feature (13x13x1024), integrates the deep and shallow features through feature splicing, enables the deep network to utilize high-resolution details and low-resolution semantic information at the same time, enhances the small target detection capability, and solves the detection difficulty of large scale difference of fire targets.

[0025] (5) Pulse activation module: the output of the convolution layer is processed by the SIF neuron, the membrane potential threshold of the SIF neuron is set to 1.0, the reset potential is 0, when the membrane potential exceeds the threshold, a pulse with a value of 1 is emitted, and the membrane potential is reset to 0, and the membrane dynamics formula of the SIF neuron is as follows:

[0026] (1-1); (1-2); Wherein: represents the membrane potential of neuron i in the first layer at time step t; represents the total membrane charge of neuron i in the first layer at time step t; represents the number of triggered pulses of neuron i in the first layer at time step t; represents the triggered time sequence pulse at the first layer at time step t; are the positive threshold and the negative threshold respectively ， ​A small negative value (-1e-3). In practical applications, the present application sets the time step t=3, the input image size is 3x640x640, and after encoding, the pulse feature tensor with a size of 3x64x640x640 is obtained.

[0027] (6) Fire target detection module: this module adopts the structure of Spiking Deep-Dilated-Dense YOLOv2, which introduces three key improvements of deepening the detection head, adding dilated convolution, and using dense connection on the basis of YOLOv2 detection head. After improvement, the steps are as follows: ① the second fire feature map (13x13x1536) is reduced through a 3x3 dimension reduction convolution layer with a step of 1 to reduce the fire pulse neural network calculation complexity and parameter quantity, while maintaining the feature expression ability, and the output is 13x13x512; ② pass through 4 3x3 convolution layers in turn to expand the field of view, enhance the feature extraction ability, the dilation rates of the 4 3x3 convolutions are 1, 2, 3 and 4 respectively to expand the receptive field without increasing the parameter quantity, and the first three convolutions are densely connected to alleviate the gradient vanishing problem, and the features at different levels can be complementary and fused; ③ finally pass through a 1x1 prediction convolution layer, output the target detection result, including the offset (tx, ty, tw, th) of the 5 Anchor Boxes, the confidence (score) and the class probability (class probabilities) data information.

[0028] Among them: deepening the detection head and using dense connection can enhance semantic extraction and feature reuse, adding dilated convolution provides multi-scale receptive field to cover different size targets, the above enhances the expression ability of the detection head network to multi-scale features, further extracts image features, improves detection accuracy, and realizes the core data of fire features; (6) Pulse information decoding: the pulse frequency decoding method is adopted to realize information output, the pulse accumulation of neurons in T time steps is accumulated, and then divided by T to obtain the continuous pulse frequency reflecting the activation intensity of neurons. The frequency value is mapped to the specific information of the fire target, including the classification, confidence and bounding box parameters of the fire target, and the conversion from time sequence pulse signal to spatial target parameter is completed. (7) Detection result output: the detection frame generated by the multi-scale feature is processed by non-maximum suppression (NMS), and after sorting from high to low according to the confidence, the overlapping redundant frame is removed according to the IoU threshold, and the optimal detection frame which can accurately cover the target is reserved. Then, through the confidence threshold, low confidence results are removed. Finally, the results containing the fire target category, pixel-level bounding box coordinates and confidence are output, and if the fire target is detected, the alarm is triggered and the position information is uploaded. Figure 3 The fire detection results of the algorithm of the present application are shown, which verifies that the method can meet the technical requirements of actual fire detection applications.

Claims

1. A fire detection system based on a pulse neural network, characterized in that, The system includes a video image acquisition module, a pulse coding module, a fire feature extraction module, a fire feature extraction and fusion module, a pulse activation module, a fire target detection module, a pulse decoding module, and a fire result output module; wherein: The image acquisition module is used to read the camera video stream or store image frames of the video; The pulse coding module arranges the image frames according to the time dimension to form image pulse sequence data; The fire feature extraction module uses a convolutional layer with residual structure to extract relevant features from the image pulse sequence data layer by layer to obtain shallow fire feature maps and deep fire feature maps respectively; The fire feature extraction and fusion module fuses the high-resolution features of shallow fire feature maps with the low-resolution features of deep fire feature maps, and obtains multi-scale fire feature data by stitching the feature maps in the channel dimension. The pulse activation module triggers a fire feature temporal pulse sequence by processing multi-scale fire feature data through the membrane dynamics model of SIF neurons; The fire target detection module obtains core fire feature data by optimizing the fire feature time-series pulse sequence through detection head depth enhancement and dilated convolution. The pulse decoding module obtains the classification and bounding box parameters of fire targets by mapping the frequency values ​​to the core fire feature data; The fire result output module suppresses redundant bounding boxes using non-maximum suppression and outputs the final fire detection result after filtering by a confidence threshold.

2. The fire detection system based on a pulse neural network according to claim 1, characterized in that, The fire feature extraction module employs a YOLOv2_ResNet34 network; the YOLOv2_ResNet34 network includes: Layers 1, 2, 3, and 4 contain 3, 4, 6, and 3 basic residual blocks, respectively. Each basic residual block contains two 3×3 convolutional layers. Each convolutional layer is followed by a batch normalization (BN) layer. The number of output channels for layers 1, 2, 3, and 4 are 64, 128, 256, and 512, respectively. The stride of the first convolutional layer in the first basic residual block of each layer is 1, 2, 2, and 2, respectively.

3. The fire detection system based on a pulse neural network according to claim 1, characterized in that, The fire feature extraction and fusion module uses YOLOv2's Passthrough neck feature fusion network.

4. A fire detection system based on a pulse neural network according to claim 3, characterized in that, The fire feature extraction and fusion module fuses high-resolution features from shallow fire feature maps with low-resolution features from deep fire feature maps, and obtains multi-scale fire feature data by stitching the feature maps along the channel dimension; including: In the fire feature extraction module, layer 3 extracts the image pulse sequence data to obtain the first shallow fire feature map; The first shallow fire feature map is sliced ​​into four sub-tensors, with each slice containing a value every pixel. This concentrates the W and H information into the channel space, expanding the input channels by four times. Simultaneously, the second shallow fire feature map is obtained by stitching the sub-tensors along the channel dimensions. The first deep feature map output from layer 4 of the fire feature extraction module is processed by a convolutional layer to obtain the second deep feature map; High-resolution features from the second shallow fire feature map of the Passthrough structure and low-resolution features from the second deep fire feature map are combined to obtain multi-scale fire feature data.

5. A fire detection system based on a pulse neural network according to claim 1, characterized in that, The pulse activation module triggers a fire feature time-series pulse sequence by processing multi-scale fire feature data through the membrane dynamics model of SIF neurons; including: The membrane dynamics model of SIF neurons is constructed using the following formula: ； in: This represents the membrane potential of neuron i in layer 1 at time step t; This represents the total membrane charge of neuron i in layer 1 at time step t; This represents the number of pulses triggered by neuron i in layer 1 at time step t; This represents the timing pulse triggered at time step t in layer 1; Positive threshold and negative threshold respectively ， It is a small negative value (-1e-3); Setting the membrane potential threshold for the membrane dynamics model of SIF neurons Set to 1.0, reset the potential to 0, and when the membrane potential exceeds the threshold, emit a pulse with a value of 1 and reset the membrane potential to 0.

6. A fire detection system based on a pulse neural network according to claim 1, characterized in that, The fire target detection module uses the Spiking Deep-Dilated-Dense YOLOv2 detection head network; the detection head network includes four 3×3 convolutional layers.

7. A fire detection system based on a pulse neural network according to claim 6, characterized in that, The process by which the pulse decoding module obtains the classification and bounding box parameters of fire targets by mapping frequency values ​​to core fire feature data includes: By summing the cumulative pulse count of neurons over T time steps and then dividing by T, we obtain the continuous pulse frequency that reflects the activation intensity of neurons. The frequency values ​​are mapped to core fire feature data, which includes offset, confidence level and category probability, thus completing the conversion from time-series pulse signals to spatial target parameters.

8. A fire detection method based on a spiking neural network, characterized in that, The method employs the system described in any one of claims 1-7 and includes the following steps: Used to read camera video streams or store image frames of videos; Image frames are arranged according to the time dimension to form image pulse sequence data; Convolutional layers with residual structures are used to extract relevant features from image pulse sequence data layer by layer to obtain shallow fire feature maps and deep fire feature maps respectively; High-resolution features from shallow fire feature maps and low-resolution features from deep fire feature maps are fused separately, and multi-scale fire feature data are obtained by stitching the feature maps along the channel dimension. Fire feature time-series pulse sequences are triggered by processing multi-scale fire feature data using a membrane dynamics model of SIF neurons; where: The membrane dynamics model of SIF neurons is constructed using the following formula: in: This represents the membrane potential of neuron i in layer 1 at time step t; This represents the total membrane charge of neuron i in layer 1 at time step t; This represents the number of pulses triggered by neuron i in layer 1 at time step t; This represents the timing pulse triggered at time step t in layer 1; Positive threshold and negative threshold respectively ， It is a small negative value (-1e-3); Setting the membrane potential threshold for the membrane dynamics model of SIF neurons Set to 1.0, reset the potential to 0, and when the membrane potential exceeds the threshold, emit a pulse with a value of 1 and reset the membrane potential to 0; Core fire feature data are obtained by optimizing the temporal pulse sequence of fire features through increasing the depth of the detection head and introducing dilated convolution. The classification and bounding box parameters of fire targets are obtained by mapping the core data of fire characteristics through frequency values. Redundant bounding boxes are suppressed by non-maximum values, and the final fire detection result is output after being filtered by a confidence threshold.

Images