Multi-mode energy storage power station fire detection system based on image and time sequence analysis

Abstract

The invention provides a multi-mode energy storage power station fire detection system based on image and time sequence analysis. The system comprises a feature extraction module, a short-term fire decision module and a long-term fire decision module. Meanwhile, the invention provides a multi-modal energy storage power station fire detection method based on image and time sequence analysis, and the method comprises the following sub-steps: S1, obtaining a video image of a fire region, and extracting a feature vector in each frame of image; s2, making a short-term fire decision; s3, making a long-term fire decision; according to the method provided by the invention, image feature extraction and time sequence analysis technologies are combined, and early detection and early warning of the fire of the energy storage power station are realized through multi-modal fusion; dynamic behaviors of continuous video frames are analyzed through a long short-term memory (LSTM) network, short-term fire decision making and long-term fire decision making are combined, and the accuracy and response speed of fire detection are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multi-modal energy storage power station fire detection system based on image and time series analysis. Background Art

[0002] With the rapid development of renewable energy, energy storage power stations are increasingly used in power systems. However, due to their high energy density and complex electrical equipment, energy storage power stations have certain fire risks. Therefore, timely and accurate detection and early warning of fires are crucial to ensure the safety of energy storage power stations. At present, the traditional fire detection method mainly relies on sensors such as smoke detectors and temperature detectors. However, this method of fire detection relying on sensors has the following disadvantages: Limited detection range: Sensors can only cover a limited area, making it difficult to fully monitor the entire energy storage power station; Susceptible to interference from environmental factors: Smoke, temperature and other sensors are easily affected by environmental factors, resulting in false alarms or missed alarms; Slow response speed: It takes a certain amount of time for the sensor to send out an alarm signal, making it difficult to achieve early warning; In order to overcome the above limitations, fire detection technology based on video image analysis has gradually emerged in recent years. It uses computer vision technology to analyze video images to achieve early detection and early warning of fires. However, the existing video image analysis fire detection method has the following disadvantages: Inaccurate feature extraction: Existing methods mainly rely on manually designed features, which makes it difficult to effectively extract fire features, resulting in low detection accuracy; Lack of consideration of dynamic characteristics of fire: Existing methods mainly focus on the static characteristics of fire and lack consideration of dynamic characteristics of fire. However, the occurrence and development of fire are temporally continuous, which easily leads to insufficient detection capabilities of early fires. Summary of the invention

[0003] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a multi-modal energy storage power station fire detection system based on image and time series analysis.

[0004] To achieve the above object, the present invention adopts the following technical solutions: A multimodal energy storage power station fire detection system based on image and time series analysis, including a feature extraction module, a short-term fire decision module, and a long-term fire decision module; The feature extraction module is used to extract the feature vector of each frame image in the video image; The short-term fire decision module arranges the feature vectors in chronological order to form a time series input sequence, inputs the time series input sequence into the long short-term memory network LSTM, and preliminarily determines whether there is a fire in the video image based on the output result; The long-term fire decision module divides the time input sequence into multiple time windows, performs fire and non-fire judgment in each time window, votes on the output results of each time window, and finally determines whether there is a fire in the video image.

[0005] The present invention also proposes a multi-modal energy storage power station fire detection method based on image and time series analysis, comprising the following sub-steps: S1: Obtain video images of the fire area and extract feature vectors in each frame of the image; Obtain video images of the fire area, use the YOLOv5 target detection model to detect each frame of the acquired video images, and obtain the final feature vector of each frame of the video images through the feature extraction module; The following sub-steps are included: S11: Acquire video images and perform preprocessing; The engineer obtains the video image of the fire area to be predicted in real time, and preprocesses each frame of the video image, including scaling, normalization, etc.; Specifically, each frame of the image is scaled to a specified size using a bilinear interpolation method, and the pixel value of each frame of the image is normalized from the standard RGB range, i.e., 0-255, to 0-1 using a pixel normalization formula, and the pixel normalization formula is as follows: ; in, is the normalized pixel value, and I is the original pixel value; S12: input the image into the target prediction model to generate a feature map corresponding to the image; Each frame of the image preprocessed in step S11 is sequentially input into the YOLOv5 target detection model, and the YOLOv5 target detection model processes the image through a feature extractor through a convolution layer and an activation function to generate a feature map corresponding to the image; The feature map includes width, height, and number of channels; The YOLOv5 target detection model has been trained for flames and smoke in advance; S13: Generate a corresponding weight map according to the feature map; The YOLOv5 target detection model divides the feature map into multiple grid cells and predicts A bounding boxes in each grid cell, where A is a positive integer; The YOLOv5 object detection model predicts the category of the area in the bounding box and the confidence score under this category, that is, the bounding box contains coordinates and corresponding confidence scores; the categories include flames and smoke, and the confidence score corresponds to the possibility of containing flames or smoke in this bounding box, and the confidence score is between 0 and 1; According to the confidence score of each bounding box contained in the feature map, a weight map of the same size as the feature map is generated; each pixel value of the weight map corresponds to the confidence score and category of each grid unit in the feature map; S14: Obtain a weighted feature map; Create a weighted feature map of the same size as the feature map, with an initial value of 0; Traverse each predicted bounding box, determine its corresponding position in the feature map according to the coordinates of the bounding box, and obtain the confidence score of the bounding box; multiply the confidence score of the bounding box by the pixel value of the corresponding area in the feature map, and accumulate it to the corresponding position of the weighted feature map; implement the update of the initial weighted feature map; S15: obtain feature vector; The weighted feature map is weighted average pooled by the following formula to obtain a feature vector; the formula is as follows: ; in, is the feature vector of the i-th channel in the weighted feature map, Point to the bounding box All pixel positions within ; Refers to the position coordinates in the weighted feature map Place The characteristic value of each channel, Finger detection frame Confidence score of And use the normalization factor method to normalize to obtain the final feature vector; The above steps are repeated until the final feature vector is obtained for each frame of the video image.

[0006] S2: Make short-term fire decisions; Set a threshold L in advance based on actual needs; Arrange the feature vectors of each frame of the image obtained in step S1 in time order, and the feature vectors of multiple consecutive frames obtained constitute a time series input sequence; The short-term fire decision module makes short-term fire decisions on the time series input sequence; Specifically, the feature vectors in the time series input sequence are input into the trained long short-term memory network LSTM in sequence, and the long short-term memory network LSTM outputs a fire probability value; if the fire probability value output by the long short-term memory network LSTM is greater than the threshold L, it is preliminarily determined that there is a fire in the video image, otherwise there is no fire.

[0007] S3: Make long-term fire decisions; The long-term fire decision module performs long-term fire decision on the time series input sequence; Specifically, the time input sequence is divided into multiple small time windows, and fire and non-fire judgments are made in each time window; the number of fire decisions and non-fire decisions made in all time windows of the long short-term memory network LSTM are counted; and the long-term fire decision is obtained through majority voting; that is, when the sum of the number of fire decisions in all time windows is greater than the sum of the number of non-fire decisions, it is finally determined that there is a fire in this video image, otherwise there is no fire.

[0008] Compared with the prior art, the present invention has the following beneficial effects: The method proposed in the present invention combines image extraction features and time series analysis technology, and realizes early detection and early warning of energy storage power station fires through multimodal fusion; the flame and smoke in each frame of the image are detected through the YOLOv5 target detection model, and a weighted feature map is generated, and the weighted feature map is weighted average pooled to obtain a feature vector, thereby improving the accuracy of feature extraction; and the problem that it is difficult to effectively extract fire features and result in low detection accuracy in the prior art due to relying on artificially designed features is solved; At the same time, the dynamic behavior of continuous video frames is analyzed through the long short-term memory network LSTM, accurate fire judgment is made in a short time, and the time input sequence is divided into multiple small time windows. The output results of the time windows are voted on to make the final fire decision, combining short-term fire decisions with long-term fire decisions, which significantly improves the accuracy and response speed of fire detection. DETAILED DESCRIPTION

[0009] In order to provide a further understanding of the purpose, structure, features, and functions of the present invention, the following detailed description is given in conjunction with the embodiments.

[0010] A multimodal energy storage power station fire detection system based on image and time series analysis, including a feature extraction module, a short-term fire decision module, and a long-term fire decision module; The feature extraction module is used to extract the feature vector of each frame image in the video image; The short-term fire decision module arranges the feature vectors in chronological order to form a time series input sequence, inputs the time series input sequence into the long short-term memory network LSTM, and preliminarily determines whether there is a fire in the video image based on the output result; The long-term fire decision module divides the time input sequence into multiple time windows, performs fire and non-fire judgment in each time window, votes on the output results of each time window, and finally determines whether there is a fire in the video image.

[0011] The present invention also proposes a multi-modal energy storage power station fire detection method based on image and time series analysis, comprising the following sub-steps: S1: Obtain video images of the fire area and extract feature vectors in each frame of the image; Obtain video images of the fire area, use the YOLOv5 target detection model to detect each frame of the acquired video images, and obtain the final feature vector of each frame of the video images through the feature extraction module; The following sub-steps are included: S11: Acquire video images and perform preprocessing; The engineer obtains the video image of the fire area to be predicted in real time, and preprocesses each frame of the video image, including scaling, normalization, etc.; Specifically, each frame of the image is scaled to a specified size using a bilinear interpolation method, and the pixel value of each frame of the image is normalized from the standard RGB range, i.e., 0-255, to 0-1 using a pixel normalization formula, and the pixel normalization formula is as follows: ; in, is the normalized pixel value, and I is the original pixel value; By performing preprocessing such as scaling and normalization on the video images, the video images are made to meet the size requirements of the YOLOv5 target detection model to avoid errors caused by size mismatch. Through normalization, pixel values ​​in different ranges can be adjusted to a relatively consistent range, which helps to speed up the convergence of the model and improve subsequent training efficiency.

[0012] S12: input the image into the target prediction model to generate a feature map corresponding to the image; Each frame of the image preprocessed in step S11 is sequentially input into the YOLOv5 target detection model, and the YOLOv5 target detection model processes the image through a feature extractor through a convolution layer and an activation function to generate a feature map corresponding to the image; The feature map includes width, height, and number of channels; The YOLOv5 target detection model has been trained for flames and smoke in advance; S13: Generate a corresponding weight map according to the feature map; The YOLOv5 target detection model divides the feature map into multiple grid cells and predicts A bounding boxes in each grid cell, where A is a positive integer; The YOLOv5 object detection model predicts the category of the area in the bounding box and the confidence score under this category, that is, the bounding box contains coordinates and corresponding confidence scores; the categories include flames and smoke, and the confidence score corresponds to the possibility of containing flames or smoke in this bounding box, and the confidence score is between 0 and 1; According to the confidence score of each bounding box contained in the feature map, a weight map of the same size as the feature map is generated; each pixel value of the weight map corresponds to the confidence score and category of each grid unit in the feature map; S14: Obtain a weighted feature map; Create a weighted feature map of the same size as the feature map, with an initial value of 0; Traverse each predicted bounding box, determine its corresponding position in the feature map according to the coordinates of the bounding box, and obtain the confidence score of the bounding box; multiply the confidence score of the bounding box by the pixel value of the corresponding area in the feature map, and accumulate it to the corresponding position of the weighted feature map; implement the update of the initial weighted feature map; S15: obtain feature vector; The weighted feature map is weighted average pooled by the following formula to obtain a feature vector; the formula is as follows: ; in, is the feature vector of the i-th channel in the weighted feature map, Point to the bounding box All pixel positions within ; Refers to the position coordinates in the weighted feature map Place The characteristic value of each channel, Finger detection frame Confidence score of And use the normalization factor method to normalize to obtain the final feature vector; The above steps are repeated until the final feature vector is obtained for each frame of the video image.

[0013] The flame and smoke in the video frame are detected by the YOLOv5 target detection model, and a weighted feature map is generated to provide accurate spatial features for subsequent timing analysis.

[0014] S2: Make short-term fire decisions; Set a threshold L in advance based on actual needs; Arrange the feature vectors of each frame of the image obtained in step S1 in time order, and the feature vectors of multiple consecutive frames obtained constitute a time series input sequence; The short-term fire decision module makes short-term fire decisions on the time series input sequence; Specifically, the feature vectors in the time series input sequence are input into the trained long short-term memory network LSTM in sequence, and the long short-term memory network LSTM outputs a fire probability value; if the fire probability value output by the long short-term memory network LSTM is greater than the threshold L, it is preliminarily determined that there is a fire in the video image, otherwise there is no fire.

[0015] Furthermore, the training method of the long short-term memory network LSTM is as follows: Collect video clips of various fire scenes as fire videos, and video clips containing easily confusing elements as non-fire videos, wherein the confusing elements include smoke, water vapor, clouds, lights, etc.; Set a video clip of n consecutive frames as a sample, where n is a positive integer; Extract corresponding numbers of positive samples and negative samples from fire videos and non-fire videos, respectively, each sample contains n consecutive video frames; The samples are divided into three parts: training, verification, and testing, and the training set, verification set, and test set are obtained; the training set is used to train the long short-term memory network LSTM, the verification set is used to verify the fire detection performance of the long short-term memory network LSTM, and finally the test set is used to evaluate the accuracy of fire detection of the long short-term memory network LSTM; the long short-term memory network LSTM is optimized according to the evaluation results, and the training is completed.

[0016] The present invention collects video clips containing easily confused elements as non-fire videos as negative samples and combines them with positive samples to train the long short-term memory network LSTM, so that the judgment results of the long short-term memory network LSTM for the time input sequence are not easily affected by non-fire factors, thereby solving the problem in the prior art that non-fire factors (such as smoke, water vapor, etc.) are easily misjudged as fires and the false alarm rate is high.

[0017] S3: Make long-term fire decisions; The long-term fire decision module performs long-term fire decision on the time series input sequence; Specifically, the time input sequence is divided into multiple small time windows, and fire and non-fire judgments are made in each time window; the number of fire decisions and non-fire decisions made in all time windows of the long short-term memory network LSTM are counted; and the long-term fire decision is obtained through majority voting; that is, when the sum of the number of fire decisions in all time windows is greater than the sum of the number of non-fire decisions, it is finally determined that there is a fire in this video image, otherwise there is no fire.

[0018] The present invention has been described by the above-mentioned relevant embodiments, however, the above-mentioned embodiments are only examples for implementing the present invention. It must be pointed out that the disclosed embodiments do not limit the scope of the present invention. On the contrary, changes and modifications made without departing from the spirit and scope of the present invention are all within the scope of patent protection of the present invention.

Claims

1. A multimodal energy storage power station fire detection method based on image and time series analysis, characterized in that: The following steps are involved: S1: Obtain video images of the fire area and extract feature vectors in each frame of the image; Obtain video images of the fire area, use the YOLOv5 target detection model to detect each frame of the acquired video images, and obtain the final feature vector of each frame of the video images through the feature extraction module; The following sub-steps are included: S11: Acquire video images and perform preprocessing; S12: input the image into the target prediction model to generate a feature map corresponding to the image; Each frame of the image preprocessed in step S11 is sequentially input into the YOLOv5 target detection model, and the YOLOv5 target detection model processes the image through a feature extractor through a convolution layer and an activation function to generate a feature map corresponding to the image; The feature map includes width, height, and number of channels; The YOLOv5 target detection model has been trained for flames and smoke in advance; S13: Generate a corresponding weight map according to the feature map; S14: Obtain a weighted feature map; Create a weighted feature map of the same size as the feature map, with an initial value of 0; Traverse each predicted bounding box, determine its corresponding position in the feature map according to the coordinates of the bounding box, and obtain the confidence score of the bounding box; multiply the confidence score of the bounding box by the pixel value of the corresponding area in the feature map, and accumulate it to the corresponding position of the weighted feature map; implement the update of the initial weighted feature map; S15: obtain feature vector; S2: Make short-term fire decisions; Set a threshold L in advance based on actual needs; Arrange the feature vectors of each frame of the image obtained in step S1 in time order, and the feature vectors of multiple consecutive frames obtained constitute a time series input sequence; The short-term fire decision module makes short-term fire decisions on the time series input sequence; Specifically, the feature vectors in the time series input sequence are sequentially input into the trained long short-term memory network LSTM, and the long short-term memory network LSTM outputs a fire probability value; if the fire probability value output by the long short-term memory network LSTM is greater than the threshold L, it is preliminarily determined that there is a fire in the video image, otherwise there is no fire; S3: Make long-term fire decisions; The long-term fire decision module performs long-term fire decision on the time series input sequence; Specifically, the time input sequence is divided into multiple small time windows, and fire and non-fire judgments are made in each time window; the number of fire decisions and non-fire decisions made in all time windows of the long short-term memory network LSTM are counted; And through majority voting, long-term fire decisions are obtained; That is, when the sum of the number of fire decisions in all time windows is greater than the sum of the number of non-fire decisions, it is finally determined that there is a fire in this video image, otherwise there is no fire.

2. A multi-modal energy storage power station fire detection method based on image and time series analysis as claimed in claim 1, characterized in that: In step S11, the engineer obtains the video image of the fire area to be predicted in real time, and preprocesses each frame of the video image, wherein the preprocessing includes scaling and normalization; Specifically, each frame of the image is scaled to a specified size using a bilinear interpolation method, and the pixel value of each frame of the image is normalized from the standard RGB range, i.e., 0-255, to 0-1 using a pixel normalization formula, and the pixel normalization formula is as follows: ； in, is the normalized pixel value, and I is the original pixel value.

3. A multi-modal energy storage power station fire detection method based on image and time series analysis as claimed in claim 1, characterized in that: In step S13, the YOLOv5 target detection model divides the feature map into multiple grid cells and predicts A bounding boxes in each grid cell, where A is a positive integer; The YOLOv5 object detection model predicts the category of the area in the bounding box and the confidence score under this category, that is, the bounding box contains coordinates and corresponding confidence scores; the categories include flames and smoke, and the confidence score corresponds to the possibility of containing flames or smoke in this bounding box, and the confidence score is between 0 and 1; According to the confidence score of each bounding box contained in the feature map, a weight map of the same size as the feature map is generated; each pixel value of the weight map corresponds to the confidence score and category of each grid unit in the feature map.

4. A multi-modal energy storage power station fire detection method based on image and time series analysis as claimed in claim 1, characterized in that: In step S15, weighted average pooling is performed on the weighted feature map by the following formula to obtain a feature vector; the formula is as follows: ； in, is the feature vector of the i-th channel in the weighted feature map, Point to the bounding box All pixel positions within ; Refers to the position coordinates in the weighted feature map Place The characteristic value of each channel, Finger detection frame Confidence score of And use the normalization factor method to normalize to obtain the final feature vector; The above steps are repeated until the final feature vector is obtained for each frame of the video image.

5. The multimodal energy storage power station fire detection method based on image and time series analysis as claimed in claim 1, characterized in that: The training method of the long short-term memory network LSTM is as follows: Collect video clips of various fire scenes as fire videos, and video clips containing easily confused elements as non-fire videos; Set a video clip of n consecutive frames as a sample, where n is a positive integer; Extract corresponding numbers of positive samples and negative samples from fire videos and non-fire videos, respectively, each sample contains n consecutive video frames; The samples are divided into three parts: training, verification, and testing, and the training set, verification set, and test set are obtained; the training set is used to train the long short-term memory network LSTM, the verification set is used to verify the fire detection performance of the long short-term memory network LSTM, and finally the test set is used to evaluate the accuracy of fire detection of the long short-term memory network LSTM; the long short-term memory network LSTM is optimized according to the evaluation results, and the training is completed.

6. A multimodal energy storage power station fire detection system based on image and time series analysis, used to implement a multimodal energy storage power station fire detection method based on image and time series analysis as described in any one of claims 1 to 5, characterized in that: It includes feature extraction module, short-term fire decision module and long-term fire decision module; The feature extraction module is used to extract the feature vector of each frame image in the video image; The short-term fire decision module arranges the feature vectors in chronological order to form a time series input sequence, inputs the time series input sequence into the long short-term memory network LSTM, and preliminarily determines whether there is a fire in the video image based on the output result; The long-term fire decision module divides the time input sequence into multiple time windows, performs fire and non-fire judgment in each time window, votes on the output results of each time window, and finally determines whether there is a fire in the video image.

Images