A smoke and flame identification method and device

Abstract

The present invention relates to the field of information recognition technology, and discloses a smoke and flame recognition method and device, which can identify potential smoke areas and potential flame areas, construct a preset color feature template, calculate a color feature vector, determine whether the color feature vector matches the preset color feature template, calculate the motion vector of the pixel point, determine whether there is a continuous motion mode according to the motion vector, and if so, determine it as a reserved sub-area, calculate texture feature parameters, and determine whether it contains features that meet the smoke requirements according to the texture feature parameters and the color feature vector, and construct a sub-area; determine whether the color feature vector and the texture feature parameters are the same, and if so, determine it as a reserved sub-area. Through preprocessing, background difference, color filtering and other steps, potential smoke and flame areas are extracted, and the color features, texture features and motion patterns of these areas are further analyzed, and finally the screened effective areas are used for the training and recognition of deep learning models.

Description

Technical Field

[0001] The present invention relates to the technical field of information identification, and in particular to a smoke and flame identification method and device. Background Art

[0002] With the development of computer vision technology, automatic recognition of smoke and flames has become one of the important research topics in the field of public safety. Traditional smoke and flame recognition methods mostly rely on image processing technology and pattern recognition algorithms, but due to the high complexity and diversity of smoke and flames in natural environments, these traditional methods often fail to meet the needs of practical applications. In recent years, deep learning technology has made remarkable achievements in the field of image recognition due to its powerful feature extraction capabilities and excellent classification performance, and has gradually been applied to smoke and flame recognition.

[0003] Most existing smoke and flame recognition methods use deep learning models to directly identify the original video or image data collected by camera devices. Video or images of the area to be detected are collected through cameras and other devices, and simple preprocessing is performed on them, such as scaling, cropping, color space conversion, etc., to facilitate subsequent deep learning model processing. The collected data is used to build training data sets, which usually contain a large number of positive samples (images containing smoke or flames) and negative samples (images that do not contain smoke or flames). The deep learning model is trained using the constructed data set, and the model learns to extract features related to smoke or flames from the image and classify them accordingly. The new video or image is input into the trained model, and the prediction result is output, that is, whether there is smoke or flame in the image.

[0004] However, existing smoke and flame recognition methods have the following shortcomings:

[0005] Huge data volume: Directly using raw video or image data for training will result in a huge data set size, which not only increases storage costs, but also prolongs model training time and reduces system response speed;

[0006] Low recognition accuracy: Due to the lack of effective data screening and feature extraction, the model is easily disturbed when facing complex backgrounds, and the recognition accuracy is low, especially in environments with large changes in lighting conditions or complex backgrounds.

[0007] High consumption of computing resources: Processing large-scale data sets requires a lot of computing resources. For applications with high real-time requirements, such methods are difficult to meet the needs. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides a smoke and flame recognition method and device to realize the screening of deep learning model data sets.

[0009] The present invention provides a smoke and flame identification method, comprising:

[0010] Step 100, collecting a video of the area to be detected and preprocessing the video, extracting a motion area from a video frame of the preprocessed video using a background difference algorithm, and applying a color filter to the motion area to identify a potential smoke area and a potential flame area;

[0011] Step 200, constructing a preset color feature template, calculating the color feature vector of the potential smoke area and the color feature vector of the potential flame area, judging whether the color feature vector of the potential smoke area matches the preset color feature template, if so, executing step 300, if not, then ending the processing of the potential smoke area; judging whether the color feature vector of the potential flame area matches the preset color feature template, if so, executing step 300, if not, then ending the processing of the potential flame area;

[0012] Step 300, calculating the motion vector of the potential smoke region and the motion vector of the pixel point of the potential flame region, judging whether there is a continuous motion pattern in the potential smoke region according to the motion vector of the pixel point of the potential smoke region, if yes, determining the potential smoke region as a suspected smoke retention sub-region, if no, ending the processing of the potential smoke region; judging whether there is a continuous motion pattern in the potential flame region according to the motion vector of the pixel point of the potential flame region, if yes, determining the potential flame region as a suspected flame retention sub-region; if no, ending the processing of the potential flame region;

[0013] Step 400, respectively calculating the texture feature parameters and color feature vectors for the suspected smoke reserved sub-region and the suspected flame reserved sub-region, and determining whether the suspected smoke reserved sub-region contains features consistent with smoke based on the texture feature parameters and color feature vectors of the suspected smoke reserved sub-region, and if so, executing step 500, and if not, ending the processing of the suspected smoke reserved sub-region; determining whether the suspected flame reserved sub-region contains features consistent with flame based on the texture feature parameters and color feature vectors of the suspected flame reserved sub-region, and if so, executing step 500, and if not, ending the processing of the suspected flame reserved sub-region;

[0014] Step 500, constructing a smoke sub-region and a flame sub-region at the edges of the suspected smoke retention sub-region and the suspected flame retention sub-region, respectively, and calculating the color feature vector and texture feature parameters of the smoke sub-region and the flame sub-region respectively; judging whether the color feature vector and texture feature parameters of the smoke sub-region are the same as those of the suspected smoke retention sub-region, if so, determining the smoke sub-region as the smoke retention sub-region; if not, ending the processing of the smoke sub-region; judging whether the color feature vector and texture feature parameters of the flame sub-region are the same as those of the suspected flame retention sub-region, if so, determining the flame sub-region as the flame retention sub-region; if not, ending the processing of the flame sub-region;

[0015] In step 600, the suspected smoke retention sub-region and the suspected flame retention sub-region determined in step 300, and the smoke retention sub-region and the flame retention sub-region determined in step 500 are combined into a data set of a deep learning model for identifying flames and smoke.

[0016] Furthermore, the determination of the continuous motion mode specifically includes the following steps:

[0017] Step 310, using the Sobel operator to calculate the gradient of the video frame;

[0018] Step 320, calculating a temporal gradient according to two adjacent video frames;

[0019] Step 330, constructing an optical flow algorithm according to the gradient and time gradient of the video frame to calculate the motion vector of the pixel;

[0020] Step 340, pairing two adjacent video frames to form a pair of adjacent frames, and obtaining a motion vector between each pair of adjacent frames according to the motion vector of the pixel point;

[0021] Step 350, calculating the average motion vector between multiple groups of adjacent frames;

[0022] Step 360, calculating a motion mode index value according to the motion vector and the average motion vector between each pair of adjacent frames;

[0023] Step 370: determine whether there is a continuous motion pattern in the potential smoke area and the potential flame area according to the motion pattern index value.

[0024] Furthermore, judging whether there is a continuous motion pattern in the potential smoke area and the potential flame area according to the motion pattern index value specifically includes:

[0025] Step 371, constructing a first threshold and a second threshold;

[0026] Step 372, calculating the direction angle of the motion vector between each pair of adjacent frames;

[0027] Step 373, calculating the average direction angle according to the direction angle;

[0028] Step 374, calculating the direction angle standard deviation according to the direction angle and the average direction angle;

[0029] Step 375, compare the motion mode index value with the first threshold, and compare the azimuth standard deviation with the second threshold. When the motion mode index value is greater than the first threshold, and the azimuth standard deviation is greater than the second threshold, a continuous motion mode exists in the smoke area and the flame area; when the motion mode index value is less than or equal to the first threshold, or the azimuth standard deviation is less than or equal to the second threshold, a discontinuous motion mode exists in the smoke area and the flame area.

[0030] Furthermore, the color feature vector of the suspected smoke retention sub-region or the color feature vector of the suspected flame retention sub-region is calculated by the first formula:

[0031] ;

[0032] represents the color feature vector of the suspected smoke reserved sub-region or the color feature vector of the suspected flame reserved sub-region, R, G, B represent the red, green, blue intensity values ​​of the pixel points of the suspected smoke reserved sub-region in the video frame, or the red, green, blue intensity values ​​of the pixel points of the suspected flame reserved sub-region in the video frame, , , Represent the weight coefficients respectively.

[0033] Furthermore, the color feature vector of the smoke sub-region or the color feature vector of the flame sub-region is calculated by the first formula.

[0034] Furthermore, the calculation formula of the time gradient is:

[0035] ;

[0036] In the formula, represents the temporal gradient between the t-th video frame and the t+1-th video frame, Indicates location and the video frame intensity value at the t-th frame, Represents in the same location and the video frame intensity value of the t+1th frame.

[0037] Furthermore, the relation of the optical flow algorithm is:

[0038] ;

[0039] In the formula, , Represent the gradient of the video frame respectively. Specifically, Represents the gradient of the video frame in the horizontal direction, represents the vertical gradient of the video frame, (u, v) represents the motion vector of the pixel, Represents the temporal gradient between the t-th video frame and the t+1-th video frame.

[0040] Furthermore, the calculation formula of the motion mode index value is:

[0041] ;

[0042] In the formula, represents the motion mode index value, T represents the maximum number of adjacent frames, Represents the pixel motion vector between the tth frame and the t+1th frame, Represents the average motion vector.

[0043] The present invention also provides a smoke and flame identification device, which is used to execute the above-mentioned smoke and flame identification method, and includes the following modules:

[0044] Potential Area Identification Module: used to apply color filters on motion areas to identify potential smoke areas and potential fire areas;

[0045] Color feature vector matching module: connected to the potential area recognition module, used to construct a preset color feature template, calculate the color feature vector of the potential smoke area and the color feature vector of the potential flame area, determine whether the color feature vector of the potential smoke area matches the preset color feature template, if so, execute step 300, if not, end the processing of the potential smoke area; determine whether the color feature vector of the potential flame area matches the preset color feature template, if so, execute step 300, if not, end the processing of the potential flame area;

[0046] Suspected reserved sub-region determination module: connected to the color feature vector matching module, used to calculate the motion vector of the potential smoke region and the motion vector of the pixel point of the potential flame region, and judge whether there is a continuous motion pattern in the potential smoke region according to the motion vector of the pixel point of the potential smoke region, if so, the potential smoke region is determined as a suspected smoke reserved sub-region, if not, the processing of the potential smoke region is terminated; according to the motion vector of the pixel point of the potential flame region, judge whether there is a continuous motion pattern in the potential flame region, if so, the potential flame region is determined as a suspected flame reserved sub-region; if not, the processing of the potential flame region is terminated;

[0047] A module for determining features that meet the requirements: connected to the module for determining suspected reserved sub-regions, and used to calculate the texture feature parameters and color feature vectors for the suspected smoke reserved sub-region and the suspected flame reserved sub-region, respectively, and determine whether the suspected smoke reserved sub-region contains features that meet the requirements of smoke based on the texture feature parameters and color feature vectors of the suspected smoke reserved sub-region. If so, the suspected smoke reserved sub-region is processed; if not, the processing of the suspected smoke reserved sub-region is terminated; and according to the texture feature parameters and color feature vectors of the suspected flame reserved sub-region, determine whether the suspected flame reserved sub-region contains features that meet the requirements of flame. If so, the suspected flame reserved sub-region is processed; if not, the processing of the suspected flame reserved sub-region is terminated;

[0048] The reserved sub-region determination module is connected to the feature determination module, and is used to construct a smoke sub-region and a flame sub-region at the edges of the suspected smoke reserved sub-region and the suspected flame reserved sub-region, respectively, and calculate the color feature vector and texture feature parameters of the smoke sub-region and the flame sub-region, respectively; determine whether the color feature vector and texture feature parameters of the smoke sub-region are the same as those of the suspected smoke reserved sub-region, and if so, determine the smoke sub-region as the smoke reserved sub-region; if not, end the processing of the smoke sub-region; determine whether the color feature vector and texture feature parameters of the flame sub-region are the same as those of the suspected flame reserved sub-region, and if so, determine the flame sub-region as the flame reserved sub-region; if not, end the processing of the flame sub-region;

[0049] Identification module: connected to the reserved sub-region determination module and the suspected reserved sub-region determination module, used to form a data set of a deep learning model with suspected smoke reserved sub-regions, suspected flame reserved sub-regions, smoke reserved sub-regions, and flame reserved sub-regions to identify flames and smoke.

[0050] The embodiments of the present invention have the following technical effects:

[0051] By gradually screening out suspected smoke retention sub-regions, suspected flame retention sub-regions, smoke retention sub-regions, and flame retention sub-regions as data sets for deep learning models, the aim is to reduce the amount of data, improve recognition accuracy, and optimize the use of computing resources. This method extracts potential smoke and flame areas by preprocessing video frames, background subtraction, color filtering, and other steps, and further analyzes the color features, texture features, and motion patterns of these areas. Finally, the screened effective areas are used for deep learning model training and recognition to improve the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 is a flow chart of a smoke and flame identification method provided by an embodiment of the present invention;

[0054] Figure 2 It is a structural schematic diagram of a smoke and flame identification method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0056] Figure 1 is a flow chart of a smoke and flame identification method provided by an embodiment of the present invention. Figure 1 , including:

[0057] Step 100, collect the video of the area to be detected and preprocess the video, use the background difference algorithm to extract the motion area from the video frame of the preprocessed video, and apply a color filter to the motion area to identify the potential smoke area and the potential flame area.

[0058] This embodiment uses a camera device to record a video of the area to be detected, extracts the video from the camera device, reads one frame each time in a loop, and converts it into a grayscale image (preprocessing) in order to perform a background difference algorithm. Create a mixed Gaussian background subtractor object, apply the background subtractor to each frame to obtain a foreground mask, remove noise through threshold segmentation and morphological operations, and find the contour of the motion area. For each detected motion area, convert the image in the area to the HSV color space. Create a mask for smoke and flames based on the defined color range. If there are non-zero pixels in the mask, it is considered that there is smoke or flame in the area, and the corresponding rectangular box and label are drawn.

[0059] The background difference algorithm and HSV color space in this step are both implemented using conventional techniques.

[0060] Step 200, construct a preset color feature template, calculate the color feature vector of the potential smoke area and the color feature vector of the potential flame area, determine whether the color feature vector of the potential smoke area matches the preset color feature template, if so, execute step 300, if not, end processing the potential smoke area; determine whether the color feature vector of the potential flame area matches the preset color feature template, if so, execute step 300, if not, end processing the potential flame area.

[0061] The preset color feature template is constructed based on the known color characteristics of smoke and flame. In the HSV color space, smoke is usually represented by lower saturation (S) and higher brightness (V), while flame is represented by a specific hue (H) range, usually concentrated between red and orange. This embodiment represents smoke and flame by HSV range. The HSV range of smoke: [0, 0, 100]-[180, 50, 255]; the HSV range of flame: [160, 100, 100]-[180, 255, 255].

[0062] The color feature vector is realized by calculating the average HSV value of each potential area. If the color feature vector of the potential smoke area is within the HSV range of the smoke, the present embodiment considers that the color feature vector of the potential smoke area matches the preset color feature template, otherwise, it is not matched. If the color feature vector of the potential flame area is within the HSV range of the flame, the present embodiment considers that the color feature vector of the potential flame area matches the preset color feature template, otherwise, it is not matched.

[0063] Step 300, calculate the motion vector of the potential smoke area and the motion vector of the pixel points of the potential flame area, and determine whether there is a continuous motion pattern in the potential smoke area based on the motion vector of the pixel points of the potential smoke area. If so, the potential smoke area is determined as a suspected smoke retention sub-area, and if not, the processing of the potential smoke area is terminated; based on the motion vector of the pixel points of the potential flame area, determine whether there is a continuous motion pattern in the potential flame area, and if so, the potential flame area is determined as a suspected flame retention sub-area; if not, the processing of the potential flame area is terminated.

[0064] First, the motion vectors of the pixels in the potential smoke area and the potential flame area are calculated, and the continuous motion mode is judged based on the motion vectors, which specifically includes the following steps:

[0065] Step 310: Calculate the gradient of the video frame using the Sobel operator.

[0066] Gradients include gradients in the horizontal direction and the vertical gradient ;

[0067] ;

[0068] ;

[0069] Indicates location and the video frame intensity value at the t-th frame.

[0070] By using the Sobel operator to calculate the gradient of the video frame, the edge information in the video frame can be captured. By calculating the gradient in the horizontal and vertical directions, the contour and edge information of the object in the video frame can be obtained, which is helpful for subsequent feature extraction.

[0071] Step 320: Calculate the temporal gradient according to two adjacent video frames.

[0072] The time gradient is obtained by using the difference between two adjacent frames of images. The calculation formula for the time gradient is:

[0073] ;

[0074] In the formula, represents the temporal gradient between the t-th video frame and the t+1-th video frame, Indicates location and the video frame intensity value at the t-th frame, Represents in the same location and the video frame intensity value of the t+1th frame (the next frame).

[0075] By calculating the temporal gradient between two adjacent frames, we can capture the changes of objects in the video over time. The temporal gradient can reflect the speed and direction of the object moving in the video frame, which is particularly useful for detecting fast-moving objects.

[0076] Step 330, construct an optical flow algorithm according to the gradient and time gradient of the video frame to calculate the motion vector of the pixel.

[0077] The relationship of the optical flow algorithm is:

[0078] ;

[0079] In the formula, , Represent the gradient of the video frame respectively. Specifically, Represents the gradient of the video frame in the horizontal direction, represents the vertical gradient of the video frame, (u, v) represents the motion vector of the pixel point. Specifically, u represents the horizontal motion speed of the pixel point, and v represents the vertical motion speed of the pixel point. represents the time gradient.

[0080] In this relational expression, the present embodiment calculates the motion vector of the pixel point by the least square method, and converts the relational expression into the standard expression of the least square method:

[0081] ;

[0082] According to the standard formula, the motion vector (u, v) of the pixel is calculated.

[0083] The optical flow algorithm can estimate the displacement of each pixel between two frames, thereby capturing the motion information in the video.

[0084] Step 340: pair two adjacent video frames to form a pair of adjacent frames, and obtain a motion vector between each pair of adjacent frames based on the motion vector of the pixel point.

[0085] Exemplarily, this embodiment obtains the horizontal and vertical movement speeds of the pixels between the tth frame and the t+1th frame. for:

[0086] ;

[0087] Represents the horizontal movement speed of the pixel between the tth frame and the t+1th frame. Represents the vertical movement speed of the pixel between the tth frame and the t+1th frame.

[0088] Two adjacent video frames are paired to form a pair of adjacent frames, and the motion vector between each pair of adjacent frames is obtained based on the motion vector of the pixel point, so that the motion information can be accumulated and the accuracy of detection can be improved.

[0089] Step 350, calculate the average motion vector between multiple groups of adjacent frames .

[0090] .

[0091] By calculating the average motion vector between multiple groups of adjacent frames and counting the motion vectors of multiple frames, the accidental error caused by a single frame can be reduced, making the detection result more stable.

[0092] Step 360, calculating the motion mode index value according to the motion vector and the average motion vector between each pair of adjacent frames.

[0093] The calculation formula of the sports mode index value is:

[0094] ;

[0095] In the formula, represents the motion mode index value, T represents the maximum number of adjacent frames, Represents the average motion vector.

[0096] The motion mode index value is calculated based on the motion vector and the average motion vector between each pair of adjacent frames. This step quantifies the motion information into a specific index value to facilitate subsequent judgment.

[0097] Step 370: determine whether there is a continuous motion pattern in the potential smoke area and the potential flame area according to the motion pattern index value.

[0098] The motion pattern index value is used to determine whether there is a continuous motion pattern in the potential smoke area and the potential flame area. In this way, the area with continuous motion characteristics can be effectively identified, thereby improving the reliability of detection.

[0099] Step 371, constructing a first threshold and a second threshold.

[0100] The first threshold is used to determine whether the motion pattern index value indicates significant continuous motion. The second threshold is used to determine whether the standard deviation of the direction angle indicates consistency of direction, thereby determining whether there is continuous motion.

[0101] By collecting a large number of video samples with real smoke and flames and observing the distribution of their motion pattern index values, a suitable first threshold is found. Similarly, the second threshold of the standard deviation of the direction angle is determined through experimental tests. If the standard deviation is large, it indicates that the direction is more dispersed and there is discontinuous motion; conversely, it indicates that the direction is more consistent and there is continuous motion. Preferably, the first threshold is 15 and the second threshold is 25.

[0102] Step 372, calculating the direction angle of the motion vector between each pair of adjacent frames.

[0103] Use the atan2 function to calculate the direction angle of each motion vector:

[0104] ;

[0105] Represents the direction angle of the pixel motion vector between the tth frame and the t+1th frame.

[0106] Step 373, calculating the average direction angle according to the direction angle.

[0107] Step 374, calculating the standard deviation of the azimuth angle based on the azimuth angle and the average azimuth angle .

[0108] .

[0109] Step 375, compare the motion mode index value with the first threshold, and compare the azimuth standard deviation with the second threshold. When the motion mode index value is greater than the first threshold, and the azimuth standard deviation is greater than the second threshold, a continuous motion mode exists in the smoke area and the flame area; when the motion mode index value is less than or equal to the first threshold, or the azimuth standard deviation is less than or equal to the second threshold, a discontinuous motion mode exists in the smoke area and the flame area.

[0110] By calculating the motion vector and its direction angle between each pair of adjacent frames, and combining the motion pattern index value and the standard deviation of the direction angle for comprehensive judgment, it is possible to more accurately identify whether there is a continuous motion pattern in the potential smoke area and the potential flame area. This method can reduce the probability of false alarms and missed alarms, improve anti-interference capabilities, effectively filter out irregular motion caused by background noise or other interference factors, and more accurately identify the true motion state of the potential smoke and flame areas.

[0111] Step 400, respectively calculate the texture feature parameters and color feature vectors for the suspected smoke retention sub-region and the suspected flame retention sub-region, and determine whether the suspected smoke retention sub-region contains features consistent with smoke based on the texture feature parameters and color feature vectors of the suspected smoke retention sub-region. If so, execute step 500; if not, end the processing of the suspected smoke retention sub-region; determine whether the suspected flame retention sub-region contains features consistent with flame based on the texture feature parameters and color feature vectors of the suspected flame retention sub-region. If so, execute step 500; if not, end the processing of the suspected flame retention sub-region.

[0112] Texture feature parameters can help distinguish different types of texture patterns. Common texture feature parameters include gray level co-occurrence matrix, local binary pattern, wavelet transform, etc. In this embodiment, gray level co-occurrence matrix is ​​preferably used to calculate texture feature parameters.

[0113] The color feature vector of the suspected smoke retention sub-region or the color feature vector of the suspected flame retention sub-region is calculated by the first formula:

[0114] ;

[0115] represents the color feature vector of the suspected reserved sub-region (the color feature vector of the suspected smoke reserved sub-region or the color feature vector of the suspected flame reserved sub-region are calculated by the first formula), R, G, B represent the red, green, blue intensity values ​​of the pixel points of the suspected smoke reserved sub-region in the video frame, or the red, green, blue intensity values ​​of the pixel points of the suspected flame reserved sub-region in the video frame, , , Represent the weight coefficients respectively.

[0116] Since the color feature vectors of the suspected smoke retention sub-region and the suspected flame retention sub-region are both calculated by the first formula, when calculating the color feature vectors of the corresponding suspected sub-regions, the red, green, and blue intensity values ​​of the pixels in the video frame corresponding to the suspected sub-regions are respectively substituted, and the weight coefficients are also obtained based on the red, green, and blue intensity values ​​of the pixels in the corresponding video frame.

[0117] , , The corresponding weight coefficient is obtained by calculation:

[0118] ;

[0119] ;

[0120] ;

[0121] In the formula, represents the calculation of covariance, Represents the calculated variance; F represents the RGB value after converting the HSV range in the preset color feature template.

[0122] Converting the HSV range in the preset color feature template into RGB values ​​is a conventional technical means.

[0123] Construct a preset texture feature template to determine whether the texture feature parameters of the suspected smoke retention sub-region meet the typical texture features of smoke. The texture features and their preset ranges are:

[0124] Contrast: reflects the difference between adjacent pixels in a video frame. Smoke usually has a lower contrast because its edges are blurred, and the contrast is between 0 and 0.1.

[0125] Energy: reflects the uniformity of grayscale in the video frame. Smoke usually has higher energy because its grayscale is more uniform. The energy of smoke is between 0.7 and 1.

[0126] Homogeneity: reflects the similarity between adjacent pixels in a video frame. Smoke usually has a higher homogeneity because its texture is more uniform. The homogeneity of smoke is between 0.7 and 1.

[0127] Entropy: reflects the randomness of the grayscale distribution in the video frame. Smoke usually has a lower entropy because its grayscale distribution is more concentrated. The entropy of smoke is between 0 and 1.

[0128] The determination of whether the suspected smoke retention sub-region contains features consistent with smoke mainly depends on whether the texture feature parameters of the suspected smoke retention sub-region are all within the preset range of the preset texture feature template. If so, step 500 is executed; if not, the processing of the suspected smoke retention sub-region is terminated.

[0129] Whether the suspected flame retention sub-region contains features consistent with flames mainly depends on whether the color feature vector of the suspected flame retention sub-region is within the HSV range of the flame in the preset color feature template. If so, step 500 is executed, and if not, the processing of the suspected smoke retention sub-region is terminated.

[0130] Step 500, constructing a smoke sub-region and a flame sub-region at the edges of the suspected smoke retention sub-region and the suspected flame retention sub-region, respectively, and calculating the color feature vector and texture feature parameters for the smoke sub-region and the flame sub-region, respectively; judging whether the color feature vector and texture feature parameters of the smoke sub-region are the same as those of the suspected smoke retention sub-region, if so, determining the smoke sub-region as a smoke retention sub-region; if not, ending the processing of the smoke sub-region; judging whether the color feature vector and texture feature parameters of the flame sub-region are the same as those of the suspected flame retention sub-region, if so, determining the flame sub-region as a flame retention sub-region; if not, ending the processing of the flame sub-region.

[0131] First, determine the location and range of the suspected smoke retention sub-region or the suspected flame retention sub-region, and expand a certain number of pixels outward from the region as the center to construct a larger rectangular region as the sub-region. The number of this expansion can be fixed or dynamically adjusted based on the size of the suspected region itself. Then calculate the color feature vector and texture feature parameters of the smoke sub-region and the flame sub-region. Use the Euclidean distance to compare the differences between the color feature vector and texture feature parameters of the sub-region and the color feature vector and texture feature parameters of the suspected retention sub-region.

[0132] When using Euclidean distance to measure the difference, the purpose is to evaluate the closeness of the color feature vectors of the sub-region and the suspected retained sub-region, and the closeness of the texture feature parameters of the sub-region and the suspected retained sub-region. Euclidean distance is an intuitive distance measurement method. Simply calculating the distance is not enough to directly determine whether two color feature vectors are similar or two texture feature parameters are similar. This embodiment uses a planning method to find the optimal solution, thereby obtaining the optimal distance, that is, obtaining a distance interval [D min , D max ], D min Represents the lower limit of the distance interval, D maxRepresents the upper limit of the distance interval. The color feature vectors within this interval are considered to be similar, and the texture feature parameters within this interval are also considered to be similar. Based on the above principle, the sub-region and the suspected reserved sub-region are analyzed to see whether the color feature vectors and texture feature parameters are similar. In this embodiment, the upper limit of the distance interval is used as the preset threshold D max When the error between the color feature vectors of the sub-region and the suspected retained sub-region is less than or equal to a preset threshold, this embodiment considers that the two vectors are the same; when the error between the color feature vectors of the sub-region and the suspected retained sub-region is greater than the preset threshold, this embodiment considers that the two vectors are different.

[0133] In this embodiment, preferably, if the color feature vector of the smoke sub-region is the same as the color feature vector of the suspected smoke reserved sub-region, and the texture feature parameter of the smoke sub-region is the same as the texture feature parameter of the suspected smoke reserved sub-region, the smoke sub-region is determined as the smoke reserved sub-region, otherwise, the processing of the smoke reserved sub-region is terminated. The process of determining the flame sub-region as the smoke reserved sub-region is the same as the above content.

[0134] The color feature vector of the smoke sub-region or the color feature vector of the flame sub-region is calculated by the first formula. During the calculation process, the red, green, and blue intensity values ​​of the pixels in the video frame corresponding to the sub-region are respectively input, and the weight coefficient is also obtained based on the red, green, and blue intensity values ​​of the pixels in the corresponding video frame.

[0135] This embodiment gradually screens out suspected smoke retention sub-regions, suspected flame retention sub-regions, smoke retention sub-regions, and flame retention sub-regions from potential smoke regions and potential flame regions. The color feature vectors in the screened regions are calculated by the first formula. These regions are further refined from the potential regions screened out initially to obtain key monitoring regions. In these regions, there may be problems such as complex colors and large changes in illumination. The first formula captures color features more accurately, especially in the case of complex or changeable colors, better copes with illumination changes and other interference factors, and improves the ability to distinguish.

[0136] In step 600, the suspected smoke retention sub-region and the suspected flame retention sub-region determined in step 300, and the smoke retention sub-region and the flame retention sub-region determined in step 500 are combined into a data set of a deep learning model for identifying flames and smoke.

[0137] Acquiring training data from the suspected smoke retention sub-region and the suspected flame retention sub-region determined in step 300 and the smoke retention sub-region and the flame retention sub-region determined in step 500 mainly includes:

[0138] Color feature vector: color features of each region, such as the mean and standard deviation of RGB values;

[0139] Texture feature parameters: texture features of each region, such as contrast and energy of gray-level co-occurrence matrix (GLCM);

[0140] Position information: the position of each area in the image, such as the coordinate range;

[0141] Size information: the size of each area, such as width and height;

[0142] Label information: Label according to the type of area (smoke / fire).

[0143] The above training data are merged to obtain a data set, which is then divided into training set, validation set, and test set.

[0144] Finally, the deep learning model is trained using the training set to obtain a trained deep learning model to identify smoke and flames. The deep learning model is preferably an improved YOLOV5 deep convolutional neural network model to detect whether there is flame and smoke in the video and realize image target recognition. Improving the YOLOV5 deep convolutional neural network to identify flames and smoke is a conventional technical means, which will not be described in detail in this embodiment.

[0145] This embodiment gradually selects suspected smoke retention sub-areas, suspected flame retention sub-areas, smoke retention sub-areas, and flame retention sub-areas from potential smoke areas and potential flame areas as data sets for deep learning models, removes a large amount of useless or irrelevant information, and only retains areas related to smoke or flames. This can significantly reduce the amount of data input into the deep learning model and reduce the possibility of model misjudgment. By reducing the interference of background information, the characteristics of smoke and flames can be made more obvious. When facing complex backgrounds, the model can better ignore interference and focus on the target area, which is convenient for model recognition.

[0146] This embodiment also discloses a smoke and flame identification device, comprising the following modules:

[0147] Potential Area Identification Module: used to apply color filters on motion areas to identify potential smoke areas and potential fire areas;

[0148] Color feature vector matching module: connected to the potential area recognition module, used to construct a preset color feature template, calculate the color feature vector of the potential smoke area and the color feature vector of the potential flame area, determine whether the color feature vector of the potential smoke area matches the preset color feature template, if so, execute step 300, if not, end the processing of the potential smoke area; determine whether the color feature vector of the potential flame area matches the preset color feature template, if so, execute step 300, if not, end the processing of the potential flame area;

[0149] Suspected reserved sub-region determination module: connected to the color feature vector matching module, used to calculate the motion vector of the potential smoke region and the motion vector of the pixel point of the potential flame region, and judge whether there is a continuous motion pattern in the potential smoke region according to the motion vector of the pixel point of the potential smoke region, if so, the potential smoke region is determined as a suspected smoke reserved sub-region, if not, the processing of the potential smoke region is terminated; according to the motion vector of the pixel point of the potential flame region, judge whether there is a continuous motion pattern in the potential flame region, if so, the potential flame region is determined as a suspected flame reserved sub-region; if not, the processing of the potential flame region is terminated;

[0150] A module for determining features that meet the requirements: connected to the module for determining suspected reserved sub-regions, and used to calculate the texture feature parameters and color feature vectors for the suspected smoke reserved sub-region and the suspected flame reserved sub-region, respectively, and determine whether the suspected smoke reserved sub-region contains features that meet the requirements of smoke based on the texture feature parameters and color feature vectors of the suspected smoke reserved sub-region. If so, the suspected smoke reserved sub-region is processed; if not, the processing of the suspected smoke reserved sub-region is terminated; and according to the texture feature parameters and color feature vectors of the suspected flame reserved sub-region, determine whether the suspected flame reserved sub-region contains features that meet the requirements of flame. If so, the suspected flame reserved sub-region is processed; if not, the processing of the suspected flame reserved sub-region is terminated;

[0151] The reserved sub-region determination module is connected to the feature determination module, and is used to construct a smoke sub-region and a flame sub-region at the edges of the suspected smoke reserved sub-region and the suspected flame reserved sub-region, respectively, and calculate the color feature vector and texture feature parameters of the smoke sub-region and the flame sub-region, respectively; determine whether the color feature vector and texture feature parameters of the smoke sub-region are the same as those of the suspected smoke reserved sub-region, and if so, determine the smoke sub-region as the smoke reserved sub-region; if not, end the processing of the smoke sub-region; determine whether the color feature vector and texture feature parameters of the flame sub-region are the same as those of the suspected flame reserved sub-region, and if so, determine the flame sub-region as the flame reserved sub-region; if not, end the processing of the flame sub-region;

[0152] Identification module: connected to the reserved sub-region determination module and the suspected reserved sub-region determination module, used to form a data set of a deep learning model with suspected smoke reserved sub-regions, suspected flame reserved sub-regions, smoke reserved sub-regions, and flame reserved sub-regions to identify flames and smoke.

[0153] This embodiment also discloses an electronic device, including one or more processors and a memory.

[0154] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0155] The memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may run the program instructions to implement a smoke and flame identification method and / or other desired functions of any embodiment of the present application described above. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage medium.

[0156] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown). The input device may include, for example, a keyboard, a mouse, etc. The output device may output various information to the outside, including early warning information, braking force, etc. The output device may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0157] In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0158] In addition to the above methods and devices, the embodiments of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of a smoke and flame identification method provided by any embodiment of the present application.

[0159] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0160] In addition, the embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of a smoke and flame identification method provided by any embodiment of the present application.

[0161] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0162] It should be noted that the terms used in the present invention are only for describing specific embodiments, rather than limiting the scope of the present application. As shown in the present specification, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of more restrictions, the elements defined by the sentence "include one..." do not exclude the presence of other identical elements in the process, method or device including the elements.

[0163] It should also be noted that the orientations or positional relationships indicated by the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc. are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A smoke and flame identification method, characterized in that: include: Step 100, collecting a video of the area to be detected and preprocessing the video, extracting a motion area from a video frame of the preprocessed video using a background difference algorithm, and applying a color filter to the motion area to identify a potential smoke area and a potential flame area; Step 200, constructing a preset color feature template, calculating the color feature vector of the potential smoke area and the color feature vector of the potential flame area, judging whether the color feature vector of the potential smoke area matches the preset color feature template, if so, executing step 300, if not, then ending the processing of the potential smoke area; judging whether the color feature vector of the potential flame area matches the preset color feature template, if so, executing step 300, if not, then ending the processing of the potential flame area; Step 300, calculating the motion vector of the potential smoke region and the motion vector of the pixel point of the potential flame region, judging whether there is a continuous motion pattern in the potential smoke region according to the motion vector of the pixel point of the potential smoke region, if yes, determining the potential smoke region as a suspected smoke retention sub-region, if no, ending the processing of the potential smoke region; judging whether there is a continuous motion pattern in the potential flame region according to the motion vector of the pixel point of the potential flame region, if yes, determining the potential flame region as a suspected flame retention sub-region; if no, ending the processing of the potential flame region; Step 310, using the Sobel operator to calculate the gradient of the video frame; Step 320, calculating a temporal gradient according to two adjacent video frames; Step 330, constructing an optical flow algorithm according to the gradient and time gradient of the video frame to calculate the motion vector of the pixel; Step 340, pairing two adjacent video frames to form a pair of adjacent frames, and obtaining a motion vector between each pair of adjacent frames according to the motion vector of the pixel point; Step 350, calculating the average motion vector between multiple groups of adjacent frames; Step 360, calculating a motion mode index value according to the motion vector and the average motion vector between each pair of adjacent frames; Step 370, judging whether there is a continuous motion pattern in the potential smoke area and the potential flame area according to the motion pattern index value; Step 371, constructing a first threshold and a second threshold; Step 372, calculating the direction angle of the motion vector between each pair of adjacent frames; Step 373, calculating the average direction angle according to the direction angle; Step 374, calculating the direction angle standard deviation according to the direction angle and the average direction angle; Step 375, comparing the motion mode index value with the first threshold value, and comparing the azimuth standard deviation with the second threshold value. When the motion mode index value is greater than the first threshold value, and the azimuth standard deviation is greater than the second threshold value, a continuous motion mode exists in the smoke region and the flame region; when the motion mode index value is less than or equal to the first threshold value, or the azimuth standard deviation is less than or equal to the second threshold value, a discontinuous motion mode exists in the smoke region and the flame region; Step 400, respectively calculating the texture feature parameters and color feature vectors for the suspected smoke reserved sub-region and the suspected flame reserved sub-region, and determining whether the suspected smoke reserved sub-region contains features consistent with smoke based on the texture feature parameters and color feature vectors of the suspected smoke reserved sub-region, and if so, executing step 500, and if not, ending the processing of the suspected smoke reserved sub-region; determining whether the suspected flame reserved sub-region contains features consistent with flame based on the texture feature parameters and color feature vectors of the suspected flame reserved sub-region, and if so, executing step 500, and if not, ending the processing of the suspected flame reserved sub-region; The color feature vector of the suspected smoke retention sub-region or the color feature vector of the suspected flame retention sub-region is calculated by the first formula: ； represents the color feature vector of the suspected smoke reserved sub-region or the color feature vector of the suspected flame reserved sub-region, R, G, B represent the red, green, blue intensity values ​​of the pixel points of the suspected smoke reserved sub-region in the video frame, or the red, green, blue intensity values ​​of the pixel points of the suspected flame reserved sub-region in the video frame, , , Represent weight coefficients respectively; , , The corresponding weight coefficient is obtained by calculation: ； ； ； In the formula, represents the calculation of covariance, represents the calculated variance; F represents the RGB value after converting the HSV range in the preset color feature template; Step 500, constructing a smoke sub-region and a flame sub-region at the edges of the suspected smoke retention sub-region and the suspected flame retention sub-region, respectively, and calculating the color feature vector and texture feature parameters of the smoke sub-region and the flame sub-region respectively; judging whether the color feature vector and texture feature parameters of the smoke sub-region are the same as those of the suspected smoke retention sub-region, if so, determining the smoke sub-region as the smoke retention sub-region; if not, ending the processing of the smoke sub-region; judging whether the color feature vector and texture feature parameters of the flame sub-region are the same as those of the suspected flame retention sub-region, if so, determining the flame sub-region as the flame retention sub-region; if not, ending the processing of the flame sub-region; In step 600, the suspected smoke retention sub-region and the suspected flame retention sub-region determined in step 300, and the smoke retention sub-region and the flame retention sub-region determined in step 500 are combined into a data set of a deep learning model for identifying flames and smoke.

2. A smoke and flame identification method according to claim 1, characterized in that: The calculation formula of time gradient is: ； In the formula, represents the temporal gradient between the t-th video frame and the t+1-th video frame, Indicates location and the video frame intensity value at the t-th frame, Represents in the same location and the video frame intensity value of the t+1th frame.

3. A smoke and flame identification method according to claim 1, characterized in that: The relationship of the optical flow algorithm is: ； In the formula, , Represent the gradient of the video frame respectively. Specifically, Represents the gradient of the video frame in the horizontal direction, represents the vertical gradient of the video frame, (u, v) represents the motion vector of the pixel, Represents the temporal gradient between the t-th video frame and the t+1-th video frame.

4. A smoke and flame identification method according to claim 1, characterized in that: The calculation formula of the sports mode index value is: ； In the formula, represents the motion mode index value, T represents the maximum number of adjacent frames, Represents the pixel motion vector between the tth frame and the t+1th frame, Represents the average motion vector.

5. A smoke and flame identification device, used to execute a smoke and flame identification method according to any one of claims 1 to 4, characterized in that: Includes the following modules: Potential Area Identification Module: used to apply color filters on motion areas to identify potential smoke areas and potential fire areas; Color feature vector matching module: connected to the potential area recognition module, used to construct a preset color feature template, calculate the color feature vector of the potential smoke area and the color feature vector of the potential flame area, determine whether the color feature vector of the potential smoke area matches the preset color feature template, if so, execute step 300, if not, end the processing of the potential smoke area; determine whether the color feature vector of the potential flame area matches the preset color feature template, if so, execute step 300, if not, end the processing of the potential flame area; Suspected reserved sub-region determination module: connected to the color feature vector matching module, used to calculate the motion vector of the potential smoke region and the motion vector of the pixel point of the potential flame region, and judge whether there is a continuous motion pattern in the potential smoke region according to the motion vector of the pixel point of the potential smoke region, if so, the potential smoke region is determined as a suspected smoke reserved sub-region, if not, the processing of the potential smoke region is terminated; according to the motion vector of the pixel point of the potential flame region, judge whether there is a continuous motion pattern in the potential flame region, if so, the potential flame region is determined as a suspected flame reserved sub-region; if not, the processing of the potential flame region is terminated; A module for determining features that meet the requirements: connected to the module for determining suspected reserved sub-regions, and used to calculate the texture feature parameters and color feature vectors for the suspected smoke reserved sub-region and the suspected flame reserved sub-region, respectively, and determine whether the suspected smoke reserved sub-region contains features that meet the requirements of smoke based on the texture feature parameters and color feature vectors of the suspected smoke reserved sub-region. If so, the suspected smoke reserved sub-region is processed; if not, the processing of the suspected smoke reserved sub-region is terminated; and according to the texture feature parameters and color feature vectors of the suspected flame reserved sub-region, determine whether the suspected flame reserved sub-region contains features that meet the requirements of flame. If so, the suspected flame reserved sub-region is processed; if not, the processing of the suspected flame reserved sub-region is terminated; The reserved sub-region determination module is connected to the feature determination module, and is used to construct a smoke sub-region and a flame sub-region at the edges of the suspected smoke reserved sub-region and the suspected flame reserved sub-region, respectively, and calculate the color feature vector and texture feature parameters of the smoke sub-region and the flame sub-region, respectively; determine whether the color feature vector and texture feature parameters of the smoke sub-region are the same as those of the suspected smoke reserved sub-region, and if so, determine the smoke sub-region as the smoke reserved sub-region; if not, end the processing of the smoke sub-region; determine whether the color feature vector and texture feature parameters of the flame sub-region are the same as those of the suspected flame reserved sub-region, and if so, determine the flame sub-region as the flame reserved sub-region; if not, end the processing of the flame sub-region; Identification module: connected to the reserved sub-region determination module and the suspected reserved sub-region determination module, used to form a data set of a deep learning model with suspected smoke reserved sub-regions, suspected flame reserved sub-regions, smoke reserved sub-regions, and flame reserved sub-regions to identify flames and smoke.

Images