Early smoke identification method based on knowledge distillation

Abstract

The invention discloses an early smoke recognition method based on knowledge distillation, and the method comprises the steps: firstly constructing a knowledge distillation training deep learning network which comprises a teacher network and a student network, and each of the teacher network and the student network selects a Faster RCNN + FPN structure; wherein the difference is reflected in that a teacher network uses a relatively complex backbone network to obtain relatively good feature data, and a student network uses a lightweight backbone network to support generalization under the condition of relatively few computing resources. In the training network, the data are respectively input into the teacher network and the student network to obtain two groups of different feature data, and the feature data obtained by the students are adjusted through the feature data obtained by the teachers. And finally, a model capable of performing accurate prediction in the edge environment is obtained. The model is used for recognizing early smoke, and the recognition accuracy, accuracy and false alarm rate of the early smoke are improved.

Description

Technical Field

[0001] The invention relates to the field of computer vision, and in particular to an early smoke recognition method based on knowledge distillation. Background Art

[0003] At present, the common method of smoke detection is to use point-type photoelectric smoke detectors, which use the absorption and scattering of light by smoke particles to identify smoke. They have good effects in relatively closed environments such as buildings and tunnels, but have many problems and limitations in open environments. The main manifestations are: (1) The coverage of a single smoke detector is limited, and dense deployment is required to cover a large area, which is difficult and uneconomical for outdoor environments. (2) Smoke detectors can only detect smoke particles when the fire reaches a certain level, and cannot effectively respond to early smoke. (3) In harsh environments such as sand and dust, particle fog detectors are easily interfered and cause false detection. Therefore, researchers began to pay attention to image-based smoke recognition technology, which uses video surveillance and image pattern recognition technology to analyze whether there is a smoke area in the image. However, smoke is a translucent, irregular, and dynamically changing object that is difficult to distinguish and detect in an image. Compared with open flames, smoke has no obvious RGB features, and its color is usually similar or the same as the background, resulting in unclear edges and low contrast. In addition, the early thin smoke from wildfires has different shapes and is very similar to conventional clouds and fog in the wild in terms of vision and texture, which increases the difficulty of recognition. To address this problem, this paper proposes an early smoke recognition method based on knowledge distillation. Summary of the invention

[0004] The present invention provides an early smoke recognition method based on knowledge distillation, which is mainly used to solve the problem of timely early warning of wildfires by traditional methods.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] The method for early smoke recognition based on knowledge distillation includes the following steps:

[0007] 1) First, the video data is processed to obtain a video data frame sequence;

[0008] 2) Screening the images in the obtained video data frame sequence to obtain training set data with larger smoke ten minutes after the fire and test set data with smaller smoke two minutes after the fire.

[0009] 3) Construct a knowledge distillation-based deep learning training network (A Knowledge Distillingfusion deep newwork), which includes a teacher feature extraction module and a student feature extraction module, such as Figure 1 As shown;

[0010] 4) constructing an early smoke recognition model based on knowledge distillation using the student feature extraction module in the network constructed in step 3);

[0011] 5) Using the knowledge distillation-based early smoke recognition model constructed in step 4), the video data frame sequence is processed to obtain the smoke recognition result.

[0012] After obtaining the video data frame sequence in step 1), the video frames are subjected to formatting and normalization processing such as denoising.

[0013] In step 2), a deep learning network based on knowledge distillation is constructed, which includes a teacher feature extraction module and a student feature extraction module. The teacher feature extraction module adopts a more complex network structure to improve the feature extraction capability and obtain better features. The student feature extraction module adopts a simpler network structure to meet the computing requirements of the construction equipment and obtain better versatility. The student feature extraction module is adjusted by the features obtained by the teacher feature extraction module, so that the results of the student feature extraction are close to the results of the teacher feature extraction, so that the final model can obtain better results with fewer computing resources. The structure of the network is designed as follows:

[0014] (a) Teacher feature extraction module: This module is based on Faster RCNN+FPN architecture, where ResNet50 is used as the backbone. A more complex backbone is used to improve the feature extraction capability of the teacher feature extraction module.

[0015] (b) Student feature extraction module: This module is based on Faster RCNN+FPN architecture, where ResNet18 is used as the backbone. A simpler backbone is used to improve the computational efficiency of the student feature extraction module, so as to facilitate the generalization of the prediction model.

[0016] The features obtained by the teacher network adjust the features obtained by the student network. Both the teacher network module and the student network module are constructed using Faster RCNN+FPN to obtain features of the same format. The adjustment method of the teacher network features to the student network features is the GLKD method.

[0017] The GLKD method is used for feature adjustment. The GLKD method theoretically integrates the two aspects of features and relationships. This method first transforms the 3rd, 4th, 5th, and 6th layers of the FPN layer of the teacher network and the student network. The transformation method is as follows: Figure 2 As shown; the conversion method of the 3rd and 4th layers is formula 1, and the conversion method of the 5th and 6th layers is formula 2. The formula is as follows:

[0018]

[0019]

[0020] Where Norm stands for normalization and ASPP stands for multi-scale dilated convolution. To indicate students, use represents the teacher, and the loss function representing the similarity of the characteristics of the student network and the teacher network is shown in formula (3):

[0021]

[0022] Among them, f uses MSELoss to calculate the similarity between the two.

[0023] The input for the calculation of the similarity loss function is (a) the features converted after the teacher network FPN feature layer is extracted and (b) the features converted after the student network FPN feature layer is extracted.

[0024] In step 3), an early smoke recognition model based on knowledge distillation is constructed.

[0025] In step 4), the constructed early smoke recognition model based on knowledge distillation processes the video data frame sequence to obtain the early smoke recognition result.

[0026] The technical effect of the present invention is unquestionable. The present invention discloses an early smoke recognition method based on knowledge distillation. The present invention utilizes the knowledge distillation algorithm in the current deep learning direction of computer vision to identify and improve the effect of early smoke recognition, thereby improving the accuracy, precision and false alarm rate of early smoke detection in the wild, so as to provide more timely fire warning. And its accuracy, precision and false alarm rate all show better performance than the current classic model, and it also proves that when performing early smoke recognition, using the knowledge distillation algorithm at the same time can obtain better smoke recognition effect while using less computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a model grid structure diagram of the present invention.

[0028] Figure 2 This is the knowledge distillation structure diagram of the present invention.

[0029] Figure 3 This is a structural diagram of the student network prediction model of the present invention. DETAILED DESCRIPTION

[0030] The present invention is further described below in conjunction with the embodiments, but it should not be understood that the above subject matter of the present invention is limited to the following embodiments. Without departing from the above technical ideas of the present invention, various substitutions and changes are made according to the common technical knowledge and customary means in the art, which should all be included in the protection scope of the present invention.

[0031] See also Figure 1 , an early smoke recognition method based on knowledge distillation, including the following steps:

[0032] 1) First, the video data is processed to obtain a video data frame sequence;

[0033] 2) Screening the images in the obtained video data frame sequence to obtain training set data with larger smoke ten minutes after the fire and test set data with smaller smoke two minutes after the fire.

[0034] 3) Construct a knowledge distillation-based deep learning training network (A Knowledge Distillingfusion deep newwork), which includes a teacher feature extraction module and a student feature extraction module, such as Figure 1 As shown;

[0035] 4) constructing an early smoke recognition model based on knowledge distillation using the student feature extraction module in the network constructed in step 3);

[0036] 5) Using the knowledge distillation-based early smoke recognition model constructed in step 4), the video data frame sequence is processed to obtain the smoke recognition result.

[0037] After obtaining the video data frame sequence in step 1), the video frames are subjected to formatting and normalization processing such as denoising.

[0038] In step 2), a deep learning network based on knowledge distillation is constructed, which includes a teacher feature extraction module and a student feature extraction module. The teacher feature extraction module adopts a more complex network structure to improve the feature extraction capability and obtain better features. The student feature extraction module adopts a simpler network structure to meet the computing requirements of the construction equipment and obtain better versatility. The student feature extraction module is adjusted by the features obtained by the teacher feature extraction module, so that the results of the student feature extraction are close to the results of the teacher feature extraction, so that the final model can obtain better results with fewer computing resources. The structure of the network is designed as follows:

[0039] Teacher feature extraction module: This module is based on Faster RCNN+FPN architecture, with ResNet50 as the backbone. A more complex backbone is used to improve the feature extraction capability of the teacher feature extraction module.

[0040] Student feature extraction module: This module is based on Faster RCNN+FPN architecture, with ResNet18 as the backbone. A simpler backbone is used to improve the computational efficiency of the student feature extraction module, so as to facilitate the generalization of the prediction model.

[0041] The features obtained by the teacher network adjust the features obtained by the student network. Both the teacher network module and the student network module are constructed using Faster RCNN+FPN to obtain features of the same format. The adjustment method of the teacher network features to the student network features is the GLKD method.

[0042] The GLKD method is used for feature adjustment. The GLKD method theoretically integrates the two aspects of features and relationships. This method first transforms the 3rd, 4th, 5th, and 6th layers of the FPN layer of the teacher network and the student network. The transformation method is as follows: Figure 2 As shown; the conversion method of the 3rd and 4th layers is formula 1, and the conversion method of the 5th and 6th layers is formula 2. The formula is as follows:

[0043]

[0044]

[0045] Where Norm stands for normalization and ASPP stands for multi-scale dilated convolution. To indicate students, use represents the teacher, and the loss function representing the similarity of the characteristics of the student network and the teacher network is shown in formula (3):

[0046]

[0047] Among them, f uses MSELoss to calculate the similarity between the two.

[0048] The input for the calculation of the similarity loss function is (a) the features converted after the teacher network FPN feature layer is extracted and (b) the features converted after the student network FPN feature layer is extracted.

[0049] In step 3), an early smoke recognition model based on knowledge distillation is constructed.

[0050] In step 4), the constructed early smoke recognition model based on knowledge distillation processes the video data frame sequence to obtain the early smoke recognition result.

Claims

1. An early smoke recognition method based on knowledge distillation, characterized by: Implementation steps: 1) First, the video data is processed to obtain a video data frame sequence; 2) Screening the images in the obtained video data frame sequence to obtain training set data with relatively large smoke in the last ten minutes after the fire occurs, and test set data with relatively small smoke in the first two minutes before the fire occurs. 3) Construct a knowledge distilling fusion deep network, which includes a teacher feature extraction module and a student feature extraction module; 4) constructing an early smoke recognition model based on knowledge distillation using the student feature extraction module in the network constructed in step 3); 5) Using the knowledge distillation-based early smoke recognition model constructed in step 4), the video data frame sequence is processed to obtain the smoke recognition result.

2. The early smoke recognition method based on knowledge distillation according to claim 1 is characterized in that: In step 2), based on the HPWREN-FIgLib source dataset, a new open source smoke target detection dataset FlgLib-Det was constructed. First, the images of 281 fires provided by the source dataset were annotated with bounding boxes and screened according to the time of the fire. In the training set, this paper selected images with larger smoke 10 minutes after the fire to improve the generalization ability of the model; the test set was images between 2 minutes and 10 minutes after the fire occurred to examine the model's detection effect on early smoke. In step 3), a knowledge distillation-based deep learning network (A Knowledge Distillingfusion deep newwork) is constructed, which includes a teacher feature extraction module (Teacher feature extractionmodule) and a student feature extraction module (Student feature extraction module). The teacher feature extraction module adopts a more complex network structure to improve the feature extraction capability and obtain better quality features. The student feature extraction module adopts a simpler network structure to meet the computing requirements of the construction equipment and obtain better versatility. The student feature extraction module is adjusted based on the features obtained by the teacher feature extraction module, so that the results of the student feature extraction are close to the results of the teacher feature extraction, so that the final model can obtain better results with fewer computing resources (the prediction model only uses the student feature extraction module). The structure of the network is designed as follows: (a) Teacher feature extraction module: This module is based on Faster RCNN+FPN architecture, where ResNet50 is used as the backbone. A more complex backbone is used to improve the feature extraction capability of the teacher feature extraction module. (b) Student feature extraction module: This module is based on the Faster RCNN+FPN architecture, in which ResNet18 is used as the backbone. A simpler backbone is used to improve the computational efficiency of the student feature extraction module to facilitate the generalization of the prediction model.

3. The method according to claim 2, characterized in that The features obtained by the teacher network adjust the features obtained by the student network. Both the teacher network module and the student network module are constructed using Faster RCNN+FPN to obtain features of the same format. The adjustment method of the teacher network features to the student network features is the GLKD method.

4. The method according to claim 3, characterized in that The GLKD method is used for feature adjustment. The GLKD method theoretically integrates the two aspects of features and relationships. This method first processes and transforms the 3rd, 4th, 5th, and 6th layers of the FPN layer of the teacher network and the student network; the conversion method of the 3rd and 4th layers is formula 1, and the conversion method of the 5th and 6th layers is formula 2. The formula is as follows: Where Norm stands for normalization and ASPP stands for multi-scale dilated convolution. To indicate students, use represents the teacher, and the loss function representing the similarity of the characteristics of the student network and the teacher network is shown in formula (3): Among them, f uses MSELoss to calculate the similarity between the two.

5. The method according to claim 4, characterized in that: The input for the calculation of the similarity loss function is (a) the features converted after the teacher network FPN feature layer is extracted and (b) the features converted after the student network FPN feature layer is extracted.

6. The method according to claim 1, characterized in that: In the step 4), an early smoke recognition model based on knowledge distillation is constructed.

Images