Fire detection method and electronic equipment

Abstract

The invention provides a fire detection method and electronic equipment, and the method comprises the steps: carrying out the coding processing of a to-be-detected image through employing an encoder of a neural network model, obtaining an image coding feature, and enabling the encoder to comprise at least one suspected flame adaptive module SFAM network; decoding the image coding features by using a decoder of a neural network model to obtain a flame transparency mask; performing feature extraction on the to-be-detected image by using a flame transparency mask to obtain flame features; and classifying the flame features by using a classifier of a neural network model to obtain a fire detection result of the to-be-detected image. Through a suspected flame adaptive module SFAM network in the encoder, the capturing and extracting capability of image encoding features such as flame contour and transparency is enhanced, so that the model can separate flame features from different scales during feature extraction according to flame transparency masks, and background features are removed, thereby improving the accuracy of fire detection.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and deep learning, and in particular, to a fire detection method and electronic equipment. Background Art

[0002] Current fire detection methods often fail to effectively eliminate the interference of background information when extracting flame features. For example, strong light, reflective objects, or backgrounds with flame-like colors may be misjudged as flames, resulting in an increased false alarm rate. Especially in complex scenes, flame features and background features overlap in color, texture, shape, etc., making it difficult to distinguish the true flame area, affecting the accuracy of fire detection. Therefore, the current accuracy of fire detection is low. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a fire detection method and electronic equipment, so as to improve the problem of low accuracy of fire detection.

[0004] The embodiment of the present application provides a fire detection method, including: using an encoder of a neural network model to encode an image to be detected to obtain image coding features, the encoder includes: at least one suspected flame adaptive module SFAM network; using a decoder of the neural network model to decode the image coding features to obtain a flame transparency mask; using the flame transparency mask to extract features of the image to be detected to obtain flame features; using a classifier of the neural network model to classify the flame features to obtain a fire detection result of the image to be detected. In the implementation process of the above scheme, the suspected flame adaptive module SFAM network in the encoder is used to enhance the ability to capture and extract image coding features such as flame contour and transparency, so that the neural network model including the encoder can obtain the flame transparency mask after decoding according to the image coding features, and the flame transparency mask is used to separate the flame features at different scales during feature extraction and remove the background features, so that the model can accurately identify the flame in different scenes and backgrounds, effectively improving the accuracy of fire detection.

[0005] Optionally, in an embodiment of the present application, the encoder includes: a convolution layer, a first SFAM network and a second SFAM network; the encoder using the neural network model performs encoding processing on the image to be detected, including: using the convolution layer to perform convolution processing on the image to be detected to obtain convolution features; using the first SFAM network to extract the features of the region of interest from the convolution features; using the second SFAM network to perform feature extraction and downsampling processing on the features of the region of interest to obtain image coding features. In the implementation process of the above scheme, the input image is subjected to preliminary feature extraction through the convolution layer to generate basic convolution features, and the first SFAM network and the second SFAM network are used to further refine these features, respectively, to extract deeper and finer-grained features, thereby enhancing the feature extraction capability of the model. Furthermore, since the flame contour branch of the first SFAM network can extract the contour boundary of the flame, it helps the model to locate the flame area more accurately and reduce background interference, and the flame transparency branch of the second SFAM network can extract the transparency feature of the flame, further enhancing the recognition capability of the flame area and reducing the interference of similar colors and textures in the background. In addition, through the multi-scale feature fusion of the first SFAM network and the second SFAM network, the model can better capture the flame characteristics of different scales and improve the detection accuracy in multi-scale complex scenes.

[0006] Optionally, in an embodiment of the present application, the first SFAM network includes: a flame outline branch and a flame transparency branch; using the first SFAM network to extract the region of interest features from the convolution features, including: using the flame outline branch to extract features from the convolution features to obtain flame outline features; using the flame transparency branch to extract features from the convolution features to obtain flame transparency features; fusing the flame outline features and the flame transparency features to obtain the region of interest features. In the implementation process of the above scheme, the flame outline branch can extract the outline boundary of the flame, which helps the model to more accurately locate the flame area and reduce background interference, and the flame transparency branch can extract the transparency features of the flame. These features reflect the translucent nature of the flame, which helps the model to better understand the internal structure and transparency changes of the flame and reduce the interference of similar colors and textures in the background. Furthermore, by fusing the flame outline features and the flame transparency features, the model can comprehensively utilize multiple feature information to improve the richness and representativeness of the features. This fusion mechanism enables the model to more accurately identify flames in complex scenes and reduce false positives and false negatives.

[0007] Optionally, in an embodiment of the present application, the flame contour branch includes: two first convolution filters and an average pooling layer; the flame contour branch is used to extract features from the convolution feature, including: using one of the two first convolution filters to perform convolution processing on the convolution feature to obtain image convolution features; using the average pooling layer to perform pooling processing on the image convolution features to obtain image boundary features; using the other first convolution filter of the two first convolution filters to perform convolution processing on the image boundary features to obtain flame contour features. In the implementation process of the above scheme, the fine-grained features of the flame, such as the edge, shape and texture of the flame, are captured through multi-layer convolution processing. These features are of great significance for distinguishing the flame from the background, and the image convolution features are pooled through the average pooling layer to obtain the image boundary features. This pooling operation can reduce the spatial dimension of the feature map while retaining key boundary information, which helps to reduce background interference. Furthermore, through multi-stage feature extraction, the model can capture the contour information of the flame more comprehensively. This multi-stage processing enables the model to more accurately identify flames in complex backgrounds, reduce false positives and negatives, and the flame contour features here can provide fine-grained boundary information, helping the model to more accurately locate the boundaries of the flame and improve the accuracy of fire detection.

[0008] Optionally, in an embodiment of the present application, the flame transparency branch includes: a first convolution filter, a second convolution filter and a compressed excitation SE channel attention layer; the flame transparency branch is used to extract the convolution feature, including: using the second convolution filter to convolve the convolution feature to obtain the image convolution feature; using the SE channel attention layer to process the image convolution feature to obtain the channel attention feature; using the first convolution filter to convolve the channel attention feature to obtain the flame transparency feature. In the implementation process of the above scheme, the image convolution feature is processed by the SE channel attention layer to obtain the channel attention feature. Since the SE channel attention layer can dynamically adjust the weights of different channels, the weights of flame-related channels are enhanced, and the interference of irrelevant background information is suppressed, and the first convolution filter further convolves the channel attention feature to extract the transparency features of the flame. These features can more accurately describe the transparency changes of the flame and reduce the interference of similar colors and textures in the background. Furthermore, through multi-stage feature extraction, the model can capture the transparency information of the flame more comprehensively. This multi-stage processing enables the model to more accurately identify flames in complex backgrounds, reduce false positives and negatives, and help the model more accurately describe the changes in flame transparency, thereby improving the accuracy of fire detection.

[0009] Optionally, in an embodiment of the present application, the second SFAM network includes: a downsampling branch, a flame outline branch, and a flame transparency branch; the second SFAM network is used to extract features and downsample features of the region of interest, including: using the flame outline branch to extract features of the region of interest to obtain flame outline features; using the flame transparency branch to extract features of the flame outline to obtain flame transparency features; using the downsampling branch to downsample the flame outline features and the flame transparency branch to obtain image coding features. In the implementation process of the above scheme, by processing the outline and transparency of the flame separately, the model can learn a richer representation of the flame features, which not only helps to distinguish the flame from other luminous objects, but also better simulates the dynamic characteristics of the flame, such as changes in brightness and color, thereby increasing the feature expression ability of the neural network model. In addition, through the independent flame outline branch and flame transparency branch, the network can more accurately capture the boundary information and transparency changes of the flame, and can accurately identify the flame in a complex background when the flame edge is blurred or partially blocked, which helps to improve the recognition accuracy of the flame in a complex background. Furthermore, due to the diverse manifestations of flames in different scenarios, the key attributes of flames can be extracted through a multi-branch structure, which can make the model more generalizable and suitable for more diverse application scenarios, thereby increasing the generalization ability of the neural network model.

[0010] Optionally, in an embodiment of the present application, a classifier of a neural network model is used to classify flame features, including: fusing the image to be detected and the flame features to obtain fused features; and classifying the fused features using the classifier of the neural network model to obtain a fire detection result of the image to be detected. In the implementation process of the above scheme, by fusing the image itself with the extracted flame features, the expressiveness of the features can be enhanced, so that the fused features contain the contextual information of the image, which is crucial for understanding the context of the flame, so that the neural network model can understand the flame features and contextual information more comprehensively, and the classifier can identify the flame more accurately, reducing the probability of misclassifying non-flame objects as flames, thereby improving the accuracy of fire detection.

[0011] Optionally, in an embodiment of the present application, before using the encoder of the neural network model to encode the image to be detected, it also includes: obtaining multiple sample images and the transparency mask label corresponding to each sample image; using the encoder of the neural network model to encode the sample image in the training data set to obtain the encoding features of the sample image; using the decoder of the neural network model to decode the encoding features of the sample image to obtain the sample transparency mask; calculating the mask loss value between the sample transparency mask and the transparency mask label; updating the weight parameters of the encoder and the weight parameters of the decoder according to the mask loss value. In the implementation process of the above scheme, by learning a large number of sample images and their corresponding transparency mask labels, the encoder can better capture the high-level features related to the flame, which include not only the contour information of the flame, but also the details such as its transparency change, so that the encoder can recognize the flame more accurately in practical applications. In addition, by introducing the mask loss value to guide the learning of the encoder and decoder, the model can learn to ignore noise and interference factors and focus on the key features of the flame, thereby enhancing the robustness of the model in different environments and improving its reliability in actual deployment.

[0012] Optionally, in an embodiment of the present application, obtaining a plurality of sample images and a transparency mask label corresponding to each sample image includes: obtaining a background image, a plurality of flame foreground images, and a transparency mask corresponding to each flame foreground image; calculating the background image and a flame foreground image to obtain a size ratio, and generating a scale factor set according to the size ratio; for each flame foreground image in the plurality of flame foreground images, scaling the flame foreground image according to a scale factor randomly selected from the scale factor set to obtain a plurality of scaled foreground images, and scaling the transparency mask corresponding to the flame foreground image according to the scale factor to obtain scaled transparency masks corresponding to the plurality of scaled foreground images; for each scaled foreground image in the plurality of scaled foreground images, synthesizing the background image, the scaled foreground image, and the scaled transparency mask corresponding to the scaled foreground image to obtain multiple flame images of different scales, and determining the multiple flame images of different scales as sample images; for the scaled transparency mask corresponding to each scaled foreground image, taking the maximum transparency value of the pixel point at the same position between the scaled transparency mask and the scaled transparency mask that participated in the last synthesis to obtain a transparency mask label corresponding to the sample image. In the implementation process of the above scheme, by randomly scaling the flame foreground image and synthesizing it into the background, the model can learn the flame characteristics at different scales, which helps to improve the model's ability to detect and identify flames of different sizes in practical applications. In addition, since the training data contains changes in multiple scales, the model will be exposed to more diverse flame forms during the training process, thereby enhancing its generalization ability on unseen data. Through random scaling and synthesis, each sample image has a unique flame layout and scale changes, which increases the diversity of training data, helps prevent model overfitting, and improves its robustness in various situations. Further, by synthesizing flame foregrounds of different scales into the background, such flame images of different scales can help the model learn the dynamic characteristics of flames, such as the blurring of flame edges, brightness changes, etc., which are very important for capturing the true appearance and behavior of flames, especially in video surveillance and fire warning systems. Further combining transparency masks for pixel-level fusion can more realistically simulate the performance of flames under different backgrounds, which helps the model better understand the relationship between flames and backgrounds and improve the accuracy of flame boundary and transparency prediction. During the synthesis process, the transparency mask is updated by taking the maximum transparency value, which ensures that the final transparency mask can accurately reflect the superposition effect of all flame foregrounds, avoids the loss of transparency information, and improves the quality of the transparency mask.

[0013] Optionally, in an embodiment of the present application, a mask loss value between a sample transparency mask and a transparency mask label is calculated, including: if the sample image is a positive sample image with flames, and the transparency value of a pixel in the positive sample image in the sample transparency mask is less than the transparency value in the transparency mask label, then the mask loss value is calculated according to a preset first penalty coefficient, the sample transparency mask and the transparency mask label, the first penalty coefficient represents a coefficient for penalizing the neural network model when there is less prediction in the positive sample image; if the sample image is a positive sample image with flames, and the transparency value of a pixel in the positive sample image in the sample transparency mask is greater than or equal to the transparency value in the transparency mask label, then the mask loss value is set to zero; if the sample image is a negative sample image without flames, and the transparency value of a pixel in the negative sample image in the sample transparency mask is greater than the transparency value in the transparency mask label, then the mask loss value is calculated according to a preset second penalty coefficient, the sample transparency mask and the transparency mask label, the second penalty coefficient represents a coefficient for penalizing the neural network model when there is more prediction in the negative sample image, and the first penalty coefficient is greater than the second penalty coefficient. In the implementation process of the above scheme, by setting different penalty coefficients, the model is trained to try not to miss any flame area in the positive sample, and to avoid over-predicting the flame in the negative sample, thereby reducing the cases of false positives (identifying the non-flame area as a flame) and false negatives (not identifying the actual flame). For the positive sample image, if the model underestimates the transparency of the flame area (i.e., under-predicts), it will be punished more, which helps the model to more accurately capture the edge and details of the flame, thereby improving the accuracy of the flame boundary positioning. In addition, since the first penalty coefficient is greater than the second penalty coefficient, the model tends to conservatively increase the prediction of the flame area during the learning process, rather than over-expanding the prediction range, which helps to form a clearer and more stable decision boundary. Furthermore, by imposing a larger penalty on under-predictions, the model is forced to learn those subtle flame features that may be difficult to detect, which helps the model better understand the essential characteristics of the flame, such as color, texture, etc., thereby improving the overall recognition performance of the neural network model.

[0014] The embodiment of the present application also provides a fire detection device, including: a coding feature acquisition module, used to use an encoder of a neural network model to encode the image to be detected to obtain image coding features, the encoder including: at least one suspected flame adaptive module SFAM network; a feature decoding processing module, used to use a decoder of the neural network model to decode the image coding features to obtain a flame transparency mask; a flame feature acquisition module, used to use the flame transparency mask to extract features of the image to be detected to obtain flame features; a detection result acquisition module, used to classify the flame features using a classifier of the neural network model to obtain a fire detection result of the image to be detected.

[0015] Optionally, in an embodiment of the present application, the encoder includes: a convolution layer, a first SFAM network and a second SFAM network; the coding feature acquisition module includes: an image convolution processing submodule, which is used to use the convolution layer to perform convolution processing on the image to be detected to obtain convolution features; a first feature extraction submodule, which is used to extract region of interest features from the convolution features using the first SFAM network; the first feature extraction submodule is used to use the second SFAM network to perform feature extraction and down-sampling processing on the region of interest features to obtain image coding features.

[0016] Optionally, in an embodiment of the present application, the first SFAM network includes: a flame contour branch and a flame transparency branch; the first feature extraction submodule includes: a first contour feature extraction unit, used to use the flame contour branch to extract features from the convolution feature to obtain a flame contour feature; a first transparency feature extraction unit, used to use the flame transparency branch to extract features from the convolution feature to obtain a flame transparency feature; a flame feature fusion unit, used to fuse the flame contour feature and the flame transparency feature to obtain a region of interest feature.

[0017] Optionally, in an embodiment of the present application, the flame contour branch includes: two first convolution filters and an average pooling layer; the first contour feature extraction unit includes: a first convolution filtering subunit, used to use one of the two first convolution filters to perform convolution processing on the convolution feature to obtain the image convolution feature; an image pooling processing subunit, used to use the average pooling layer to perform pooling processing on the image convolution feature to obtain the image boundary feature; the first convolution processing subunit, used to use the other first convolution filter of the two first convolution filters to perform convolution processing on the image boundary feature to obtain the flame contour feature.

[0018] Optionally, in an embodiment of the present application, the flame transparency branch includes: a first convolution filter, a second convolution filter and a compressed excitation SE channel attention layer; the first transparency feature extraction unit includes: a second convolution filter subunit, used to use the second convolution filter to convolve the convolution feature to obtain the image convolution feature; a channel attention processing subunit, used to use the SE channel attention layer to process the image convolution feature to obtain the channel attention feature; the second convolution processing subunit, used to use the first convolution filter to convolve the channel attention feature to obtain the flame transparency feature.

[0019] Optionally, in an embodiment of the present application, the second SFAM network includes: a downsampling branch, a flame contour branch and a flame transparency branch; the first feature extraction submodule includes: a second contour feature extraction unit, used to use the flame contour branch to extract features of the region of interest to obtain flame contour features; a second transparency feature extraction unit, used to use the flame transparency branch to extract features of the flame contour features to obtain flame transparency features; a flame feature downsampling processing unit, used to use the downsampling branch to downsample the flame contour features and the flame transparency branch to obtain image coding features.

[0020] Optionally, in an embodiment of the present application, the detection result acquisition module includes: an image feature fusion submodule, which is used to fuse the image to be detected and the flame feature to obtain a fusion feature; and a fusion feature classification submodule, which is used to classify the fusion feature using a classifier of a neural network model to obtain a fire detection result of the image to be detected.

[0021] Optionally, in an embodiment of the present application, the fire detection device also includes: an image label acquisition module, which is used to acquire multiple sample images and a transparency mask label corresponding to each sample image; a coding feature acquisition module, which is used to use the encoder of the neural network model to encode the sample images in the training data set to obtain the coding features of the sample images; a transparency mask acquisition module, which is used to use the decoder of the neural network model to decode the coding features of the sample images to obtain the sample transparency mask; a mask loss calculation module, which is used to calculate the mask loss value between the sample transparency mask and the transparency mask label; a weight parameter update module, which is used to update the weight parameters of the encoder and the weight parameters of the decoder according to the mask loss value.

[0022] Optionally, in an embodiment of the present application, the image label acquisition module includes: an image mask acquisition submodule, which is used to acquire a background image, multiple flame foreground images, and a transparency mask corresponding to each flame foreground image; a size set generation submodule, which is used to calculate the background image and the flame foreground image to obtain a size ratio, and generate a scale factor set according to the size ratio; an image mask scaling submodule, which is used to scale each flame foreground image in the multiple flame foreground images according to a scale factor randomly selected from the scale factor set, to obtain multiple scaled foreground images, and to scale the transparency mask corresponding to the flame foreground image according to the scale factor. The brightness mask is scaled to obtain scaled transparency masks corresponding to multiple scaled foreground images; an image mask synthesis submodule is used to synthesize the background image, the scaled foreground image and the scaled transparency mask corresponding to the scaled foreground image for each scaled foreground image in the multiple scaled foreground images to obtain multi-flame images of different scales, and determine the multi-flame images of different scales as sample images; a mask label acquisition submodule is used to obtain the maximum transparency value of the pixel point at the same position between the scaled transparency mask and the scaled transparency mask that participated in the last synthesis for the scaled transparency mask corresponding to each scaled foreground image, so as to obtain the transparency mask label corresponding to the sample image.

[0023] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions execute the method described above when executed by the processor.

[0024] An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is executed.

[0025] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 A flow chart of a fire detection method provided in an embodiment of the present application is shown;

[0028] Figure 2 A schematic diagram of the structure of a neural network model provided in an embodiment of the present application is shown;

[0029] Figure 3 A schematic diagram of the structure of the SFAM network provided in an embodiment of the present application is shown;

[0030] Figure 4 A schematic diagram of the structure of a first SFAM network provided in an embodiment of the present application is shown;

[0031] Figure 5 A schematic diagram of data synthesis for multi-scale transformation of an image provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] To make the purpose, technical scheme and advantages of the embodiment of the present application clearer, the technical scheme in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. It should be understood that the drawings in the embodiment of the present application only serve the purpose of explanation and description, and are not used to limit the protection scope of the embodiment of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in the embodiment of the present application shows the operation implemented according to some embodiments of the embodiment of the present application. It should be understood that the operation of the flowchart can be implemented out of order, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the embodiment of the present application, or remove one or more operations from the flowchart.

[0033] It is understandable that the "first" and "second" in the embodiments of the present application are used to distinguish similar objects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit the difference. In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that there may be three relationships, such as A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the related objects before and after are in an "or" relationship. The term "multiple" refers to more than two (including two), and similarly, "multiple groups" refers to more than two groups (including two groups).

[0034] It should be noted that the fire detection method provided in the embodiment of the present application can be executed by an electronic device, where the electronic device refers to a device terminal or server with the function of executing a computer program, such as a smart phone, a personal computer, a tablet computer, a personal digital assistant or a mobile Internet device. A server refers to a device that provides computing services through a network, such as an x86 server and a non-x86 server, and a non-x86 server includes a mainframe, a minicomputer and a UNIX server.

[0035] In the relevant fire detection technology, a large amount of background information is mixed in the image, so the flame features extracted from the image cannot effectively eliminate the interference of background information. For example, strong light may produce light spots or bright areas similar to flames. These areas are very similar to flames in color and brightness and are easily misjudged as flames. Another example: reflective objects (such as metal surfaces, glass, etc.) will produce strong reflections under lighting conditions. These reflective areas may be similar to the visual characteristics of flames. Another example: objects with certain background colors (such as sunset glow, fire clouds, red lights and red objects, etc.) are very close to the color of flames, and background areas of these colors are easily misjudged as flames. In particularly complex scenes, it is found that flame features and background features overlap in color, texture, shape, etc. For example, color overlaps, such as the orange wall or yellow furniture in the photo is similar to the color of the flame, texture overlaps, such as the texture of the flame (such as the ripples of the flame, etc.) may be similar to some background textures (such as fabric or wood in the photo, etc.), and shape overlaps, such as the shape of the flame (such as the sharp edge of the flame, irregular shape) may be similar to the shape of some background objects (such as knives or broken transparent glass in the photo, etc.). The interference of the above overlapping factors will lead to feature contamination, making it difficult to distinguish between the real flame features and the background features, affecting the accuracy of fire detection.

[0036] In view of the above problems, although the background area can be removed by cutting out the image with a manually made trisection map, thereby reducing the interference of background information, the trisection map here divides an image into three parts: foreground area, background area and unknown area. In the process of cutting out the image, the unknown area is predicted based on the foreground area and the background area, and the interference of background information is reduced based on these three areas. However, in a real fire detection scene, each image to be detected is different, and it is impossible to manually make a trisection map to cut out the background area every time. Therefore, it is unrealistic to perform fire detection by cutting out the image with a manually made trisection map.

[0037] To improve the above problems, see Figure 1The flowchart of the fire detection method provided by the embodiment of the present application is shown; after long-term research, it is found that the flame in a real fire is usually a single semantic object. Therefore, when detecting a real fire, it is not necessary to learn the features of the foreground and background areas of different objects from the tri-map as in the traditional cutout task, but to allow some background features not to be accurately removed, and to retain and not miss the flame features as much as possible. Based on this discovery, the embodiment of the present application provides a tri-map-free fire detection method, which uses a suspected flame adaptive module (SFAM) network to efficiently capture suspected flame features in shallow space, thereby retaining all flame features as much as possible. The implementation methods of the above-mentioned fire detection method may include:

[0038] Step S110: using an encoder of a neural network model to encode the image to be detected to obtain image coding features, the encoder comprising: at least one suspected flame adaptive module SFAM network.

[0039] See also Figure 2 A structural schematic diagram of a neural network model provided in an embodiment of the present application is shown; the neural network model may include: an encoder, a decoder and a classifier; wherein the encoder and the decoder are linked in sequence, and the connection relationship between the classifier and the encoder and the decoder is shown in the accompanying drawings, the encoder includes: at least one suspected flame adaptive module SFAM network, and the figure shows that it includes two SFAM networks, namely a first SFAM network and a second SFAM network, but in fact, in the specific implementation process, more SFAM networks may be included, or only one SFAM network may be included.

[0040] See also Figure 3The structural schematic diagram of the SFAM network provided by the embodiment of the present application is shown; the SFAM network includes: a first structure (see the left side of the figure) and / or a second structure (see the right side of the figure); wherein the first structure is a dual-branch design structure that focuses on and processes flame features, and its stride (stride) can be 1 (which can be represented by stride=1), and the first structure is mainly used to capture more suspicious flame features in shallow spatial features. The second structure is a single-branch design structure for reducing the amount of calculation, and its stride (stride) can be 2 (which can be represented by stride=2). It can be understood that different structures can be included in different SFAM networks, which can include multiple first structures with stride=1, in which case the scale of the input feature is the same as the scale of the output feature. Of course, multiple second structures with stride=2 can also be included. If there are only two second structures, the scale of the output feature is 1 / 4 of the scale of the input feature, thereby effectively reducing the amount of calculation of the SFAM network. Therefore, the neural network model including such an SFAM network can be considered as a lightweight network model. Of course, in order to characterize the features of interest, the first structure and the second structure are connected front and back. In this case, the first structure and the second structure are included at the same time. The first structure of the former is used to extract the flame features of interest, and the second structure of the latter is used to accelerate the calculation. Therefore, when the SFAM network in the neural network model includes the first structure and the second structure at the same time, it can effectively reduce the amount of calculation caused by the complexity of the model while improving the accuracy of the neural network model in fire detection.

[0041] Step S120: using the decoder of the neural network model to decode the image coding features to obtain a flame transparency mask.

[0042] Step S130: extracting features of the image to be detected using the flame transparency mask to obtain flame features.

[0043] Step S140: using the classifier of the neural network model to classify the flame features to obtain the fire detection result of the image to be detected.

[0044] In the implementation process of the above scheme, the suspected flame adaptive module SFAM network in the encoder is used to enhance the ability to capture and extract image coding features such as flame contour and transparency, so that the neural network model including the encoder can obtain a flame transparency mask after decoding according to the image coding features. The flame transparency mask is used to separate flame features from different scales during feature extraction and remove background features, so that the model can accurately identify flames in different scenes and backgrounds, effectively improving the accuracy of fire detection.

[0045] As an optional implementation of the above step S110, the above encoder may include: a convolutional layer, a first SFAM network and a second SFAM network; the implementation of the above encoder using the neural network model to encode the image to be detected may include:

[0046] Step S111: Use a convolution layer to perform convolution processing on the image to be detected to obtain convolution features.

[0047] Step S112: Use the first SFAM network to extract region of interest features from the convolutional features.

[0048] See also Figure 4 The structural schematic diagram of the first SFAM network provided by the embodiment of the present application is shown; the above-mentioned first SFAM network may include: a flame contour branch (see the left side of the figure) and a flame transparency branch (see the right side of the figure), wherein the input of the flame contour branch and the input of the flame transparency branch are the same, and the output of the flame contour branch and the output of the flame transparency branch are fused before output.

[0049] The implementation of the above steps S111 to S112 is, for example: using a convolution layer (Conv layer) to perform convolution processing on the image to be detected, obtaining convolution features, and using the first SFAM network to extract the region of interest features from the convolution features. The first SFAM network may include a plurality of first structures, and the plurality of first structures may be connected in series to form the first SFAM network, or may be connected in parallel to form the first SFAM network.

[0050] Step S113: Use the second SFAM network to perform feature extraction and down-sampling processing on the features of the region of interest to obtain image coding features.

[0051] The implementation method of the above step S113 is, for example: using a second SFAM network including multiple first structures and multiple second structures to extract features of the region of interest and perform downsampling processing to obtain image coding features. The number of first structures and the number of second structures in the second SFAM network may be the same, and the first structures and the second structures may be alternately connected in series to form the second SFAM network.

[0052] As an optional implementation of the above step S112, the above implementation of using the first SFAM network to extract the region of interest features from the convolutional features may include:

[0053] Step S112a: Use the flame contour branch to extract the convolution feature to obtain the flame contour feature.

[0054] The flame outline branch, also known as the average pooling branch, refers to a neural network branch used to extract flame outline features. The branch may include two 1×1 convolution filters and an average pooling (AvgPool) layer, where the 1×1 convolution filter is represented by Conv(1×1). The flame outline branch can smoothly capture the global shape of the flame in a larger range by downsampling the input feature map, thereby compensating for the ignored global context information and effectively reducing the interference of background noise.

[0055] The flame outline feature, also known as the flame morphology feature, is a feature mainly used to remove the background outside the flame outline, that is, to remove background interference information that is not related to the flame, such as sunset, sunrise, and red objects.

[0056] Step S112b: Use the flame transparency branch to extract the convolution feature to obtain the flame transparency feature.

[0057] The flame transparency branch, also known as the attention branch, refers to the neural network branch used to extract flame transparency features. This branch includes a 3×3 convolution filter with a stride of 1, a 1×1 convolution filter, and a Squeeze-and-Excitation (SE) channel attention layer. The flame transparency branch pays more attention to flame transparency, and the edges inside the flame are transparent, which effectively preserves the details inside the flame.

[0058] The flame transparency feature is mainly used to reduce the influence of the mixed background inside the flame outline and unify the flame intensity of multiple scenes. This part of the mixed background may be caused by the scene deviation in the flame characteristics of different scenes.

[0059] The implementation method of the above step S112b is, for example: after analyzing the flames in multiple real fire scenes, it is found that the edge areas of the flames are translucent, so that the flames and backgrounds of different scenes form linear combinations of different transparency, and the linear combination can reflect the intensity of the flames at different positions. These edge areas are easily mixed into the mixed background, for example, objects similar to the flames in color, texture, shape, etc. are easily mixed into the mixed background. Therefore, in order to retain the core distinguishing features of the flames as much as possible and unify the flame intensity of multiple scenes, the flame transparency branch is used here to extract the convolution features to obtain the flame transparency features.

[0060] Step S112c: Fusing the flame outline feature and the flame transparency feature to obtain the feature of the region of interest.

[0061] The implementation method of the above step S112c is, for example: by fusing the flame outline feature and the flame transparency feature through the first SFAM network, the first SFAM network can capture the global outline information of the flame and pay attention to the details of the transparent edge inside the flame, thereby obtaining a more comprehensive feature in the region of interest. The above process of fusing the flame outline feature and the flame transparency feature can be expressed by the formula: M3 = M1 + M2; wherein M1 represents the flame outline feature, M2 represents the flame transparency feature, and M3 represents the region of interest feature.

[0062] It is understandable that the activation function used in the first SFAM network is the Leaky ReLU function. Compared with the non-monotonic swish function, this Leaky ReLU function can make the flame transparency branch more accurately reflect the intensity of the flame. In addition, since the swish function is non-monotonic, its negative axis may contain potential suspicious flame features that are lost by the model, and the Leaky ReLU function can allow the first SFAM network to automatically learn and adjust these potential flame features, thereby effectively improving the ability of the neural network model to capture potential suspicious flame features. Furthermore, the Leaky ReLU function has a linear characteristic on the negative axis, which makes the neural network model more flexible in processing potential suspicious flame features, thereby improving the overall detection accuracy and robustness of fire detection.

[0063] As an optional implementation of the above step S112a, the above flame profile branch may include: two first convolution filters and an average pooling layer; the above implementation of extracting convolution features using the flame profile branch may include:

[0064] Step S112d: Use one of the two first convolution filters to perform convolution processing on the convolution feature to obtain the image convolution feature.

[0065] The implementation method of the above step S112d is as follows: Assume that the input image convolution feature is represented as M∈R c×h×w , the two first convolution filters mentioned above can both be 1×1 convolution layers, and one of the first convolution filters can be used to expand the convolution feature M by the 1×1 convolution layer dimension to map it to n times the original spatial feature (M ′ 1∈R nc ×h×w ) to obtain more detailed mapping features. Then, the mapping features are processed by batch normalization (BN) and leaky ReLU activation function in turn to obtain more detailed image convolution features.

[0066] Step S112e: Use the average pooling layer to perform pooling processing on the image convolution features to obtain image boundary features.

[0067] An implementation example of the above step S112e is: using a 3×3 average pooling layer with a stride of 1 to simplify the input feature map by calculating the average value of the local area, and performing pooling processing on the image convolution features, thereby capturing the image boundary features between the global flame and the background area.

[0068] Step S112f: Use another first convolution filter of the two first convolution filters to perform convolution processing on the image boundary features to obtain flame outline features.

[0069] An implementation example of the above step S112f is: using another first convolution filter (eg, a 1×1 convolution layer) of the two first convolution filters to perform convolution processing on the image boundary features to obtain flame contour features, where the flame contour features can be represented by M1.

[0070] It is understandable that the processing in the above-mentioned step S112d to step S112f can be expressed by the formula: M1 = Conv 1×1 (BL(Avgpool(Conv 1×1 (M)))); where M1 represents the flame profile feature, Conv 1×1 represents a 1×1 convolutional layer, BL(·) represents batch normalization (BN) processing and Leaky ReLU activation function processing, Avgpool represents an average pooling layer, and M represents image convolution features.

[0071] As an optional implementation of the above step S112b, the above flame transparency branch may include: a first convolution filter, a second convolution filter and a compressed excitation (SE) channel attention layer; the above implementation of extracting convolution features using the flame transparency branch may include:

[0072] Step S112g: Use the second convolution filter to perform convolution processing on the convolution feature to obtain the image convolution feature.

[0073] The implementation method of the above step S112g is as follows: Assume that the input image convolution feature is represented as M∈R c×h×w , the convolution feature M can be convolved using a second convolution filter (e.g., a 3×3 convolution layer) to obtain a convolution filter feature, which is then batch normalized (BN) and processed with a Leaky ReLU activation function. The obtained image convolution feature can be represented as M′2. Since the 3×3 convolution layer aggregates the surrounding local features, the obtained image convolution feature can reflect the intensity changes of the flame at different locations.

[0074] Step S112h: Use the SE channel attention layer to process the image convolution features to obtain channel attention features.

[0075] The implementation of the above step S112h is, for example, to process the image convolution features using the compression-excitation SE channel attention layer, redistribute weights to different feature channels, and thus capture the channel attention features of the flame transparency changes. It is understandable that the processing process of the channel attention feature can be expressed as SE = S (Conv 1×1 (ReLU(Conv 1×1 (Avgpool(M′2))));where M′2 represents the image convolution feature, Avgpool represents the average pooling layer, ReLU represents the ReLU activation function, Conv 1×1 represents a 1×1 convolutional layer, and S(·) represents the Sigmod activation function.

[0076] Step S112i: Use the first convolution filter to perform convolution processing on the channel attention feature to obtain the flame transparency feature.

[0077] An implementation example of the above step S112i is: using batch normalization (BN) and the first convolution filter to perform normalization and convolution processing on the channel attention features respectively, and the flame transparency feature after dimension reduction can be expressed as M2.

[0078] It is understandable that the processing in the above-mentioned step S112g to step S112i can be expressed by the formula: M2 = Conv 1×1 (BL(SE(Conv 3×3 (M)))); where M2 represents the flame transparency feature, Conv 1×1 represents a 1×1 convolutional layer, BL(·) represents batch normalization (BN) processing and Leaky ReLU activation function processing, SE represents the squeezed excitation (SE) channel attention layer, Conv 3×3 represents a 3×3 convolutional layer, and M represents the image convolution feature.

[0079] As an optional implementation of the above step S113, the above second SFAM network may include: a downsampling branch, a flame contour branch and a flame transparency branch; the above implementation of using the second SFAM network to extract features of the region of interest and perform downsampling processing may include:

[0080] Step S113a: Use the flame contour branch to extract features of the region of interest to obtain flame contour features.

[0081] Step S113b: Use the flame transparency branch to extract the flame contour feature to obtain the flame transparency feature.

[0082] Step S113c: Use the downsampling branch to downsample the flame contour feature and the flame transparency branch to obtain image coding features.

[0083] Optionally, as an optional implementation of the above step S120, for example: the decoder of the above neural network model may include a first upsampling network and a second upsampling network, and may actually include more upsampling networks, and only two upsampling networks are shown in the figure. Specifically, the first upsampling network may be used to perform upsampling decoding processing on the image coding feature to obtain the first upsampling feature, and the second upsampling network may be used to perform upsampling decoding processing on the first upsampling feature to obtain the flame transparency mask. Among them, the above-mentioned first upsampling network and / or the second upsampling network may adopt a bilinear interpolation algorithm for upsampling, and after the first upsampling network image coding feature is used for upsampling decoding processing, the obtained first upsampling feature may also be subjected to convolution processing, batch normalization (BN) processing and / or LeakyReLU activation function processing, etc. Similarly, after the second upsampling network is used to perform upsampling decoding processing on the first upsampling feature, convolution processing and tanh activation function processing may also be performed to obtain the above-mentioned flame transparency mask.

[0084] Optionally, as an optional implementation of the above step S130, for example: the flame transparency mask obtained above is multiplied with the image to be detected to obtain the above flame feature.

[0085] As an optional implementation of the above step S140, the implementation of using the classifier of the neural network model to classify the flame characteristics may include:

[0086] Step S141: Fusing the image to be detected and the flame feature to obtain a fused feature.

[0087] The implementation method of the above step S141 is, for example: performing a concatenation and fusion operation on the image to be detected and the flame feature, so as to obtain the above fusion feature.

[0088] Step S142: Use the classifier of the neural network model to classify the fused features to obtain the fire detection result of the image to be detected.

[0089] The implementation method of the above step S142 is, for example: using a classifier of a neural network model to classify flame features to obtain a fire detection result of the image to be detected. It can be understood that using the flame features determined by the flame transparency mask as the input of the neural network model can effectively improve the sensitivity of the classifier to the flame features, thereby improving the accuracy and robustness of the fire detection task.

[0090] As an optional implementation of the above method, before using the encoder of the neural network model to encode the image to be detected, the encoder and decoder of the neural network model may also be trained. This implementation may include:

[0091] Step S210: obtaining a plurality of sample images and a transparency mask label corresponding to each sample image.

[0092] The implementation of step S210 is described in detail below.

[0093] Step S220: Use the encoder of the neural network model to encode the sample images in the training data set to obtain the encoding features of the sample images.

[0094] Step S230: Use the decoder of the neural network model to decode the encoded features of the sample image to obtain a sample transparency mask.

[0095] The implementation of the above steps S220 to S230 is similar to the implementation of the above steps S110 to S120. Therefore, if there are any unclear points, please refer to the implementation of the above steps S110 to S120, and no further details will be given here.

[0096] Step S240: Calculate the mask loss value between the sample transparency mask and the transparency mask label.

[0097] Step S250: Update the weight parameters of the encoder and the weight parameters of the decoder according to the mask loss value.

[0098] It is understandable that the calculation process of the mask loss value in the above-mentioned step S240 and step S250 is relatively complicated, and therefore, it will be introduced in detail after the introduction of step S210 below.

[0099] As an optional implementation of the above step S210, in order to enhance the neural network model's ability to extract features of multiple flames at different scales in a fire scene, a plurality of synthesized sample images and a transparency mask label corresponding to each sample image may be used to train the neural network model. The synthesis method of the sample images and the transparency mask label may include:

[0100] Step S211: obtaining a background image, a plurality of flame foreground images and a transparency mask corresponding to each flame foreground image.

[0101] For example, the above-mentioned step S211 may be implemented as follows: the above-mentioned background image may be represented as B, and the height and width of the background image are calculated. The height and width of the background image may be represented as H, respectively. b and W b The above multiple flame foreground images can be represented as f1,f2,f3,...,f n , for each flame foreground f i ,i∈{1,2,3,..,n}, the height and width are uniformly scaled to the same size H f and W f Similarly, the height and width of the transparency mask corresponding to each flame foreground image are also uniformly scaled to the same size.

[0102] Step S212: Calculate the size ratio of the background image and a flame foreground image, and generate a scale factor set according to the size ratio.

[0103] The implementation of the above step S212 is, for example, to calculate the size ratio of the flame foreground to the background image, including: the height ratio is The width ratio is Among them, r h ,r w ∈{0,1,2,3,...,N}. Then, a scale factor set is generated according to the size ratio, which can be expressed as S←{0.25,0.5,0.75,1,2,3,...,max{r h ,r w}}, where max{r h ,r w} means taking the maximum value of the height ratio and width ratio of the flame foreground relative to the background image.

[0104] Step S213: for each flame foreground image among the multiple flame foreground images, scale the flame foreground image according to a scale factor randomly selected from a scale factor set to obtain multiple scaled foreground images, and scale the transparency mask corresponding to the flame foreground image according to the scale factor to obtain scaled transparency masks corresponding to the multiple scaled foreground images.

[0105] In the implementation of the above step S213, for example, the scale factor randomly selected from the scale factor set can be expressed as s←rand(S), and the flame foreground image is scaled according to the scale factor, which can be expressed as MS(f i ,H f ,Wf ,s), and obtain multiple scaled foreground images. Among them, S represents the scale factor set, rand represents the random screening operation, s represents the randomly screened scale factor, H f and W f represents the height and width of the flame foreground image, f i represents the i-th flame foreground image, and MS represents the transformation function of the multi-scale scaling operation. Similarly, the transparency mask corresponding to the flame foreground image is scaled according to the scale factor to obtain scaled transparency masks corresponding to multiple scaled foreground images.

[0106] Step S214: for each of the multiple scaled foreground images, synthesize the background image, the scaled foreground image and the scaled transparency mask corresponding to the scaled foreground image to obtain multiple flame images of different scales, and determine the multiple flame images of different scales as sample images.

[0107] The implementation of the above step S214 is, for example, represented by a pseudo code of synthesizing the background image, the scaled foreground image, and the scaled transparency mask corresponding to the scaled foreground image as follows:

[0108] ms_h←randint(0,r h ),ms_w←randint(0,r w ), / / Randomly select a window position

[0109] start_y←min(ms_h*H f ),start_x←min(ms_w*W f ) / / Calculate the position of the starting window

[0110] B x,y =I ms [start_y:start_y+H f ,start_x:start_x+W f ] / / Take the corresponding window as the background

[0111] I ms [start_y:start_y+H f ,start_x:start_x+W f ] = f n *α n +(1-α n )B x,y / / synthesis

[0112] Among them, ms_h represents the high index number of the window position, ms_w represents the wide index number of the window position, randint represents randomly selecting an integer, and rh represents the high-scale ratio, r w Indicates the width scale ratio, start_x indicates the horizontal coordinate of the start window position, start_y indicates the vertical coordinate of the start window position, min indicates the minimize operation, H f and W f represents the height and width of the flame foreground image, B x,y Indicates that the area image corresponding to the window in the background image is used as the synthetic background, f n represents the nth scaled foreground image, α n Indicates the scaled transparency mask corresponding to the nth scaled foreground image, I ms It represents the multi-flame images of different scales obtained by synthesis, that is, the multi-flame image here can be determined as the sample image above.

[0113] Step S215: for each scaled transparency mask corresponding to the scaled foreground image, the maximum transparency value is taken for the pixel points at the same position between the scaled transparency mask and the scaled transparency mask participating in the last synthesis, so as to obtain the transparency mask label corresponding to the sample image.

[0114] For example, the pseudo code of the implementation of the above step S215 is: taking the maximum transparency value of the pixel at the same position in the scaled transparency mask and the scaled transparency mask participating in the previous synthesis can be expressed as:

[0115]

[0116] in, Indicates the last scaled transparency mask involved in the synthesis, α n Indicates the scaling transparency mask that needs to be synthesized this time (i.e. the scaling transparency mask above), (x, y) indicates the coordinates of the pixel point in the scaling transparency mask that is at the same position as the scaling transparency mask that participated in the previous synthesis, α ms Indicates the transparency mask label corresponding to the sample image, start_x indicates the horizontal coordinate of the start window position, start_y indicates the vertical coordinate of the start window position, H f and W f Indicates the height and width of the flame foreground image.

[0117] It can be understood that the flame transparency mask synthesis rule above is that, assuming that the transparency of the entire image after the nth synthesis is α n , and the transparency of the foreground will change as the images are gradually superimposed during the synthesis process. For example, if the first synthesized image is I1, then the transparency of the first flame foreground synthesized by the image is α1, and the transparency of the background is 1-α1. Therefore, the transparency mask of the image can be expressed as For the second synthesized image I2, the transparency mask of the image can be expressed as The flame foreground of the second synthesized image is represented by f2, and the corresponding transparency of this image is represented by α2, while the transparency of the previously synthesized image I1 is 1-α2 (because the sum of the transparency of the foreground and background is 1). Similarly, the transparency mask of the nth image, the current foreground f n Corresponding transparency mask α n , and the transparency of the first n-1 synthesized There is the following recursion

[0118] In the actual practice, it was found that when multiple flames were synthesized, these transparency masks would have overlapping areas. Therefore, some trade-offs were made while choosing to remove background interference while preserving the complete flames as much as possible. In other words, a certain degree of flame transparency is allowed to be higher than the original degree. Such an operation may cause some background to be mixed in the foreground of the flame, but it can minimize the omission of flame features in the fire scene, which helps to improve the accuracy of subsequent classification tasks. We use pixel-level and operation (that is, the maximum transparency of each pixel position) to handle the overlap of transparency masks during each synthesis. The following formula can be used:

[0119]

[0120] in, Indicates the transparency of the previous n synthesis. Indicates the transparency of the previous n-1 synthesis, α n The transparency mask of the nth composited image.

[0121] As an optional implementation of the above steps S240 and S250, a loss function including a penalty coefficient may be used to calculate the mask loss value between the sample transparency mask and the transparency mask label, wherein the loss function including the penalty coefficient may be expressed as:

[0122]

[0123] Among them, l m represents a loss function containing a penalty coefficient for not missing flames, which is used to calculate the mask loss value between the sample transparency mask and the transparency mask label. represents the sample transparency mask predicted by the neural network model, α represents the transparency mask label, which refers to the real flame foreground transparency mask, δ represents the first penalty coefficient for the prediction result in the positive sample image containing less flame, max(a)≠0 represents the sample image is a positive sample image with flame, max(a)=0 represents the sample image is a negative sample image without flame, and ω represents the second penalty coefficient for the prediction result in the negative sample image containing more flame.

[0124] The meaning of the above loss function including the penalty coefficient is interpreted as follows:

[0125] The meaning of the first equation of the above loss function is: if the sample image is a positive sample image with flames, and the transparency value of the pixel in the positive sample image in the sample transparency mask is less than the transparency value in the transparency mask label, then the mask loss value is calculated according to the preset first penalty coefficient, the sample transparency mask and the transparency mask label, and the first penalty coefficient represents the coefficient for penalizing the neural network model when there are fewer predictions in the positive sample image.

[0126] The second equation of the above loss function means that if the sample image is a positive sample image with flames, and the transparency value of the pixel in the positive sample image in the sample transparency mask is greater than or equal to the transparency value in the transparency mask label, the mask loss value is set to zero.

[0127] The meaning of the third equation of the above loss function is: if the sample image is a negative sample image in which there is no flame, and the transparency value of the pixel in the negative sample image in the sample transparency mask is greater than the transparency value in the transparency mask label, then the mask loss value is calculated according to the preset second penalty coefficient, the sample transparency mask and the transparency mask label, the second penalty coefficient represents the coefficient for penalizing the neural network model when multiple predictions are made in the negative sample image, and the first penalty coefficient is greater than the second penalty coefficient.

[0128] Then, according to the mask loss value calculated by the above loss function, the weight parameters of the encoder and the weight parameters of the decoder are updated until the encoder and decoder tend to converge, and the trained encoder and decoder can be obtained.

[0129] The present application also provides a fire detection device, including:

[0130] The coding feature acquisition module is used to use the encoder of the neural network model to encode the image to be detected to obtain the image coding features. The encoder includes: at least one suspected flame adaptive module SFAM network.

[0131] The feature decoding processing module is used to decode the image coding features using the decoder of the neural network model to obtain the flame transparency mask.

[0132] The flame feature acquisition module is used to extract features of the image to be detected using a flame transparency mask to obtain flame features.

[0133] The detection result acquisition module is used to classify the flame features using the classifier of the neural network model to obtain the fire detection result of the image to be detected.

[0134] As an optional implementation of the above device, the encoder includes: a convolutional layer, a first SFAM network and a second SFAM network; and a coding feature acquisition module includes:

[0135] The image convolution processing submodule is used to perform convolution processing on the image to be detected using a convolution layer to obtain convolution features.

[0136] The first feature extraction submodule is used to extract the region of interest features from the convolutional features using the first SFAM network.

[0137] The first feature extraction submodule is used to use the second SFAM network to perform feature extraction and downsampling processing on the features of the region of interest to obtain image coding features. By performing feature extraction and downsampling processing on the features of the region of interest, the calculation amount of the cutout model is effectively reduced.

[0138] As an optional implementation of the above device, the first SFAM network includes: a flame contour branch and a flame transparency branch; the first feature extraction submodule includes:

[0139] The first contour feature extraction unit is used to extract the convolution feature using the flame contour branch to obtain the flame contour feature.

[0140] The first transparency feature extraction unit is used to extract the convolution feature using the flame transparency branch to obtain the flame transparency feature.

[0141] The flame feature fusion unit is used to fuse the flame contour feature and the flame transparency feature to obtain the feature of the region of interest.

[0142] As an optional implementation of the above device, the flame contour branch includes: two first convolution filters and an average pooling layer; the first contour feature extraction unit includes:

[0143] The first convolution filtering subunit is used to use one of the two first convolution filters to perform convolution processing on the convolution feature to obtain the image convolution feature.

[0144] The image pooling processing subunit is used to perform pooling processing on the image convolution features using the average pooling layer to obtain image boundary features.

[0145] The first convolution processing subunit is used to use another first convolution filter of the two first convolution filters to perform convolution processing on the image boundary feature to obtain the flame outline feature.

[0146] As an optional implementation of the above device, the flame transparency branch includes: a first convolution filter, a second convolution filter and a compression excitation SE channel attention layer; a first transparency feature extraction unit includes:

[0147] The second convolution filtering subunit is used to perform convolution processing on the convolution feature using the second convolution filter to obtain the image convolution feature.

[0148] The channel attention processing subunit is used to process the image convolution features using the SE channel attention layer to obtain the channel attention features.

[0149] The second convolution processing subunit is used to perform convolution processing on the channel attention feature using the first convolution filter to obtain the flame transparency feature.

[0150] As an optional implementation of the above device, the second SFAM network includes: a downsampling branch, a flame contour branch and a flame transparency branch; the first feature extraction submodule includes:

[0151] The second contour feature extraction unit is used to extract features of the region of interest using the flame contour branch to obtain flame contour features.

[0152] The second transparency feature extraction unit is used to extract the flame outline feature using the flame transparency branch to obtain the flame transparency feature.

[0153] The flame feature downsampling processing unit is used to use the downsampling branch to downsample the flame contour feature and the flame transparency branch to obtain the image coding feature.

[0154] As an optional implementation of the above device, the detection result obtaining module includes:

[0155] The image feature fusion submodule is used to fuse the image to be detected and the flame feature to obtain the fusion feature.

[0156] The fusion feature classification submodule is used to classify the fusion features using the classifier of the neural network model to obtain the fire detection result of the image to be detected.

[0157] As an optional implementation of the above device, the fire detection device further includes:

[0158] An image label acquisition module, used to acquire multiple sample images and a transparency mask label corresponding to each sample image;

[0159] A coding feature acquisition module is used to encode sample images in a training data set using an encoder of a neural network model to obtain coding features of the sample images;

[0160] A transparency mask obtaining module is used to decode the encoded features of the sample image using a decoder of the neural network model to obtain a sample transparency mask;

[0161] A mask loss calculation module is used to calculate the mask loss value between the sample transparency mask and the transparency mask label;

[0162] The weight parameter updating module is used to update the weight parameters of the encoder and the weight parameters of the decoder according to the mask loss value.

[0163] As an optional implementation of the above device, the image tag acquisition module includes:

[0164] An image mask acquisition submodule is used to acquire a background image, multiple flame foreground images and a transparency mask corresponding to each flame foreground image;

[0165] A size set generation submodule is used to calculate the size ratio of the background image and a flame foreground image, and generate a scale factor set according to the size ratio;

[0166] An image mask scaling submodule is used to scale each flame foreground image among the multiple flame foreground images according to a scale factor randomly selected from a scale factor set to obtain multiple scaled foreground images, and to scale a transparency mask corresponding to the flame foreground image according to the scale factor to obtain scaled transparency masks corresponding to the multiple scaled foreground images;

[0167] An image mask synthesis submodule is used to synthesize the background image, the zoomed foreground image and the zoomed transparency mask corresponding to the zoomed foreground image for each zoomed foreground image in the multiple zoomed foreground images to obtain multiple flame images of different scales, and determine the multiple flame images of different scales as sample images;

[0168] The mask label acquisition submodule is used to obtain the transparency mask label corresponding to the sample image by taking the maximum transparency value of the pixel point with the same position in the scaled transparency mask and the scaled transparency mask participating in the last synthesis for each scaled foreground image.

[0169] As an optional implementation of the above device, the mask loss calculation module includes:

[0170] A first mask loss calculation submodule, for calculating a mask loss value according to a preset first penalty coefficient, the sample transparency mask and the transparency mask label if the sample image is a positive sample image with flames, and the transparency value of the pixel in the positive sample image in the sample transparency mask is less than the transparency value in the transparency mask label, wherein the first penalty coefficient represents a coefficient for penalizing the neural network model when there is less prediction in the positive sample image;

[0171] A second mask loss calculation submodule, configured to set the mask loss value to zero if the sample image is a positive sample image with flames, and the transparency value of the pixel in the positive sample image in the sample transparency mask is greater than or equal to the transparency value in the transparency mask label;

[0172] The third mask loss calculation submodule is used to calculate the mask loss value according to the preset second penalty coefficient, the sample transparency mask and the transparency mask label if the sample image is a negative sample image in which there is no flame, and the transparency value of the pixel in the negative sample image in the sample transparency mask is greater than the transparency value in the transparency mask label. The second penalty coefficient represents the coefficient for penalizing the neural network model when multiple predictions are made in the negative sample image, and the first penalty coefficient is greater than the second penalty coefficient.

[0173] It should be understood that the device corresponds to the above-mentioned fire detection method embodiment and can execute the various steps involved in the above-mentioned method embodiment. The specific functions of the device can be referred to in the above description, and the detailed description is appropriately omitted here. The device includes at least one software function module that can be stored in a memory in the form of software or firmware or fixed in the operating system (OS) of the device.

[0174] An embodiment of the present application further provides an electronic device, including: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the machine-readable instructions are executed by the processor, the above method is performed.

[0175] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. Wherein, the computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk.

[0176] The embodiment of the present application also provides a computer program product, including: a computer program or a computer instruction, and the computer program or the computer instruction executes the method described above when executed by a processor.

[0177] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0178] In several embodiments provided in the embodiments of the present application, it should be understood that the disclosed devices and methods may also be implemented in other ways. The device embodiments described above are only schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code includes one or more executable instructions for implementing a specified logical function.

[0179] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.

Claims

1. A fire detection method, characterized in that: include: An encoder of a neural network model is used to encode the image to be detected to obtain image encoding features, wherein the encoder includes: at least one suspected flame adaptive module SFAM network; Using the decoder of the neural network model to decode the image coding features to obtain a flame transparency mask; Using the flame transparency mask to extract features from the image to be detected to obtain flame features; The flame features are classified using the classifier of the neural network model to obtain a fire detection result of the image to be detected.

2. The method according to claim 1, characterized in that The encoder comprises: a convolutional layer, a first SFAM network and a second SFAM network; the encoder using the neural network model performs encoding processing on the image to be detected, comprising: Using the convolution layer to perform convolution processing on the image to be detected to obtain convolution features; Extracting region of interest features from the convolutional features using the first SFAM network; The second SFAM network is used to perform feature extraction and down-sampling processing on the features of the region of interest to obtain the image coding features.

3. The method according to claim 2, characterized in that The first SFAM network comprises: a flame contour branch and a flame transparency branch; and the step of extracting a region of interest feature from the convolutional feature using the first SFAM network comprises: Using the flame contour branch to perform feature extraction on the convolution feature to obtain a flame contour feature; Using the flame transparency branch to perform feature extraction on the convolution feature to obtain a flame transparency feature; The flame outline feature and the flame transparency feature are fused to obtain the region of interest feature.

4. The method according to claim 3, characterized in that The flame outline branch includes: two first convolution filters and an average pooling layer; the use of the flame outline branch to extract features from the convolution feature includes: Using one of the two first convolution filters to perform convolution processing on the convolution feature to obtain an image convolution feature; Using the average pooling layer to perform pooling processing on the image convolution features to obtain image boundary features; The image boundary feature is convolved using another first convolution filter of the two first convolution filters to obtain the flame outline feature.

5. The method according to claim 3, characterized in that: The flame transparency branch includes: a first convolution filter, a second convolution filter and a compression excitation SE channel attention layer; the use of the flame transparency branch to extract features from the convolution features includes: Using the second convolution filter to perform convolution processing on the convolution feature to obtain an image convolution feature; Using the SE channel attention layer to process the image convolution features to obtain channel attention features; The channel attention feature is convolved using the first convolution filter to obtain the flame transparency feature.

6. The method according to claim 2, characterized in that The second SFAM network comprises: a downsampling branch, a flame contour branch and a flame transparency branch; the step of using the second SFAM network to perform feature extraction and downsampling processing on the features of the region of interest comprises: Using the flame contour branch to extract features of the region of interest, to obtain flame contour features; Using the flame transparency branch to extract the flame contour feature to obtain a flame transparency feature; The flame contour feature and the flame transparency branch are downsampled using the downsampling branch to obtain the image coding feature.

7. The method according to any one of claims 1 to 6, characterized in that: Before the encoder using the neural network model encodes the image to be detected, it also includes: Obtaining a plurality of sample images and a transparency mask label corresponding to each of the sample images; Using the encoder of the neural network model to encode the sample images in the training data set to obtain encoding features of the sample images; Decoding the encoded features of the sample image using a decoder of the neural network model to obtain a sample transparency mask; Calculating a mask loss value between the sample transparency mask and the transparency mask label; The weight parameters of the encoder and the weight parameters of the decoder are updated according to the mask loss value.

8. The method according to claim 7, characterized in that The step of obtaining a plurality of sample images and a transparency mask label corresponding to each of the sample images includes: Obtain a background image, a plurality of flame foreground images and a transparency mask corresponding to each of the flame foreground images; Calculating the background image and one of the flame foreground images to obtain a size ratio, and generating a scale factor set according to the size ratio; For each flame foreground image among the multiple flame foreground images, scaling the flame foreground image according to a scale factor randomly selected from the scale factor set to obtain multiple scaled foreground images, and scaling a transparency mask corresponding to the flame foreground image according to the scale factor to obtain multiple scaled transparency masks corresponding to the scaled foreground images; For each scaled foreground image in the plurality of scaled foreground images, synthesizing the background image, the scaled foreground image and a scaled transparency mask corresponding to the scaled foreground image to obtain multiple flame images of different scales, and determining the multiple flame images of different scales as the sample images; For each scaled transparency mask corresponding to the scaled foreground image, the maximum transparency value is taken for the pixel points at the same position between the scaled transparency mask and the scaled transparency mask participating in the last synthesis, so as to obtain the transparency mask label corresponding to the sample image.

9. The method according to claim 7, characterized in that: The calculating the mask loss value between the sample transparency mask and the transparency mask label includes: If the sample image is a positive sample image with flames, and the transparency value of the pixel in the positive sample image in the sample transparency mask is less than the transparency value in the transparency mask label, the mask loss value is calculated according to a preset first penalty coefficient, the sample transparency mask and the transparency mask label, wherein the first penalty coefficient represents a coefficient for penalizing the neural network model when there is less prediction in the positive sample image; If the sample image is a positive sample image with flames, and the transparency value of the pixel in the positive sample image in the sample transparency mask is greater than or equal to the transparency value in the transparency mask label, then the mask loss value is set to zero; If the sample image is a negative sample image in which no flame exists, and the transparency value of the pixel in the negative sample image in the sample transparency mask is greater than the transparency value in the transparency mask label, then the mask loss value is calculated based on a preset second penalty coefficient, the sample transparency mask and the transparency mask label, the second penalty coefficient represents a coefficient for penalizing the neural network model when multiple predictions are made in the negative sample image, and the first penalty coefficient is greater than the second penalty coefficient.

10. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the machine-readable instructions are executed by the processor to perform any method according to claims 1 to 9.

Images