Flame and smoke detection method, apparatus and storage medium

Abstract

A flame and smoke detection method, device and storage medium, the method comprising: acquiring an image to be detected; using a trained detection model to perform flame detection and / or smoke detection on the image, and outputting classification and positioning results of the flame and / or smoke in the image; wherein the detection model comprises an improved network obtained by improving a yolov5 network, and the improvement of the yolov5 network comprises: replacing a spatial pyramid pooling module in a backbone network of the yolov5 network with a multi-scale attention module. The flame and smoke detection method and device according to the embodiments of the present application use an improved yolov5 network comprising a multi-scale attention module to perform flame and smoke detection on an image, and have high capturing capability for flames with variable shapes and scales and / or smoke generated when objects are not completely burned, so as to quickly and accurately judge and locate the flame and / or smoke in the image.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and more specifically to a flame and smoke detection method, apparatus and storage medium. Background Technology

[0002] Fire is a serious natural disaster and social catastrophe. On the one hand, fire poses a great threat to human life and property safety; on the other hand, it also causes enormous damage to the natural, socio-economic, and social environment. Therefore, preventing and reducing the occurrence of fires as much as possible has always been a topic of widespread concern.

[0003] Initial flame detection methods were sensor-based, which can be categorized into five types based on different sensor types and applications: light sensors, temperature sensors, smoke sensors, gas sensors, and synthetic sensors. Temperature and smoke sensors are the most commonly used due to the heat release and smoke concentration characteristics of fires. However, this sensor-based flame detection method has significant limitations in terms of detection range and speed.

[0004] With the development of video surveillance technology, flame detection methods based on traditional image processing have emerged. This method detects flames based on information such as color, deformation, texture structure, and jitter in the flame image, effectively improving the speed of flame detection. However, this traditional image processing method requires fusing information from multiple image frames, consuming significant computational resources for image preprocessing, making it difficult to meet the demands of real-time feedback. Furthermore, traditional image algorithms require parameter adjustments to adapt to different scenarios, resulting in unsatisfactory performance in the complex and ever-changing flame scenes of reality.

[0005] In recent years, thanks to the emergence of convolutional neural networks (CNNs), computer vision technology has developed rapidly. Object detection, in particular, has seen rapid advancements, with many object detection algorithms demonstrating satisfactory results. Currently, detection algorithms can be categorized into two structures: two-stage and single-stage. Representative networks in two-stage detection algorithms include Fast Region Convolutional Neural Networks (Fast-RCNN), Faster Region Convolutional Neural Networks (Faster-RCNN), and Masked Region Convolutional Neural Networks (Mask-RCNN). Specifically, the network first extracts features to obtain candidate regions, and then classifies these regions using deep convolutional blocks. However, because this type of detection algorithm requires two stages of feature processing, it is difficult to achieve real-time detection performance on edge chips with limited computing resources. In single-stage detection algorithms, the Single Shot Multi-Box Object Detection (SSD) series and the YOLO (You Only Look Once) series have attracted significant attention. They combine the two-stage feature processing into a single stage, meaning the algorithm simultaneously calculates the candidate region and the target category within that region in one stage, which greatly improves the speed of the detection network.

[0006] Based on this, there are currently deep learning-based flame and smoke detection algorithms, but these algorithms mainly encode feature information through standard convolutions and lack the ability to extract key features, which limits the network's representation capabilities and makes it limited in its ability to capture flames and smoke with varying shapes and scales. Summary of the Invention

[0007] According to one aspect of this application, a flame and smoke detection method is provided, the method comprising: acquiring an image to be detected; performing flame detection and / or smoke detection on the image using a trained detection model, and outputting the classification and localization results of the flame and / or smoke in the image; wherein the detection model comprises an improved network obtained by improving the YOLOv5 network, wherein the improvement of the YOLOv5 network comprises: replacing the spatial pyramid pooling module in the backbone network of the YOLOv5 network with a multi-scale attention module.

[0008] In one embodiment of this application, the multi-scale attention module includes a first branch, a second branch, and a third branch, wherein: the first branch is used to calculate attention weights on the input feature map to obtain a first calculation result, the input feature map being the feature map extracted by the backbone network from the image; the second branch is used to perform a nonlinear transformation on the input feature map and multiply the nonlinear transformation result and the first calculation result to obtain a second calculation result; the third branch is used to perform a matrix addition operation on the input feature map and the second calculation result to obtain an output feature map.

[0009] In one embodiment of this application, the improvement to the YOLOv5 network further includes: replacing the standard convolutional modules in the backbone network of the YOLOv5 network with depthwise separable convolutional modules; the first branch performs attention weight calculation on the input feature map to obtain a first calculation result, including: performing a nonlinear transformation on the input feature map to obtain a nonlinear transformation result; performing max pooling on the nonlinear transformation result at multiple different scales and upsampling it to the same resolution as the input feature map to obtain multiple upsampling results; fusing the multiple upsampling results to obtain a fused feature map; and sequentially performing depthwise separable convolution, batch normalization, global average pooling, and scale scaling on the fused feature map to obtain the first calculation result.

[0010] In one embodiment of this application, the kernel sizes used for the multiple max pooling operations of different scales are 3x3, 5x5, and 7x7, respectively; and the kernel size used for the depthwise separable convolution is 3x3.

[0011] In one embodiment of this application, the improvement to the YOLOv5 network further includes replacing the center point and scale prediction module in the backbone network of the YOLOv5 network with a ghosting module; wherein the input feature map is obtained by the ghosting module processing the image.

[0012] In one embodiment of this application, the improvement to the YOLOv5 network further includes at least one of the following: replacing convolutional modules with more than 256 channels in the backbone network of the YOLOv5 network with convolutional modules with 256 channels; replacing the center point and scale prediction modules in the neck network of the YOLOv5 network with ghosting modules, and replacing the standard convolutional modules in the neck network with depth-separable convolutional modules; and expanding the three output branches in the head network of the YOLOv5 network to four output branches.

[0013] In one embodiment of this application, training the detection model includes: collecting sample images containing flames and / or smoke; labeling and data augmenting the sample images to obtain a training dataset; preprocessing the data in the training dataset to obtain a preprocessed dataset; building a virtual environment required for training the model on a server; and training the improved network in the virtual environment based on the preprocessed dataset to obtain the trained detection model.

[0014] According to another aspect of this application, a flame and smoke detection device is provided, the device including a memory and a processor, the memory storing a computer program executed by the processor, the computer program, when executed by the processor, causing the processor to perform the above-described flame and smoke detection method.

[0015] In one embodiment of this application, the application scenarios of the device include at least one of the following: forest fire prevention, factory fire prevention, warehouse fire prevention, residential fire prevention, and shopping mall fire prevention.

[0016] According to another aspect of this application, a storage medium is provided, wherein a computer program executed by a processor is stored on the storage medium, and the computer program, when executed by the processor, causes the processor to perform the above-described flame and smoke detection method.

[0017] The flame and smoke detection method and apparatus according to the embodiments of this application use an improved YOLOv5 network including a multi-scale attention module to detect flames and smoke in images. It has a high capture capability for flames and / or smoke generated when objects are not completely burned due to varying shapes and scales, thereby enabling rapid and accurate identification and location of flames and / or smoke in images. Attached Figure Description

[0018] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0019] Figure 1 A schematic block diagram of an example electronic device for implementing a flame and smoke detection method and apparatus according to embodiments of the present invention is shown.

[0020] Figure 2 A schematic flowchart of a flame and smoke detection method according to an embodiment of this application is shown.

[0021] Figure 3 This document illustrates an exemplary process flow diagram of the detection model used in the flame and smoke detection method according to an embodiment of this application, from training to application.

[0022] Figure 4 The diagram illustrates the model structure and operation schematic of the improved network used in the flame and smoke detection method according to an embodiment of this application.

[0023] Figure 5This diagram illustrates the structure and operation of a multi-scale attention module in a model of an improved network used in a flame and smoke detection method according to an embodiment of this application.

[0024] Figure 6 A schematic structural block diagram of a flame and smoke detection device according to an embodiment of this application is shown. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0026] First, refer to Figure 1 This describes an example electronic device 100 for implementing the flame and smoke detection method and apparatus of embodiments of the present invention.

[0027] like Figure 1 As shown, the electronic device 100 includes one or more processors 102, one or more storage devices 104, input devices 106, and output devices 108, which are interconnected via a bus system 110 and / or other forms of connection mechanisms (not shown). It should be noted that... Figure 1 The components and structure of the electronic device 100 shown are merely exemplary and not limiting; the electronic device may also have other components and structures as needed.

[0028] The processor 102 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0029] The storage device 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may execute the program instructions to implement the client functions (implemented by the processor) in the embodiments of the present invention described below, and / or other desired functions. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the applications.

[0030] The input device 106 can be a device used by a user to input commands, and may include one or more of a keyboard, mouse, microphone, and touchscreen. Furthermore, the input device 106 can also be any interface for receiving information.

[0031] The output device 108 can output various information (e.g., images or sounds) to the outside (e.g., a user), and may include one or more of a display, speaker, etc. Furthermore, the output device 108 can also be any other device with output functionality.

[0032] For example, an example electronic device for implementing the flame and smoke detection method and apparatus according to embodiments of the present invention can be implemented as a terminal such as a smartphone, tablet computer, camera, etc.

[0033] Below, we will refer to Figure 2 A flame and smoke detection method 200 according to an embodiment of this application is described. For example... Figure 2 As shown, the flame and smoke detection method 200 may include the following steps:

[0034] In step S210, the image to be detected is acquired.

[0035] In step S220, the trained detection model is used to perform flame detection and / or smoke detection on the image, and the classification and localization results of flame and / or smoke in the image are output. The detection model includes an improved network obtained by improving the YOLOv5 network. The improvement of the YOLOv5 network includes replacing the spatial pyramid pooling module in the backbone network of the YOLOv5 network with a multi-scale self-attention (MSA) module.

[0036] In the embodiments of this application, an improved network obtained by modifying the YOLOv5 network is used as the detection model for flame detection and / or smoke detection. The improvement of the YOLOv5 network includes replacing the Spatial Pyramid Pooling (SPP) module in the backbone network of the YOLOv5 network with a multi-scale attention module. The multi-scale attention module can effectively extract target information of various scales and filter key features, so that the YOLOv5 network detection model including this module has a high capture capability for flames and / or smoke generated when objects are not completely burned (at which time no flame has been generated) with varying shapes and scales. Thus, it can quickly and accurately determine and locate whether there are flames and / or smoke in the image to be detected (here, image can refer to pictures or videos). When used in fire early warning systems, it can effectively improve the reliability and safety of the system and has a wide range of application scenarios and market value.

[0037] In the embodiments of this application, the aforementioned multi-scale attention module may include a first branch, a second branch, and a third branch, wherein: the first branch is used to calculate attention weights on the input feature map to obtain a first calculation result, the input feature map being the feature map extracted from the image to be detected by the backbone network of the aforementioned detection model; the second branch is used to perform a nonlinear transformation on the input feature map and multiply the nonlinear transformation result and the first calculation result to obtain a second calculation result; the third branch is used to perform a matrix addition operation on the input feature map and the second calculation result to obtain an output feature map. In this embodiment, the first branch can complete the calculation of attention weights on the feature map in the channel dimension; the second branch uses attention weights to represent the effectiveness of the current channel features and completes the screening of key features; the third branch performs a matrix addition operation on the attention-weighted features to avoid network degradation caused by gradient vanishing or explosion, and finally obtains an output feature map with the same scale as the input feature map. This will be discussed later in conjunction with... Figure 5 The structure and operation of the multi-scale attention module are described exemplarily.

[0038] In embodiments of this application, improvements to the YOLOv5 network may further include: replacing the standard convolutional modules in the backbone network of the YOLOv5 network with depthwise separable convolution (DSC) modules; based on this, the aforementioned first branch performs attention weight calculation on the input feature map to obtain a first calculation result, including: performing a nonlinear transformation on the input feature map to obtain a nonlinear transformation result; performing max pooling on the nonlinear transformation result at multiple different scales and upsampling it to the same resolution as the input feature map to obtain multiple upsampling results; fusing the multiple upsampling results to obtain a fused feature map; and sequentially performing depthwise separable convolution, batch normalization, global average pooling, and scale scaling on the fused feature map to obtain the first calculation result. In this embodiment, the input feature map undergoes a nonlinear transformation to achieve stronger representational capabilities; fusing multiple upsampling results yields a feature map with multi-scale feature representations; performing depthwise separable convolution on the fused feature map combined with batch normalization further enriches the feature representation; global average pooling is used to compress the effectiveness of global information in each channel into a single point for identification, and then an activation function is used to scale the feature point, completing the attention weight calculation of the feature map in the channel dimension. This will be discussed later in conjunction with... Figure 5 The structure and operation of the first branch in the multi-scale attention module are described exemplarily.

[0039] In embodiments of this application, improvements to the YOLOv5 network may further include replacing the Center and Scale Prediction (CSP) module in the backbone network of the YOLOv5 network with a Ghost module; based on this, the input feature map mentioned above can be obtained by processing the image to be detected using the Ghost module. Further improvements to the YOLOv5 network may also include replacing the Center and Scale Prediction module in the neck network of the YOLOv5 network with a Ghost module, and replacing the standard convolutional module in the neck network with a depthwise separable convolutional module. In this embodiment, by replacing the standard convolutional structure and CSP module in the YOLOv5 network with a depthwise separable convolutional structure and a Ghost module respectively, the number of network parameters can be reduced while maintaining network accuracy and performance, thereby improving the detection efficiency of flames and / or smoke without affecting detection reliability. This will be discussed later in conjunction with... Figure 4 The structure and operation of the improved network obtained by improving the YOLOv5 network are described exemplarily.

[0040] In embodiments of this application, improvements to the YOLOv5 network may further include expanding the three output branches in the head network of the YOLOv5 network to four output branches. In this embodiment, improving the head network of the YOLOv5 network to four output branches enhances the network's ability to fit multi-scale targets and accelerates network convergence. In the application scenario of this application, it can improve the detection model's ability to fit multi-scale flames and multi-scale smoke, and accelerate the convergence speed of models used for flame and smoke detection. In similar embodiments, the three output branches in the head network of the YOLOv5 network can also be expanded to other numbers of branches to improve the network's ability to fit multi-scale targets. This will be discussed later in conjunction with... Figure 4 The structure and operation of the improved network obtained by improving the YOLOv5 network are described exemplarily.

[0041] In embodiments of this application, improvements to the YOLOv5 network may further include: replacing convolutional modules with more than 256 channels in the backbone network of the YOLOv5 network with convolutional modules of 256 channels. This reduces the number of network parameters while maintaining network accuracy and performance, thereby improving the detection efficiency of flames and / or smoke without affecting detection reliability. This will be discussed later in conjunction with... Figure 4 The structure and operation of the improved network obtained by improving the YOLOv5 network are described exemplarily.

[0042] In embodiments of this application, training the improved network, i.e., the detection model, may include: collecting sample images containing flames and / or smoke; labeling and data augmenting the sample images to obtain a training dataset; preprocessing the data in the training dataset to obtain a preprocessed dataset; building a virtual environment on a server required for training the model; and training the improved network in the virtual environment based on the preprocessed dataset to obtain a trained detection model. Based on the trained detection model, flames and / or smoke can be detected in the input image, and the classification results and location coordinates of the flames and / or smoke in the image can be obtained.

[0043] The following is combined with Figure 3 This describes, exemplarily, the process from training to application of the detection model in the flame and smoke detection method of this application, and combines... Figure 4 and Figure 5 Describe the structure and operation examples of the network model obtained after improving the YOLOv5 network.

[0044] like Figure 3As shown, to train a detection model for flame and smoke detection, the following steps are taken: First, web crawlers can be used to collect and label images and videos to build a dataset. Next, the data is analyzed, organized, and augmented using data augmentation to complete the dataset construction. Then, a base network can be selected and improved to address the specific problem, resulting in an improved YOLOv5 network. Next, Anaconda (an open-source Python distribution) is used to build a virtual environment for network training on a graphics processing unit (GPU) server. Then, a dataset is provided for training the improved YOLOv5 network. Finally, the trained flame and smoke detection network is used to detect flames and smoke in images or videos.

[0045] In the process of establishing the aforementioned dataset, the required categories of flame and smoke can be determined. Web scraping techniques are used to collect publicly available datasets and videos of the corresponding categories from the internet. The Labelme annotation tool is then used to label the collected data with target categories and locations. The annotation information can be stored in XML format, and may include the category information of flame and smoke within the image (e.g., flame label is 0, smoke label is 1), center point coordinates (x, y), and dimensions (w, h), resulting in the initial image data (datasets_original) after annotation.

[0046] In constructing the aforementioned dataset, a statistical analysis script can be written to count the number of flames and smoke, and to statistically analyze the category (label) information of datasets_original. For categories with fewer entries, images can be manually collected, labeled, and added to the dataset under different environments, distances, and shooting angles to obtain category-balanced image data (datasets_balance). Common object detection data augmentation operations such as color transformation, scale transformation, random cropping, and random scaling are then used to further expand the datasets_balance, resulting in the final training data (datasets_train).

[0047] In the construction of the improved YOLOv5 network described above, the basic YOLOv5 network can be selected as the detection model. This basic network mainly consists of a backbone network, a neck network, and a head network. Improvements to this basic network can include: replacing the standard convolutional structures and CSP modules in the network with depthwise separable convolutional structures and Ghost modules to reduce the number of network parameters while maintaining network accuracy and performance; improving the SPP modules in the backbone network to MSA modules to enhance the network's ability to extract key features; replacing convolutional modules in the backbone network with more than 256 channels to 256 channels; and expanding the head part of the network to four output branches to improve the network's ability to fit multi-scale targets and accelerate network convergence. Specifically, for example, a 64x downsampling module can be connected to the 32x downsampling module in the neck part, and then combined with the Conv 1x1 convolutional module in the head part to obtain a fourth output branch. In general, by stacking the modified structures and modules in the same order as the original YOLOv5 network, the improved YOLOv5 network can be obtained, such as... Figure 4 As shown (here) Figure 4 The functions concat, upsample, and dw in this context are all present in the original YOLOv5 and will not be described further.

[0048] For multi-scale attention modules (MSA) to enhance the extraction of key semantic information, their structure and computational process are as follows: Figure 5As shown. The input feature map is broadcast to three branches: the rightmost branch is the attention weight calculation branch (i.e., the first branch mentioned above). First, the feature map undergoes a non-linear transformation (Conv->BN->ReLU) to obtain stronger representation capabilities. Second, it is upsampled to the resolution of the original feature map (C / 3, H, W) through multiple max pooling operations of different scales (kernel sizes such as 3x3, 5x5, 7x7). Third, the three are fused to obtain a feature map with multi-scale feature representation (C, H, W). Fourth, the feature representation is further enriched by using a depthwise separable convolution (DSC) with a kernel size of 3x3 and batch normalization (BN). Fifth, global average pooling (GAP) is used to compress the effectiveness of global information in each channel into a single point (C, 1, 1), and then the Sigmoid activation function is used to scale the feature point to [0, 1], completing the attention weight calculation of the feature map in the channel dimension. The second branch from the right is the attention-weighted branch (i.e., the second branch mentioned earlier). After the feature map undergoes a nonlinear transformation (Conv->BN->ReLU), it is multiplied by the attention weights of the first branch from the right. The attention weights are used to represent the effectiveness of the current channel feature (the closer the weight is to 1, the more important the channel feature is), thus completing the selection of key features. The third branch from the right is the identity transformation branch (i.e., the third branch mentioned earlier). It performs a matrix addition operation with the attention-weighted features from the second branch from the right to avoid network degradation caused by gradient vanishing or explosion. Finally, an output feature map (C, H, W) with the same scale as the input feature map is obtained.

[0049] Based on the improved network described above, during the process of providing a dataset for training the improved network, the dataset `datasets_train` can be preprocessed. The standard XML file undergoes coordinate normalization, resulting in an XML file containing the target category, center point coordinates, target bounding box width, and target bounding box height. Furthermore, the image data is uniformly scaled to 416x416 pixels, with 80% of the data used as the training set and the remaining 20% ​​as the validation set. This completes the image data preprocessing. Next, a virtual environment for training the model is set up on a GPU server. After this is complete, the training set is input into the improved network to train the flame and smoke detection model. Once training is complete, an inference model for detecting flames and smoke is obtained.

[0050] Finally, by inputting the image or video stream to be detected into the inference model obtained after the above training, the flame and smoke detection results in image or video file format can be obtained, realizing the classification (label) and localization (x,y,w,h) of flames and smoke.

[0051] The above exemplarily illustrates a flame and smoke detection method according to an embodiment of this application. Based on the above description, the flame and smoke detection method according to an embodiment of this application employs an improved YOLOv5 network including a multi-scale attention module to detect flames and smoke in images. It has a high capture capability for flames and / or smoke generated when objects are not completely burned, which have varying shapes and scales, thereby enabling rapid and accurate identification and location of flames and / or smoke in images.

[0052] The following is combined with Figure 6 This application describes a flame and smoke detection device provided in another aspect. Figure 6 A schematic block diagram of a flame and smoke detection device 600 according to an embodiment of this application is shown. Figure 6 As shown, the flame and smoke detection device 600 according to an embodiment of this application may include a memory 610 and a processor 620. The memory 610 stores a computer program executed by the processor 620. When the computer program is executed by the processor 620, it causes the processor 620 to perform the flame and smoke detection method 200 described above according to an embodiment of this application. Those skilled in the art can understand the specific operation of the flame and smoke detection device 600 according to the embodiments of this application in conjunction with the foregoing description. For the sake of brevity, specific details will not be repeated here, only some main operations of the processor 620 will be described.

[0053] In one embodiment of this application, when the computer program is run by the processor 620, the processor 620 performs the following steps: acquiring an image to be detected; using a trained detection model to perform flame detection and / or smoke detection on the image, and outputting the classification and localization results of the flame and / or smoke in the image; wherein, the detection model includes an improved network obtained by improving the YOLOv5 network, wherein the improvement of the YOLOv5 network includes: replacing the spatial pyramid pooling module in the backbone network of the YOLOv5 network with a multi-scale attention module.

[0054] In one embodiment of this application, the multi-scale attention module includes a first branch, a second branch, and a third branch, wherein: the first branch is used to calculate attention weights on the input feature map to obtain a first calculation result, and the input feature map is a feature map extracted from the image by the backbone network; the second branch is used to perform a nonlinear transformation on the input feature map and multiply the nonlinear transformation result and the first calculation result to obtain a second calculation result; the third branch is used to perform a matrix addition operation on the input feature map and the second calculation result to obtain an output feature map.

[0055] In one embodiment of this application, the improvement to the YOLOv5 network further includes: replacing the standard convolutional modules in the backbone network of the YOLOv5 network with depthwise separable convolutional modules; the first branch calculates attention weights on the input feature map to obtain a first calculation result, including: performing a nonlinear transformation on the input feature map to obtain a nonlinear transformation result; performing max pooling on the nonlinear transformation result at multiple different scales and upsampling it to the same resolution as the input feature map to obtain multiple upsampling results; fusing the multiple upsampling results to obtain a fused feature map; and sequentially performing depthwise separable convolution, batch normalization, global average pooling, and scale scaling on the fused feature map to obtain the first calculation result.

[0056] In one embodiment of this application, the kernel sizes used for max pooling at multiple different scales are 3x3, 5x5, and 7x7, respectively; the kernel size used for depthwise separable convolution is 3x3.

[0057] In one embodiment of this application, the improvement to the YOLOv5 network further includes replacing the center point and scale prediction modules in the backbone network of the YOLOv5 network with a ghosting module; wherein the input feature map is obtained by the ghosting module processing the image.

[0058] In one embodiment of this application, the improvement to the YOLOv5 network further includes at least one of the following: replacing convolutional modules with more than 256 channels in the backbone network of the YOLOv5 network with convolutional modules with 256 channels; replacing the center point and scale prediction modules in the neck network of the YOLOv5 network with ghosting modules, and replacing the standard convolutional modules in the neck network with depthwise separable convolutional modules; and expanding the three output branches in the head network of the YOLOv5 network to four output branches.

[0059] In one embodiment of this application, training the detection model includes: collecting sample images containing flames and / or smoke; labeling and data augmenting the sample images to obtain a training dataset; preprocessing the data in the training dataset to obtain a preprocessed dataset; building a virtual environment required for training the model on a server; and training the improved network in the virtual environment based on the preprocessed dataset to obtain a trained detection model.

[0060] In one embodiment of this application, the application scenarios of device 600 may include at least one of the following: forest fire prevention, factory fire prevention, warehouse fire prevention, residential fire prevention, and shopping mall fire prevention. For relatively open spaces such as living rooms, yards, and warehouses, traditional smoke detection systems have limited effectiveness. The flame and smoke detection device of this application can be used in a deep learning-based video analysis-based automatic smoke detection and early warning system, overcoming the shortcomings of traditional fire alarm equipment. It is fully adaptable to indoor and outdoor scenarios, and can provide rapid alarms, thus possessing a very broad market potential.

[0061] Furthermore, according to embodiments of this application, a storage medium is also provided, on which program instructions are stored. When executed by a computer or processor, these program instructions are used to perform corresponding steps of the flame and smoke detection method of this application. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0062] Based on the above description, the flame and smoke detection method and apparatus according to the embodiments of this application use an improved YOLOv5 network including a multi-scale attention module to detect flames and smoke in images. It has a high capture capability for flames and / or smoke generated when objects are not completely burned due to varying shapes and scales, thereby enabling rapid and accurate identification and location of flames and / or smoke in images.

[0063] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0064] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0065] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0066] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0067] Similarly, it should be understood that, in order to streamline this application and aid in understanding one or more of the various inventive aspects, features of this application may sometimes be grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with features fewer than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0068] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0069] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0070] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0071] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0072] The above description is merely a specific embodiment or illustration of the embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. A method of flame and smoke detection, characterized in that, The method includes: Acquire the image to be detected; The trained detection model is used to perform flame detection and / or smoke detection on the image, and the classification and localization results of the flames and / or smoke in the image are output. The detection model includes an improved network obtained by improving the YOLOv5 network. The improvement of the YOLOv5 network includes replacing the spatial pyramid pooling module in the backbone network of the YOLOv5 network with a multi-scale attention module. The multi-scale attention module includes a first branch, a second branch, and a third branch. The first branch calculates the attention weights of the input feature map, the second branch uses the attention weights obtained from the first branch to filter key features of the input feature map, and the third branch obtains an output feature map with the same scale as the input feature map based on the input feature map and the filtered feature map obtained from the second branch.

2. The method according to claim 1, characterized in that, The first branch performs attention weight calculation on the input feature map to obtain a first calculation result, wherein the input feature map is the feature map extracted by the backbone network from the image; The second branch performs a nonlinear transformation on the input feature map and multiplies the nonlinear transformation result with the first calculation result to obtain the second calculation result; The third branch performs a matrix addition operation on the input feature map and the second calculation result to obtain the output feature map.

3. The method of claim 2, wherein, The improvement to the YOLOv5 network also includes replacing the standard convolutional modules in the backbone network of the YOLOv5 network with depthwise separable convolutional modules; The first branch calculates attention weights on the input feature map to obtain a first calculation result, including: Perform a nonlinear transformation on the input feature map to obtain the nonlinear transformation result; The nonlinear transformation result is subjected to max pooling at multiple different scales and then upsampled to the same resolution as the input feature map to obtain multiple upsampling results; The multiple upsampling results are fused to obtain a fused feature map; The fused feature map is then subjected to depthwise separable convolution, batch normalization, global average pooling, and scale-up sequentially to obtain the first calculation result.

4. The method of claim 3, wherein, The kernel sizes used for the multiple max pooling operations of different scales are 3x3, 5x5, and 7x7, respectively. The depthwise separable convolution uses a 3x3 kernel.

5. The method of claim 2, wherein, The improvements to the YOLOv5 network also include: Replace the center point and scale prediction modules in the backbone network of the YOLOv5 network with ghosting modules; The input feature map is obtained by the ghosting module processing the image.

6. The method of claim 1, wherein, The improvements to the YOLOv5 network also include at least one of the following: Replace the convolutional modules with more than 256 channels in the backbone network of the YOLOv5 network with convolutional modules with 256 channels. Replace the center point and scale prediction modules in the neck network of the YOLOv5 network with ghosting modules, and replace the standard convolution modules in the neck network with depthwise separable convolution modules. The three output branches in the head network of the YOLOv5 network are expanded to four output branches.

7. The method according to any one of claims 1-6, characterized by, Training the detection model includes: Collect sample images containing flames and / or smoke, and annotate and augment the sample images to obtain a training dataset; The data in the training dataset is preprocessed to obtain the preprocessed dataset; Set up the virtual environment required for training the model on the server; The improved network is trained in the virtual environment based on the preprocessed dataset to obtain the trained detection model.

8. A flame and smoke detection apparatus, characterized in that, The device includes a memory and a processor, the memory storing a computer program executed by the processor, the computer program, when executed by the processor, causing the processor to perform the flame and smoke detection method as described in any one of claims 1-7.

9. The apparatus of claim 8, wherein, The application scenarios of the device include at least one of the following: forest fire prevention, factory fire prevention, warehouse fire prevention, residential fire prevention, and shopping mall fire prevention.

10. A storage medium, characterized by The storage medium stores a computer program that is executed by a processor, which, when run by the processor, causes the processor to perform the flame and smoke detection method as described in any one of claims 1-7.

Images