A flame smoke recognition method and apparatus

Abstract

The application discloses a flame smoke recognition method and device, and belongs to the technical field of fire detection. The method comprises the following steps: extracting the flame smoke features of a received target image based on a preliminary feature extraction module to obtain initial image features; obtaining local image features by performing multiple local feature extraction on the initial image features in multiple dimensions and multiple scales based on a local attention module; obtaining global image features by aggregating the global feature information of the initial image features and the local image features based on a self-attention aggregation module; performing feature fusion on the initial image features, the local image features and the global image features based on a neck module to obtain fused image features; and detecting the flame smoke recognition results of different scales in the target image according to the fused image features based on multiple lightweight detection heads. The application realizes accurate recognition of small target flames and thin smoke by optimizing and strengthening the local features and the global features and using the lightweight detection heads.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fire detection, and in particular to a flame smoke recognition method and device. BACKGROUND

[0002] Fire is a kind of disaster event with great destructive power, which not only poses a serious threat to people's life and property safety, but also causes great damage to the ecological environment. Although people's awareness of fire prevention is constantly improving, the number of fires in large space scenes such as forests, grasslands, and large public places is still high. Therefore, in order to cope with the sudden disaster of fire in large space scenes, reduce the loss caused by fire to the environment and reduce the casualties caused by fire, it is necessary to study the early detection method of fire in large space scenes.

[0003] In the prior art, image-based fire detectors based on deep learning technology are widely used in fire monitoring in large space scenes, such as forest and grassland monitoring. However, in the early stage of fire, the size of the flame and smoke is small, the shape is irregular, and the transparency is high. Current algorithms are difficult to accurately locate, and still face the problems of missed detection and false detection. In the prior art, although YOLOv9 performs best in fire detection tasks compared to other advanced target detection algorithms, there are still problems of missed detection of small target flames and thin smoke and background false detection in terms of detection accuracy. That is, the existing technology YOLOv9 still has the technical problem of being unable to accurately identify local detail features and global features.

[0004] Therefore, there is an urgent need for a flame smoke recognition method to solve the technical problems of missed detection of small target flames and thin smoke and background false detection in the prior art. SUMMARY

[0005] Therefore, there is an urgent need for a flame smoke recognition method to solve the technical problems of missed detection of small target flames and thin smoke and background false detection in the prior art.

[0006] In a first aspect, the present application provides a flame smoke recognition method, comprising:

[0007] extracting the flame smoke features of the received target image based on the preliminary feature extraction module to obtain initial image features;

[0008] obtaining local image features by performing multiple local feature extractions of the initial image features in multiple dimensions and multiple scales based on the local attention module;

[0009] obtaining global image features by aggregating the global feature information of the initial image features and the local image features based on the self-attention aggregation module;

[0010] The initial image feature, the local image feature and the global image feature are fused based on a neck module to obtain a fused image feature;

[0011] The flame smoke recognition result of different scales in a target image is detected based on a plurality of lightweight detection heads according to the fused image feature.

[0012] In a possible implementation manner, the local attention module includes two first local attention sub-modules;

[0013] The first local attention sub-module includes an adaptive convolution layer, a local generalized efficient layer aggregation network layer and a cross-space local attention layer;

[0014] The adaptive convolution layer is used for adaptive scale feature extraction on the initial image feature to obtain a first multi-scale local image feature;

[0015] The local generalized efficient layer aggregation network layer is used for further local feature extraction on the first multi-scale local image feature to obtain a second multi-scale local image feature, and the first multi-scale local image feature and the second multi-scale local image feature are fused to obtain a multi-scale local image feature;

[0016] The cross-space local attention layer is used for multi-dimensional position information enhancement and multi-scale feature splicing on the multi-scale local image feature to obtain the local image feature.

[0017] In a possible implementation manner, the cross-space local attention layer includes a first branch, a second branch and a first fusion branch;

[0018] The first branch is used for different-dimensional convolution operations on the multi-scale local image feature to obtain a position attention representation;

[0019] The second branch is used for different-scale convolution operations on the multi-scale local image feature, and the convolution results are merged to obtain a multi-scale attention representation;

[0020] The first fusion branch fuses the position attention representation and the multi-scale attention representation to obtain the local image feature.

[0021] In a possible implementation manner, the first branch includes a horizontal branch, a vertical branch and a second fusion branch;

[0022] The horizontal branch is used for extracting a horizontal direction feature of the multi-scale local image feature, and performing convolution operation on the horizontal direction feature to obtain a horizontal position attention representation;

[0023] The vertical branch is used for extracting a vertical direction feature of the multi-scale local image feature, and performing a convolution operation on the vertical direction feature to obtain a vertical position attention representation;

[0024] The second fusion branch is used for fusing the horizontal position attention representation and the vertical position attention representation to obtain a position attention representation of the multi-scale local image feature.

[0025] In a possible implementation manner, the self-attention aggregation module includes a spatial pyramid pooling enhanced local attention network sub-module and a global efficient aggregation network sub-module;

[0026] The spatial pyramid pooling enhanced local attention network sub-module extracts upper and lower correlation information of the initial image feature and the local image feature through different scale pooling kernels to obtain global semantics, and fuses the global semantics with the local image feature to obtain a preliminary global image feature;

[0027] The global efficient aggregation network sub-module is used for extracting global information of different levels from the preliminary global image feature to obtain a global image feature.

[0028] In a possible implementation manner, the global efficient aggregation network sub-module includes a one-dimensional convolution layer, a feature segmentation layer, a first PSABlock processing layer, a second PSABlock processing layer, and a global fusion layer;

[0029] The one-dimensional convolution layer is used for performing one-dimensional convolution processing on the preliminary global image feature to obtain a one-dimensional feature;

[0030] The feature segmentation layer is used for segmenting the one-dimensional feature to obtain a first feature and a second feature;

[0031] The first PSABlock processing layer is used for processing the first feature to obtain a first processed feature;

[0032] The second PSABlock processing layer is used for processing the first processed feature to obtain a second processed feature;

[0033] The global fusion layer is used for globally fusing the first feature, the second feature, the first processed feature, and the second processed feature, and then performing convolution reduction to obtain a global image feature.

[0034] In a possible implementation manner, the lightweight detection head includes a regression head and a classification head;

[0035] The regression head is used for performing bounding box prediction on the flame smoke in the image according to the fused image feature;

[0036] The classification head is used for classifying the flame smoke in the image according to the fused image features.

[0037] In a possible implementation manner, the classification head comprises a depth separable convolution layer, and the depth separable convolution layer comprises a depth convolution layer, a detection head cross-space local attention layer and a point convolution layer.

[0038] The depth convolution layer is used for extracting local features and semantic correlations of the fused image features.

[0039] The detection head cross-space local attention layer is used for dynamically adjusting a weight distribution of features according to the local features and the semantic correlations.

[0040] The point convolution layer is used for fusing and reducing dimensions of the fused image features according to the weight distribution.

[0041] In a possible implementation manner, the preliminary feature extraction module comprises an input layer, a multi-dimensional convolution layer, a preliminary generalized efficient aggregation network layer, a preliminary adaptive convolution layer and a preliminary generalized efficient layer aggregation network layer.

[0042] The input layer is used for receiving a target image.

[0043] The multi-dimensional convolution layer is used for extracting features of the target image by using convolution kernels of different scales to obtain first initial features.

[0044] The preliminary generalized efficient aggregation network layer is used for stacking and aggregating the first initial features output by the multi-dimensional convolution layer to extract initial semantic features.

[0045] The preliminary adaptive convolution layer is used for performing multi-scale convolution operations on the first initial features to obtain first initial multi-scale features.

[0046] The preliminary generalized efficient layer aggregation network layer is used for performing feature enhancement and feature fusion on the first initial multi-scale features to obtain initial image features.

[0047] In a second aspect, the present application further provides a flame smoke identification device, a memory and a processor, wherein,

[0048] The memory is used for storing programs.

[0049] The processor is coupled with the memory and is used for executing the programs stored in the memory to implement the steps in the flame smoke identification method.

[0050] The beneficial effects of the present application are: first, based on the preliminary feature extraction module, the flame smoke features of the received target image are extracted to obtain initial image features; then, based on the local attention module, the initial image features are extracted multiple times in multiple dimensions and multiple scales to obtain local image features, the cross-space local attention layer can enable the model to focus on the region of interest in the image, enhance the accuracy and robustness of local feature extraction, and more accurately locate and extract small target flame smoke features; then, based on the self-attention aggregation module, the global feature information of the initial image features and the local image features is aggregated to obtain global image features, the global feature information of each feature is aggregated to capture the interaction between different features, the semantics between upper and lower features, enhance the learning ability of the complex relationship of flame smoke, and in the case of thin smoke or similar background and smoke, the global feature information and the local feature information are combined to identify the flame smoke; then, based on the neck module, the initial image features, the local image features and the global image features are fused to obtain fused image features; finally, based on multiple detection heads, different scale flame smoke recognition results in the target image are detected according to the fused image features, and a lightweight detection head is used to effectively reduce the parameter quantity and computing power of the network while maintaining the flame smoke recognition accuracy. The present application extracts and fuses the initial image features, the local features and the global features, uses a lightweight detection head to detect small target flames, thin smoke and smoke features similar to background and smoke features. The accuracy and robustness of flame smoke recognition are improved. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0052] Figure 1 An embodiment flowchart of the flame smoke recognition method provided by the present application is shown in the figure.

[0053] Figure 2 A schematic diagram of the Fire-YOLO model provided by the present application is shown in the figure.

[0054] Figure 3 A structure diagram of the cross-space local attention layer provided by the present application is shown in the figure.

[0055] Figure 4 A structure diagram of the global efficient aggregation network sub-module provided by the present application is shown in the figure.

[0056] Figure 5A structural schematic diagram of a light weight detection head LDDetect provided by the present application is shown in the figure;

[0057] Figure 6 A structural schematic diagram of one embodiment of a flame smoke identification device provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present application will be clearly and completely described in the description of the embodiments of the present application in combination with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0059] In the description of the embodiments of the present application, the meaning of “a plurality of” is two or more, unless otherwise specified.

[0060] The “first”, “second” and the like descriptions involved in the embodiments of the present application are only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the technical features limited by “first” and “second” can explicitly or implicitly include at least one of the features.

[0061] In this document, the reference to “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily mutually exclusive of other embodiments. It is explicitly and implicitly understood that the embodiments described herein can be combined with other embodiments.

[0062] The present application provides a flame smoke identification method and device, which are described below respectively.

[0063] Figure 1 An embodiment flowchart of a flame smoke identification method provided by the present application is shown in the figure, and the flame smoke identification method comprises: Figure 1

[0064] S101, extracting the flame smoke features of the received target image based on the preliminary feature extraction module 100 to obtain initial image features;

[0065] S102, obtaining local image features by performing multiple local feature extractions in multiple dimensions and multiple scales on the initial image features based on the local attention module 200;

[0066] ​S103, obtaining a global image feature based on the self-attention aggregation module 300 by aggregating the global feature information of the initial image feature and the local image feature;

[0067] S104, performing feature fusion on the initial image feature, the local image feature and the global image feature based on the neck module 400 to obtain a fused image feature;

[0068] S105, detecting flame smoke recognition results of different scales in a target image based on the plurality of lightweight detection heads 500 according to the fused image feature.

[0069] It should be noted that the present embodiment is based on the GELAN-S network structure in the target detection network YOLOv9 for optimization and improvement. In the present embodiment, the preliminary feature extraction module 100, the local attention module 200, the self-attention aggregation module 300, the neck module 400 and the plurality of lightweight detection heads 500 constitute the optimized Fire-YOLO model, and the above method steps are realized through the Fire-YOLO model. The result graph of the Fire-YOLO model is shown in Figure 2 Figure 2 The schematic diagram of the Fire-YOLO model provided by the present application is shown.

[0070] It should be further noted that the present embodiment compares and experiments six target detectors, i.e., YOLOv5, YOLOv7, YOLOv8, YOLOv9, YOLOv10 and RTDETR. In the model experiment, the software operating system is Windows 11. The Python version used in the experiment is 3.10.12. The Pytorch deep learning framework is selected as the framework for network training and inference. The cudatoolkit version used in the experiment is 11.8.0. In terms of hardware environment, the GPU model used is RTX4090, the video memory size is 24G, the memory is 128G, and the CPU model is Intel 13900K. The average precision mean, parameter quantity and computing power of each model under different IoU threshold conditions are calculated to evaluate the complexity and computing cost and performance of each model. Among them, GELAN-S in YOLOv9 performs best. When mAP50, the average precision mean reaches 91.33, and under the more stringent mAP50:95 evaluation standard, the average precision mean reaches 76.55, which is significantly better than other target detection models. Therefore, the GELAN-S network not only can provide excellent detection results under the condition of loose IoU threshold, but also can maintain high performance under more stringent positioning requirements, showing accuracy and robustness in complex fire scenes. At the same time, the GELAN-S network has a low parameter quantity and computing power, and performs well in model complexity and computing cost. ​

[0071] Further need to be explained, GELAN-S network although in the fire detection task compared with other advanced target detection algorithm performance is the most excellent, but in the detection precision aspect still exists small target flame, thin smoke and background false detection problem, therefore this embodiment aims at these problems to GELAN-S network structure makes optimization adjustment.

[0072] Specifically, the target image is preliminarily feature extracted by the preliminary feature extraction module 100; in the local attention module 200, a cross-space local attention layer is added in the process of local feature extraction on the initial image features, the local features are strengthened, and the regions of interest in the image are more focused, and more fine pixel-level features are extracted. In this embodiment, by setting two layers of local feature extraction process, the accuracy of local feature extraction is further improved, which can accurately identify the local detail features of small target flame smoke, thin smoke and similar background and flame smoke; in the backbone of the GELAN-S network, the global features output by the backbone are optimized by the self-attention aggregation module 300, further capturing the complex patterns and internal relations in the feature map, enhancing the model's ability to summarize the features, improving the accuracy of global feature extraction, and more helping to optimize the gradient propagation path, keeping the gradient stable, and thus improving the accuracy of model recognition; in the neck module, the global image features are up-sampled and fused with the local image features extracted by the first layer of local feature extraction, and the fused features are further extracted and then up-sampled. The up-sampled result is fused with the initial image features, and the global image features are obtained at this time. In this way, different image features are up-sampled and fused, which can ensure the integrity of the image feature information, and also can further strengthen the accurate identification of the features; the original DDetect detection head in the GELAN-S network structure is replaced by a lightweight detection head 500, which reduces the parameter quantity and computing power of the model, while effectively ensuring the accuracy of flame smoke identification. In this embodiment, three lightweight detection heads 500 are used for target detection of different scales. First, the input end of the first lightweight detection head is the global image features, which are used for small target detection. Then, the global image features are convolved to extract local image features and fused with the local image features output by the backbone, to obtain local fusion features. The second lightweight detection head is used for medium target detection. Finally, the local fusion features are convolved to extract large target image features, which are fused with the global image features, and the third lightweight detection head is used for large target detection. Through the recognition of different scale feature maps, small target flame or smoke, thin smoke or smoke features similar to background can be accurately distinguished.

[0073] In the embodiments of the present application, the features refer to feature maps.

[0074] In addition, the present embodiment also constructs a high-quality fire detection dataset, which is based on DFS software and D-Fire and other image datasets containing fire smoke. At the same time, fire images under different lighting conditions, climates and scenes are collected from the Internet, the collected fire images are preprocessed to remove duplicate fire images, and the remaining 7250 images are re-labeled using Labeling software. The labeled images are randomly divided into training set, validation set and test set according to the ratio of 8:1:1 to construct a complete high-quality fire monitoring dataset. The fire detection dataset is divided into training dataset, validation dataset and test dataset. The training dataset is used to train the flame smoke recognition model, the validation dataset is used to verify the flame smoke recognition model, and the test dataset is used to evaluate the performance of the model.

[0075] Compared with the prior art, the present embodiment extracts the flame smoke features of the received target image based on the preliminary feature extraction module 100 to obtain initial image features; based on the local attention module 200, the initial image features are extracted multiple times in multiple dimensions and multiple scales to obtain local image features. Based on the cross-space local attention layer, the model can focus on the region of interest in the image, enhance the accuracy and robustness of local feature extraction, and more accurately locate and extract small target flame smoke features; based on the self-attention aggregation module 300, the global feature information of the initial image features and the local image features is aggregated to obtain global image features. By aggregating the global feature information of each feature, the interaction between different features, the semantics between upper and lower features, and the learning ability of the complex relationship of flame smoke are captured. In the case of thin smoke or similar background and smoke, the flame smoke can be recognized by combining global feature information and local feature information; based on the neck module 400, the initial image features, the local image features and the global image features are fused to obtain fused image features; based on multiple detection heads, the flame smoke recognition results of different scales in the target image are detected according to the fused image features. The lightweight detection head 500 is adopted to effectively reduce the parameter quantity and computing power of the network while maintaining the flame smoke recognition accuracy. The present embodiment strengthens the extraction and fusion of local features and global features by preliminary image features, and uses the lightweight detection head 500 to detect small target flames, thin smoke and smoke features similar to background smoke features. The accuracy and robustness of flame smoke recognition are improved.

[0076] In some embodiments of the present application, the local attention module 200 includes two first local attention sub-modules 210;

[0077] The first local attention submodule 210 includes a local adaptive convolutional layer AConv, a local generalized efficient layer aggregation network layer RepNCSPELAN, and a cross-space local attention layer CLA.

[0078] The local adaptive convolutional layer AConv is used to perform adaptive scale feature extraction on the initial image features to obtain the first multi-scale local image features;

[0079] The RepNCSPELAN local generalized high efficiency aggregation network layer is used to further extract local features from the first multi-scale local image features to obtain the second multi-scale local image features. The first multi-scale local image features and the second multi-scale local image features are then fused to obtain the multi-scale local image features.

[0080] The Cross-Spatial Local Attention Layer (CLA) is used to enhance the location information of multi-scale local image features in multiple dimensions and to stitch together the features at multiple scales to obtain local image features.

[0081] Specifically, the cross-spatial local attention layer added during the local feature extraction process of the original backbone network can make it more focused on the region of interest in the image, resulting in more refined pixel-level local features.

[0082] In some embodiments of the present invention, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the cross-space local attention layer (CLA) provided by the present invention. The cross-space local attention layer (CLA) includes a first branch 10, a second branch 20, and a first fusion branch 30.

[0083] The first branch 10 is used to perform convolution operations of different dimensions on multi-scale local image features to obtain positional attention representations;

[0084] Specifically, the cross-spatial local attention layer (CLA) first processes the multi-scale local image features by grouping them into spatial and channel dimensions through the Groups layer to achieve cross-dimensional interaction. The feature maps processed by the Groups layer are then input into the first branch 10 and the second branch 20, respectively. In the first branch 10, the horizontal and vertical features of the multi-scale local image features are processed to obtain the horizontal positional attention representation and the vertical positional attention representation obtained from the vertical branch. These are then weighted and fused to obtain the positional attention representation of the features.

[0085] The second branch 20 is used to perform convolution operations on multi-scale local image features at different scales and merge the convolution results to obtain a multi-scale attention representation;

[0086] Further, the embodiment performs three-dimensional convolution operation on the feature map through the three-dimensional convolution layer in the second branch 20 to obtain a three-dimensional attention representation, performs one-dimensional convolution operation on the feature map through the one-dimensional convolution layer to obtain a one-dimensional attention representation, and then splices the three-dimensional attention representation and the one-dimensional attention representation to obtain a multi-scale attention representation. The branch adopts the reparameterization technology, which can enhance the ability of extracting multi-scale features, obtain more rich receptive field information of small targets, and further improve the reasoning speed of the model. The calculation formula of the multi-scale attention representation is:

[0087] ,

[0088] wherein, is the multi-scale attention representation, is the three-dimensional attention representation output by the three-dimensional convolution layer, is the one-dimensional attention representation output by the one-dimensional convolution layer.

[0089] The first fusion branch 30 fuses the position attention representation and the multi-scale attention representation to obtain a local image feature.

[0090] It should be noted that, based on the cross-space learning method in the EMA, the position attention representation obtained by the first branch 10 and the multi-scale attention representation obtained by the second branch 20 are fused to obtain a pixel-level feature map.

[0091] In order to further illustrate the performance of the cross-space local attention mechanism in the embodiment, the cross-space local attention mechanism in the embodiment is compared and analyzed with other advanced attention mechanisms through experiments. Among them, CBAM, ECA, CA, EMA and ELA are selected as various advanced attention mechanisms, the adding positions of these attention mechanisms on the GELAN-S network are ensured to be consistent, the average precision mean, parameter quantity and computing power of each model under different IoU threshold conditions are calculated to evaluate the complexity and calculation cost and performance of each model, as shown in Table 1, Table 1 is the experimental results of different attention mechanisms provided by the present application.

[0092] Table 1 Experimental results of different attention mechanisms

[0093]

[0094] As can be seen from Table 1, the GELAN-S network added with the cross-space local attention mechanism CLA has obviously better performance in the flame smoke identification process than the GELAN-S network added with other attention mechanisms, that is, the cross-space local attention mechanism CLA, the mAP50 and mAP50:95 on the test set are increased by 1.55% and 1.14% respectively, and the parameter quantity and FLOPs are almost not increased, which shows that the cross-space local attention mechanism CLA can not only significantly improve the detection accuracy of the GELAN-S network, but also does not increase the calculation burden.

[0095] The embodiment introduces the cross-space local attention mechanism into the deep feature extraction layer, generates more fine pixel-level attention for the feature image, and can more accurately obtain the small target, thin smoke and receptive field information similar to the smoke feature.

[0096] In some embodiments of the application, the first branch 10 includes a horizontal branch 11, a vertical branch 12 and a second fusion branch 13.

[0097] The horizontal branch 11 is used for extracting horizontal direction features of multi-scale local image features, performing convolution operation on the horizontal direction features, and obtaining horizontal position attention representation;

[0098] The vertical branch 12 is used for extracting vertical direction features of multi-scale local image features, performing convolution operation on the vertical direction features, and obtaining vertical position attention representation;

[0099] The second fusion branch 13 is used for fusing the horizontal position attention representation and the vertical position attention representation to obtain the position attention representation of the multi-scale local image features.

[0100] Specifically, the horizontal branch 11 and the vertical branch 12 have the same processing mode for the features. Here, taking the horizontal direction as an example, first, the feature vector of the feature map in the horizontal direction is extracted through the horizontal pooling layer X-AVG Pool; one-dimensional convolution operation is performed on the feature vector through the one-dimensional convolution layer Conv1d to generate a spatial weight map, and the convolution kernel of the convolution layer is set to 7 here to enhance the interaction ability of local features; the horizontal feature vector after one-dimensional convolution operation is normalized through the group normalization layer to alleviate the feature distribution difference problem; the attention weight map is mapped to [0, 1] through the Sigmoid layer, which represents the importance degree of different positions or channels, enhances the response of important regions, and finally the horizontal position attention representation obtained by the horizontal branch, wherein the calculation formula of the horizontal position attention representation is:

[0101] ,

[0102] wherein, is the horizontal attention representation, , is a horizontal direction of the initial feature, h is a horizontal feature of the initial feature, H is a horizontal height of the initial feature, i , and the value range of is [0, 0，H ], h is a horizontal direction height, is a one-dimensional convolution function in the horizontal direction, is a normalization function, is a Sigmoid nonlinear activation function.

[0103] Similarly, the calculation formula of the vertical position attention representation is:

[0104] ,

[0105] wherein, is a vertical attention representation, , is a vertical feature of the initial feature, w is a vertical height of the feature map, W , and the value range of is [0, j ], W is a vertical direction height, w is a one-dimensional convolution function in the vertical direction, is a normalization function, is a Sigmoid nonlinear activation function. After obtaining the horizontal position attention representation and the vertical position attention representation, based on the cross-space learning method in EMA, the horizontal position attention representation and the vertical position attention representation are fused, and the fusion formula is:

[0106]

[0107] , wherein,

[0108] is a position attention representation, is a normalization function, is a feature map after pooling, X ， is a Sigmoid nonlinear activation function, is a horizontal attention representation, is a vertical attention representation. In some embodiments of the present application, the self-attention aggregation module 300 includes a spatial pyramid pooling enhanced local attention network sub-module SPPELAN and a global efficient aggregation network sub-module EPSAAN;

[0109]

[0110] ​The spatial pyramid pooling enhanced local attention network submodule SPPELAN extracts the upper and lower correlation information of the initial image features and the local image features through different scale pooling kernels, obtains global semantics, and fuses the global semantics with the local image features to obtain preliminary global image features.

[0111] The global efficient aggregation network submodule EPSAAN is used for extracting global information of different levels from the preliminary global image features to obtain global image features.

[0112] In some embodiments of the present application, as shown in Figure 4 Figure 4 The global efficient aggregation network submodule provided by the present application is shown in the structural diagram, and the global efficient aggregation network submodule EPSAAN includes a one-dimensional convolution layer 21, a feature segmentation layer 22, a first PSABlock processing layer 23, a second PSABlock processing layer 24, and a global fusion layer 25.

[0113] The one-dimensional convolution layer 21 is used for one-dimensional convolution processing of the preliminary global image features to obtain one-dimensional features.

[0114] The feature segmentation layer 22 is used for segmenting the one-dimensional features to obtain first features and second features.

[0115] The first PSABlock processing layer 23 is used for processing the first features to obtain first processed features.

[0116] The second PSABlock processing layer 24 is used for processing the first processed features to obtain second processed features.

[0117] The global fusion layer 25 is used for global fusion of the first features, the second features, the first processed features, and the second processed features, and then convolution restoration to obtain global image features.

[0118] Specifically, the preliminary global image features are subjected to one-dimensional convolution through the one-dimensional convolution layer 21, and after one-dimensional convolution, they are evenly divided into two parts through the feature segmentation layer 22, one part remains unchanged, and the other part will undergo two PSABlock processes. After two PSABlock processes, the calculation complexity can be reduced, and the global information of the input features can be extracted more effectively, the complex patterns and internal relations in the fire smoke features can be fully captured, and finally, the features after PSABlock processing and the features before processing are spliced together, and one-dimensional convolution processing is performed on the spliced features to restore them to the original feature form. This process can obtain multi-level global feature information while ensuring the integrity of the features, and can enhance the model feature summarization ability.

[0119] ​Further in order to illustrate the optimization effect of the efficient partial self-attention aggregation module 300 on the GELAN-S network, the embodiment studies the performance of the PSA network and the efficient partial self-attention aggregation module 30 EPSAAN in the flame smoke identification through experiments, as shown in Table 2, Table 2 is the experimental comparison results of the PSA network and the EPSAAN network provided by the application.

[0120] Table 2 Experimental comparison results of the PSA network and the EPSAAN network

[0121]

[0122] Among them, the performance evaluation results of the GELAN-S-CLA-LDDetect network without adding the EPSAAN network or the PSA network, the performance evaluation results after adding the PSA network and the performance evaluation results after adding the EPSAAN network are listed respectively, and the performance evaluation results are still realized by calculating the average precision mean, the parameter quantity and the computing power of each model under different IoU threshold conditions. As can be seen from Table 2, after using the EPSAAN network, the mAP50:95 of the GELAN-S-CLA-LDDetect on the test set is improved from 77.25 to 77.72. This shows that the EPSAAN module can effectively enhance the learning ability of the network to complex fire features, and the mAP of the EPSAAN network on the test set is better than that of the PSA module, which shows that the generalization performance of the EPSAAN network on unknown data is also better than that of the PSA network.

[0123] In some embodiments of the application, the lightweight detection head 500 includes a regression head and a classification head;

[0124] The regression head is used for bounding box prediction of flame smoke in the image according to the fused image features;

[0125] The classification head is used for classifying the flame smoke in the image according to the fused image features.

[0126] It should be noted that in the GELAN-S network structure, the parameter quantity of the original DDetect detection head accounts for 75.3% of the parameter quantity of the detection head and 15.4% of the parameter quantity of the entire model. Since the classification head of the fire smoke detection task only needs to output the classification score of flame or smoke, the lightweight detection head LDDetect detection head is used to replace the original DDetect detection head, and the optimized GELAN-S network, i.e., GELAN-S-CLA-EPS-LDDetect network, is obtained. It is verified through experiments that the parameter quantity and computing power thereof are reduced by 13.73% and 16.73% respectively. Compared with the original DDetect detection head, the parameter quantity of LDDetect is significantly reduced by 65.97%. At the same time, the GELAN-S-CLA-EPS-LDDetect network is similar to the GELAN-S-CLA-EPS in the performance evaluation result, which indicates that the LDDetect detection head can significantly reduce the parameter quantity while keeping the detection accuracy unchanged.

[0127] In some embodiments of the present application, the classification head includes two deep separable convolution DCConv layers, and the deep separable convolution DCConv layer includes a deep convolution sublayer 31, a detection head cross-space local attention sublayer 32 and a point convolution sublayer 33.

[0128] The deep convolution sublayer 31 is used to extract local features and semantic associations of the fused image features;

[0129] The detection head cross-space local attention sublayer 32 is used to dynamically adjust the weight distribution of the features according to the local features and semantic associations;

[0130] The point convolution sublayer 33 is used to fuse and reduce the dimension of the fused image features according to the weight distribution.

[0131] It should be noted that the network structure diagram of the lightweight detection head LDDetect is as shown in Figure 5 , Figure 5 The structure diagram of the lightweight detection head LDDetect provided by the present application is shown in the figure, wherein the lightweight detection head LDDetect includes a regression head and a classification head. In the classification head, two deep separable convolution DCConv and a standard convolution are included. A deep separable convolution DCConv is constructed by combining a deep convolution and a point convolution with a cross-space local attention mechanism to replace the ordinary convolution Conv in the DDetect detection head structure, which can reduce the parameter quantity and computing power of the classification head.

[0132] Specifically, a convolution operation is independently applied to each input feature channel by a depth convolution sublayer 31, and in this embodiment, the convolution kernel is set to 3; a point convolution sublayer 33 is used to fuse different channels through 1*1 convolution, which significantly reduces the parameter quantity and computing power of the detection head; by adding a detection head cross-space local attention sublayer 32 between the depth convolution sublayer 31 and the point convolution sublayer 33, the attention of the classification head to the region of interest is enhanced, some unimportant information is ignored, and the accuracy of the classification head in classifying the flame smoke is enhanced.

[0133] The embodiment replaces the original DDetect detection head with a lightweight detection head, thereby reducing the parameter quantity and computing power of the model.

[0134] In some embodiments of the present application, the preliminary feature extraction module 100 includes an input layer input, a multi-dimensional convolution layer Conv, a preliminary generalized efficient aggregation network layer ELAN1, a preliminary adaptive convolution layer AConv, and a preliminary generalized efficient layer aggregation network layer RepNCSPELAN.

[0135] The input layer input is configured to receive a target image.

[0136] The multi-dimensional convolution layer Conv is configured to extract features of the target image through different scale convolution kernels to obtain first initial features.

[0137] The preliminary generalized efficient aggregation network layer ELAN1 is configured to stack and aggregate the first initial features output by the multi-dimensional convolution layer to extract initial semantic features.

[0138] The preliminary adaptive convolution layer AConv is configured to perform multi-scale convolution operations on the first initial features to obtain first initial multi-scale features.

[0139] The preliminary generalized efficient layer aggregation network layer RepNCSPELAN is configured to perform feature enhancement and feature fusion on the first initial multi-scale features to obtain initial image features.

[0140] As shown in Figure 6 The present application also provides a flame smoke recognition device 600, which includes a processor 601, a memory 602, and a display 603. Figure 6 Only some components of the flame smoke recognition device 600 are shown, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented.

[0141] The flame smoke recognition device 600 includes a processor 601, a memory 602, and a display 603. Figure 6Only part of the components of the flame smoke identification device 600 are shown, but it should be understood that all the shown components are not required to be implemented, and more or less components can be alternatively implemented.

[0142] The memory 602 can be an internal storage unit of the flame smoke identification device 600 in some embodiments, for example, a hard disk or a memory of the flame smoke identification device 600. The memory 602 can also be an external storage device of the flame smoke identification device 600 in other embodiments, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the flame smoke identification device 600.

[0143] The processor 601 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, for running program codes or processing data stored in the memory 602, for example, the flame smoke identification method in the present application, and / or the unmanned ship route optimization method.

[0144] The display 603 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 603 is used to display information of the flame smoke identification device 600 and to display a visualized user interface. The components 801-803 of the flame smoke identification device 600 communicate with each other through a system bus.

[0145] In some embodiments of the present application, when the processor 601 executes the flame smoke identification program in the memory 602, the following steps can be implemented:

[0146] Based on the preliminary feature extraction module, the flame smoke features of the received target image are extracted to obtain initial image features;

[0147] Based on the local attention module, the initial image features are extracted multiple times in multiple dimensions and multiple scales to obtain local image features;

[0148] Based on the self-attention aggregation module, the global feature information of the initial image features and the local image features is aggregated to obtain global image features;

[0149] Based on the neck module, the initial image features, the local image features and the global image features are fused to obtain fused image features;

[0150] The flame smoke recognition result of different scales in the target image is detected based on multiple lightweight detection heads according to fused image features.

[0151] It should be understood that, in addition to the above functions, the processor 601 can also implement other functions when executing the flame smoke recognition program and / or the unmanned ship route optimization program in the memory 602. For details, refer to the descriptions of the related method embodiments.

[0152] The flame smoke recognition method and device provided by the present application are described in detail above, and the principles and implementation manners of the present application are described by applying specific examples. The above example is only used to help understand the method and core idea of the present application. Meanwhile, for those skilled in the art, the specific implementation manner and application range can be changed according to the idea of the present application. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A flame smoke recognition method, characterized by, The method comprises: extracting flame smoke features of a received target image based on a preliminary feature extraction module to obtain initial image features; obtaining local image features by performing multiple local feature extraction on the initial image features in multiple dimensions and multiple scales based on a local attention module; the local attention module comprises two first local attention sub-modules; the first local attention sub-module comprises a local adaptive convolution layer, a local generalized efficient layer aggregation network layer and a cross-space local attention layer; the local adaptive convolution layer is used for adaptive scale feature extraction on the initial image features to obtain first multi-scale local image features; the local generalized efficient aggregation network layer is used for further local feature extraction on the first multi-scale local image features to obtain second multi-scale local image features; the first multi-scale local image features and the second multi-scale local image features are fused to obtain multi-scale local image features; and the cross-space local attention layer is used for multi-dimensional position information strengthening and multi-scale feature splicing on the multi-scale local image features to obtain local image features; the cross-space local attention layer comprises a first branch, a second branch and a first fusion branch; the first branch is used for convolution operation on the multi-scale local image features in different dimensions to obtain position attention representation; the second branch is used for convolution operation on the multi-scale local image features in different scales, and the convolution results are combined to obtain multi-scale attention representation; and the first fusion branch fuses the position attention representation and the multi-scale attention representation to obtain local image features; the first branch comprises a horizontal branch, a vertical branch and a second fusion branch; the horizontal branch is used for extracting horizontal direction features of the multi-scale local image features, and convolution operation is performed on the horizontal direction features to obtain horizontal position attention representation; the vertical branch is used for extracting vertical direction features of the multi-scale local image features, and convolution operation is performed on the vertical direction features to obtain vertical position attention representation; and the second fusion branch is used for fusing the horizontal position attention representation and the vertical position attention representation to obtain position attention representation of the multi-scale local image features; obtaining global image features by aggregating global feature information of the initial image features and the local image features based on a self-attention aggregation module; performing feature fusion on the initial image features, the local image features and the global image features based on a neck module to obtain fused image features; detecting flame smoke recognition results of different scales in the target image based on a plurality of lightweight detection heads according to the fused image features.

2. The flame smoke detection method of claim 1, wherein, the self-attention aggregation module comprises a spatial pyramid pooling enhanced local attention network sub-module and a global efficient aggregation network sub-module; the spatial pyramid pooling enhanced local attention network sub-module extracts upper and lower correlation information of the initial image features and the local image features by using different scale pooling kernels to obtain global semantics, fuses the global semantics with the local image features to obtain preliminary global image features; and the global efficient aggregation network sub-module aggregates the preliminary global image features and the initial image features to obtain global image features. The global efficient aggregation network submodule is used for extracting global information of different levels from the preliminary global image feature to obtain a global image feature.

3. The flame smoke detection method of claim 2, wherein, The global efficient aggregation network submodule comprises a one-dimensional convolution layer, a feature segmentation layer, a first PSABlock processing layer, a second PSABlock processing layer, and a global fusion layer. The one-dimensional convolution layer is used for performing one-dimensional convolution processing on the preliminary global image feature to obtain a one-dimensional feature. The feature segmentation layer is used for segmenting the one-dimensional feature to obtain a first feature and a second feature. The first PSABlock processing layer is used for processing the first feature to obtain a first processed feature. The second PSABlock processing layer is used for processing the first processed feature to obtain a second processed feature. The global fusion layer is used for globally fusing the first feature, the second feature, the first processed feature, and the second processed feature and then performing convolution reduction to obtain a global image feature.

4. The flame smoke detection method of claim 1, wherein, The lightweight detection head comprises a regression head and a classification head. The regression head is used for performing boundary box prediction on the flame smoke in the image according to the fused image feature. The classification head is used for classifying the flame smoke in the image according to the fused image feature.

5. The flame smoke detection method of claim 4, wherein, The classification head comprises a depth separable convolution layer, which comprises a depth convolution layer, a detection head cross-space local attention layer, and a point convolution layer. The depth convolution layer is used for extracting local features and semantic correlations of the fused image feature. The detection head cross-space local attention layer is used for dynamically adjusting the weight distribution of the features according to the local features and the semantic correlations. The point convolution layer is used for fusing and reducing dimensions of the fused image feature according to the weight distribution.

6. The flame puff recognition method of claim 1, wherein, The preliminary feature extraction module comprises an input layer, a multi-dimensional convolution layer, a preliminary generalized efficient aggregation network layer, a preliminary adaptive convolution layer, and a preliminary generalized efficient layer aggregation network layer. The input layer is used for receiving a target image. The multi-dimensional convolution layer is used for extracting features of the target image through convolution kernels of different scales to obtain a first initial feature. The preliminary generalized efficient aggregation network layer is used for stacking and aggregating the first initial feature output by the multi-dimensional convolution layer to extract an initial semantic feature. The preliminary adaptive convolution layer is used for performing multi-scale convolution operation on the first initial feature to obtain a first initial multi-scale feature. The preliminary generalized efficient layer aggregation network layer is used for performing feature enhancement and feature fusion on the first initial multi-scale feature to obtain an initial image feature.

7. A flame smoke detection apparatus, characterized by comprise: a memory and a processor, wherein the memory is configured to store a program; the processor is coupled to the memory and is configured to execute the program stored in the memory to implement the steps in the flame smoke identification method of any one of claims 1 to 6.

Images