A multi-scale interactive fusion smoke image segmentation method

Abstract

The present application relates to the technical field of image segmentation, and discloses a multi-scale interactive fusion smoke image segmentation method, comprising the following steps: constructing a smoke segmentation dataset; training a smoke segmentation model using the smoke segmentation dataset; the smoke segmentation model comprises a Transformer encoder, a CNN encoder, a multi-level feature complementary module and a multi-scale fusion decoding module; an original image is input into the Transformer encoder and the CNN encoder respectively; F transformeri output by the Transformer encoder and F CNNi output by the CNN encoder are jointly input into the multi-level feature complementary module to obtain a feature map F i ; n feature maps F i are input into the multi-scale fusion decoding module for fusion to obtain a smoke segmentation result. The present application is used for extracting a smoke area and can provide more accurate smoke area information, which is of great significance for early warning of a fire.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image segmentation, in particular to a multi-scale interactive fusion smoke image segmentation method. BACKGROUND

[0002] Visual fire detection methods need to identify and locate the specific position of smoke and flame, because smoke usually appears first in most fires, detecting smoke can provide earlier fire early warning than flame. Smoke image segmentation is a full-image pixel-by-pixel identification task, which can provide more abundant information by separating smoke and its fine boundaries from the image, but the semi-transparent, non-rigid, fuzzy boundary and other characteristics of smoke bring great challenges to identification. Compared with other objects, smoke has many specialities and uncertainties with time and environment. In addition, the diffusivity of smoke and the angle and distance of video monitoring are not fixed. Therefore, the size of the smoke target in the video image varies greatly.

[0003] However, there are relatively few studies and applications of smoke segmentation facing real scenes at present, and there is a lack of smoke segmentation datasets in real scenes and effective methods to solve the dependence of global information for smoke segmentation. The existing smoke segmentation methods are difficult to capture enough context information and spatial details at the same time, because of the contradiction between local spatial details and global semantic information. Therefore, the model for smoke segmentation still needs to be improved. SUMMARY

[0004] The purpose of the present application is to provide a multi-scale interactive fusion smoke image segmentation method for extracting smoke regions, which can provide more accurate smoke region information and is of great significance for early warning of fire.

[0005] In order to achieve the above-mentioned application purpose, the embodiment of the present application provides the following technical scheme:

[0006] A multi-scale interactive fusion smoke image segmentation method, comprising the following steps:

[0007] Step 1, constructing a smoke segmentation dataset;

[0008] Step 2, training a smoke segmentation model using the smoke segmentation dataset;

[0009] The smoke segmentation model comprises a Transformer encoder, a CNN encoder, a multi-level feature complementary module and a multi-scale fusion decoding module; the original image is input into the Transformer encoder and the CNN encoder respectively; the feature map F transformeri output by the Transformer encoder and the feature map F CNNi output by the CNN encoder are jointly input into the multi-level feature complementary module to obtain a feature map Fi , i = 1, 2,..., n, n is the number of levels of the multi-level feature complementary module;n feature maps F i The multi-scale fusion decoding module is inputted for fusion, and a smoke segmentation result is obtained.

[0010] Compared with the prior art, the present application has the following beneficial effects:

[0011] The application discloses a smoke image segmentation method based on a multi-scale interaction fusion of a Transformer-CNN, and realizes smoke segmentation in an early stage of a fire. First, two encoders of a Transformer and a CNN are designed, the ability of the CNN encoder to capture spatial context information and the ability of the Transformer encoder to extract long-range dependencies are fused, the global modeling capability of the model is improved, and richer local detail features are reserved. In addition, high-level semantic information of an image is helpful to determine the category to which a pixel belongs, and low-level image information is conducive to obtaining a fine segmentation result. For complementary features generated by a double channel, a multi-level feature complementary module (MFCM) is designed, information interaction between different levels of two encoding paths can be flexibly focused, and the expression capability of features is improved. Finally, a multi-scale fusion decoding module (MFD) is designed, image information of different scales is fused, information interaction is performed in feature layers of different scales, and balance between global and local information is realized.

[0012] The method designed in the application can extract a smoke area, can provide more accurate smoke area information, and has important significance for early warning of a fire. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the application, the drawings needed to be used in the embodiments will be briefly introduced below, and it should be understood that the following drawings only show some of the embodiments of the application, and therefore should not be regarded as a limitation on the scope, and other related drawings can also be obtained by those skilled in the art without creative labor on the premise of not paying.

[0014] Figure 1 Fig. 1 is a network structure schematic diagram of a Transformer encoder, a CNN encoder and a multi-level feature complementary module of the application;

[0015] Figure 2 Fig. 2 is a network structure schematic diagram of a Trans-Block layer in the Transformer encoder of the application;

[0016] Figure 3A network structure schematic diagram of the MFCM layer in the multi-level feature complementary module of the present application;

[0017] Figure 4 A network structure schematic diagram of the multi-scale fusion decoding module of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0019] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance, or implying any such actual relationship or order between these entities or operations. In addition, the terms "connected", "connected", etc. can be direct connection between elements, or indirect connection via other elements.

[0020] The present application is realized by the following technical solutions: a multi-scale interactive fusion smoke image segmentation method, comprising the following steps:

[0021] Step 1, constructing a smoke segmentation data set.

[0022] The step 1 specifically comprises the following steps:

[0023] Step 1-1, the ordinary optical camera is directed at the flame smoke at multiple angles outdoors;

[0024] Step 1-2, continuously collecting optical images in a fixed time period;

[0025] Step 1-3, changing the length of the time period (such as 0.1s, 0.2s,..., 1s), and returning to step 1-2 until repeating step 1-3 for 10 times;

[0026] Step 1-4, changing the relative position of the ordinary optical camera by random stretching, and returning to step 1-1 until repeating step 1-4 for 100 times;

[0027] Step 1-5, replace the outdoor flame smoke scene, and return to step 1-1 until step 1-5 is repeated several times;

[0028] Step 1-6, in order to ensure the generalization of the model, non-smoke videos (such as mountain roads, heavy fog, buildings, clouds, and swaying leaves) are added to the data set, and each video is about 10 minutes long;

[0029] Step 1-7, smoke data in the collected optical image is segmented and labeled, and segmentation and labeling need to ensure that each frame of smoke area is accurately labeled as foreground (smoke label), and the rest is labeled as background (background label); for non-smoke data, label according to the corresponding category (such as buildings, leaves, etc.);

[0030] Step 1-8, the smoke image data and non-smoke image data are combined to form a smoke segmentation data set.

[0031] Step 2, use the smoke segmentation data set to train the smoke segmentation model.

[0032] The training set and the test set are divided from the smoke segmentation data set according to the proportion of 9:1, and carry the corresponding label data (i.e. mask). Data augmentation is adopted in the form of scaling, cropping, mirroring and noise to expand the scale of the training set and the test set. Specifically, the smoke image data and non-smoke image data in the training set and the test set are scaled outward by 15%, randomly cropped to 224x224, mirrored, and superimposed with Gaussian noise to expand the training set and the test set to 3 times the original size.

[0033] After preprocessing the image data, the training set is used as the input of the smoke segmentation model, and the corresponding label data is used as the output of the smoke segmentation model. The model is trained so that the model can output smoke features after feature extraction and inference.

[0034] The smoke segmentation model is deployed based on the Transformer-CNN multi-scale interactive fusion, please refer to Figure 1The smoke segmentation model includes a Transformer encoder, a CNN encoder, a multi-level feature complementary module, and a multi-scale fusion decoding module. The Transformer encoder is used to extract global long-range dependency information. The CNN encoder is used to extract local context information, which helps to refine the local details of the segmentation mask. The multi-level feature complementary module is used to fuse the global long-range dependency information and the local context information, integrate multi-dimensional data from the two encoders, and output features with rich levels and scales. This combination aims to provide accurate smoke segmentation and capture a wider range of backgrounds and minor details. During the diffusion process, smoke mixes with air to form a gradual concentration field, making the target boundary exhibit a fuzzy transition area. Therefore, an edge-sensitive loss function is designed for the smoke segmentation model, which can accurately locate the edge area of the smoke and pay more attention to the edge features of the smoke, effectively addressing the problems of edge blur, texture loss, and dynamic changes in smoke segmentation.

[0035] In detail, please refer to Figure 1 The Transformer encoder includes an Embedding layer, a first semantic segmentation unit, a second semantic segmentation unit, a third semantic segmentation unit, and a fourth semantic segmentation unit connected in sequence. The first semantic segmentation unit includes a Trans-Block1 layer and a Trans-Block2 layer connected in sequence, wherein the input end of the Trans-Block1 layer is connected with the output end of the Embedding layer. The second semantic segmentation unit includes a PatchMerging1 layer, a Trans-Block3 layer, and a Trans-Block4 layer connected in sequence, wherein the input end of the PatchMerging1 layer is connected with the output end of the Trans-Block2 layer. The third semantic segmentation unit includes a PatchMerging2 layer, a Trans-Block5 layer, and a Trans-Block6 layer connected in sequence, wherein the input end of the PatchMerging2 layer is connected with the output end of the Trans-Block4 layer. The fourth semantic segmentation unit includes a PatchMerging3 layer, a Trans-Block7 layer, and a Trans-Block8 layer connected in sequence, wherein the input end of the PatchMerging3 layer is connected with the output end of the Trans-Block6 layer.

[0036] The four groups of stacked semantic segmentation units in the Transformer encoder can generate feature maps of different scales, and perform global interaction at fine resolution and coarse resolution.

[0037] The structure of each Trans-Block layer in the Transformer encoder is the same, please refer to Figure 2Any one Trans-Block layer includes an Efficient Self-Attention layer, a Mix-FFN layer and a LayerNorm layer connected in sequence.

[0038] The Efficient Self-Attention layer performs feature pooling on the feature map by using a self-attention matrix, and reduces the dimension of the output feature to obtain a multi-scale feature output. For the Efficient Self-Attention layer, the input information is weighted by attention by calculating the correlation between different parts of the input information, which enables the model to obtain a global view of the input information at a shallow layer. Specifically, the input information X is mapped to a query matrix (Q), a key matrix (K) and a value matrix (V), the correlation between the query matrix and different key matrices is calculated, that is, the weight coefficients of different values are calculated; and the weighted average result of the value matrix is taken as the attention value. The formula is as follows:

[0039]

[0040] wherein, softmax is an activation function, d k is a scaling coefficient.

[0041] For the Mix-FFN layer, a convolution kernel with a size of 3x3 Conv and a multi-layer perceptron (MLP) are mixed into each feedforward network (FFN) for feature mapping to obtain the feature X out output by the Trans-Block layer. The formula is as follows:

[0042] X out = LayerNorm(GELU(Conv 3×3 (MLP(X in ))) + X in ;

[0043] wherein, X in represents the output feature of the Efficient Self-Attention layer, GELU is a Gaussian error-based activation function, MLP is a multi-layer perceptron, and LayerNorm is a normalization layer.

[0044] Please continue to refer to Figure 1 , the CNN encoder includes a convolution block, a CNN Layer1 unit, a CNN Layer2 unit, a CNN Layer3 unit and a CNN Layer4 unit connected in sequence. The convolution block includes a Conv layer, a BN layer and a ReLU activation function. The multi-level feature complementary module includes an MFCM1 layer, an MFCM2 layer, an MFCM3 layer and an MFCM4 layer.

[0045] The original image is input into the Embedding layer of the Transformer encoder and the convolution block of the CNN encoder respectively, and image processing is performed in the channels of the respective encoders. The feature map F Transformer1 output by the Trans-Block2 layer and the feature map F CNN1 output by the CNNLayer1 unit are input together into the MFCM1 layer for interaction, and after integration, a feature map F1 with rich levels and scales is output; the feature map F Transformer2 output by the Trans-Block4 layer and the feature map F CNN2 output by the CNN Layer2 unit are input together into the MFCM2 layer to obtain a feature map F2; the feature map F Transformer3 output by the Trans-Block6 layer and the feature map F CNN3 output by the CNN Layer3 unit are input together into the MFCM3 layer to obtain a feature map F3; the feature map F Transformer4 output by the Trans-Block8 layer and the feature map F CNN4 output by the CNN Layer4 unit are input together into the MFCM4 layer to obtain a feature map F4.

[0046] Each MFCM layer in the multi-level feature complementary module has the same structure, please refer to Figure 3 , any MFCM layer includes a Transformer integration channel, a CNN integration channel, and a feature complementary channel. In the Transformer integration channel, the input feature map F Transformer is processed by 1x1Conv to generate weight coefficients W v1 , W k1 , and W q1 , which are mapped into a value matrix (V1), a key matrix (K1), and a query matrix (Q1) based on the weight coefficients; in the CNN integration channel, the input feature map F CNN is processed by 1x1Conv to generate weight coefficients W v2 , W k2 , and W q2 , which are mapped into a value matrix (V2), a key matrix (K2), and a query matrix (Q2) based on the weight coefficients; then, in the Transformer integration channel, the key matrix (K1) and the query matrix (Q2) are fused by a Softmax activation function to generate an Attention Map1, and then the value matrix (V1) and the Attention Map1 are fused and processed by 1x1Conv to generate a feature map F Transformer`; the query matrix (Q1) and the key matrix (K2) are fused after the Softmax activation function in the CNN integration channel to generate Attention Map2, and then the value matrix (V2) and Attention Map2 are fused and subjected to 1x1Conv, thereby generating the feature map F CNN` ; finally, the feature map F Transformer` and the feature map F CNN` are spliced and subjected to 3x3Conv, thereby generating the feature map F i (i = 1, 2, 3, 4).

[0047] As shown in Figure 4 , the multi-scale fusion decoding module includes an up-sampling layer, a splicing layer, a 3x3Conv, and a 1x1Conv. The feature maps F1, F2, F3, and F4 are input into the up-sampling layer to generate feature maps F 1` , F 2` , F 3` , and F 4` of the same scale through different scales of up-sampling; the feature maps F 1` , F 2` , F 3` , and F 4` are input into the splicing layer for splicing, and then are sequentially input into the 3x3Conv and the 1x1Conv, and finally the smoke segmentation result output by the smoke segmentation model is obtained.

[0048] The edge-sensitive loss function designed in the scheme is extracted by an adaptive threshold Canny operator, which can accurately locate the edge region of the smoke, and the edge-sensitive loss function is:

[0049]

[0050] wherein, E T is the aversion edge region extracted by the adaptive threshold Canny operator, j represents the index of the jth pixel point in E T ; η represents a balance coefficient; y j represents a real pixel value, p j represents a predicted pixel value; γ is a scale parameter between the weighted cross-entropy loss and the gradient consistency loss of the edge pixel;

[0051] is the weighted cross-entropy loss of the edge pixel; L Gradient is the gradient consistency loss, and has:

[0052]

[0053] wherein, N represents the number of edge pixel points, and i represents the index of the ith pixel point; p idenotes a predicted pixel value, y i denotes a real pixel value.

[0054] The weighted cross-entropy loss of the edge pixel can strengthen the accuracy of the classification of the edge pixel, and the gradient consistency loss can ensure that the gradient information at the edge is more stable. The balance coefficient is used to flexibly adjust the weights of the two characteristics of the smoke, i.e., the smoke semi-transparent characteristic and the dynamic change characteristic, so as to adapt to different smoke scenes.

[0055] The above merely describes a specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for multi-scale interactive fusion smoke image segmentation, characterized in that: The method comprises the following steps: Step 1, constructing a smoke segmentation dataset; Step 2, training a smoke segmentation model using the smoke segmentation dataset; The smoke segmentation model comprises a Transformer encoder, a CNN encoder, a multi-level feature complementary module and a multi-scale fusion decoding module; an original image is input into the Transformer encoder and the CNN encoder respectively; a feature map F transformeri output by the Transformer encoder and a feature map F CNNi output by the CNN encoder are jointly input into the multi-level feature complementary module to obtain a feature map F i , i=1, 2,..., n, n is the number of levels of the multi-level feature complementary module; n feature maps F i are input into the multi-scale fusion decoding module for fusion to obtain a smoke segmentation result; The Transformer encoder comprises an Embedding layer, a first semantic segmentation unit, a second semantic segmentation unit, a third semantic segmentation unit and a fourth semantic segmentation unit connected in sequence; The first semantic segmentation unit comprises a Trans-Block1 layer and a Trans-Block2 layer connected in sequence, wherein an input end of the Trans-Block1 layer is connected with an output end of the Embedding layer; the Trans-Block2 layer outputs a feature map F Transformer1 ​ The second semantic segmentation unit comprises a Patch Merging1 layer, a Trans-Block3 layer and a Trans-Block4 layer connected in sequence, wherein an input end of the Patch Merging1 layer is connected with an output end of the Trans-Block2 layer; the Trans-Block4 layer outputs a feature map F Transformer2 ; The third semantic segmentation unit comprises a Patch Merging2 layer, a Trans-Block5 layer and a Trans-Block6 layer connected in sequence, wherein the input end of the Patch Merging2 layer is connected with the output end of the Trans-Block4 layer; the Trans-Block6 layer outputs a feature map F Transformer3 ; The fourth semantic segmentation unit comprises a Patch Merging3 layer, a Trans-Block7 layer and a Trans-Block8 layer connected in sequence, wherein the input end of the Patch Merging3 layer is connected with the output end of the Trans-Block6 layer; the Trans-Block8 layer outputs a feature map F Transformer4 ; The structure of each MFCM layer in the multi-level feature complementary module is the same, and any MFCM layer includes a Transformer integration channel, a CNN integration channel and a feature complementary channel; in the Transformer integration channel, the input feature map F Transformer After 1×1Conv, the weight coefficient W v1 , W k1 , W q1 is mapped into a value matrix V1, a key matrix K1 and a query matrix Q1 based on the weight coefficient; in the CNN integration channel, the input feature map F CNN After 1×1Conv, the weight coefficient W v2 , W k2 , W q2 is mapped into a value matrix V2, a key matrix K2 and a query matrix Q2 based on the weight coefficient; then, in the Transformer integration channel, the key matrix K1 and the query matrix Q2 are fused after Softmax activation function to generate an Attention Map1, and then the value matrix V1 and the Attention Map1 are fused after 1×1Conv, thereby generating a feature map F Transformer` ; in the CNN integration channel, the query matrix Q1 and the key matrix K2 are fused after Softmax activation function to generate an Attention Map2, and then the value matrix V2 and the Attention Map2 are fused after 1×1Conv, thereby generating a feature map F CNN` ; finally, the feature map F Transformer` and the feature map F CNN` are spliced and then subjected to 3×3Conv, thereby generating a feature map F i , i=1,2,3,4.

2. The multi-scale interactive fusion smog image segmentation method according to claim 1, characterized in that: The CNN encoder comprises a convolutional block, a CNN Layer1 unit, a CNN Layer2 unit, a CNN Layer3 unit and a CNN Layer4 unit connected in sequence; CNN Layer1 unit output feature map F CNN1 , CNN Layer2 unit output feature map F CNN2 , CNN Layer3 unit output feature map F CNN3 , CNN Layer4 unit output feature map F CNN4 .

3. The multi-scale interactive fusion smoke image segmentation method according to claim 2, characterized in that: The multi-level feature complementary module comprises an MFCM1 layer, an MFCM2 layer, an MFCM3 layer and an MFCM4 layer; The feature map F Transformer1 and the feature map F CNN1 are input into the MFCM1 layer together to obtain a feature map F1; The feature map F Transformer2 and the feature map F CNN2 are input into the MFCM2 layer together to obtain a feature map F2; The feature map F Transformer3 and the feature map F CNN3 are input into the MFCM 3 layer to obtain a feature map F3; The feature map F Transformer4 and the feature map F CNN4 are input into the MFCM 4 layer to obtain a feature map F4.

4. The multi-scale interactive fusion smoke image segmentation method according to claim 3, characterized in that: The multi-scale fusion decoding module comprises an up-sampling layer, a splicing layer, a 3x3 Conv and a 1x1 Conv. The feature map F1, the feature map F2, the feature map F3, and the feature map F4 are input into an upsampling layer to be upsampled in different scales to generate feature maps F of the same scale 1` , feature map F 2` , feature map F 3` , feature map F 4` ; the feature map F 1` , feature map F 2` , feature map F 3` , feature map F 4` are input into a splicing layer to be spliced, and then are sequentially input into a 3*3Conv and a 1*1Conv to finally obtain a smoke segmentation result output by the smoke segmentation model.

Images