A lightweight fireball and smoke detection method based on token pyramid and transformer framework

Abstract

The application discloses a kind of lightweight fireball and smoke detection methods based on Token pyramid and Transformer framework, first construct fireball and smoke detection network based on Token pyramid and Transformer framework;Then the field test image is normalized and input into detection network;Again the input end of detection network is used to extract the features of different levels of image;Next, the features in different levels are fused using the feature extraction network to obtain discriminative features;Finally, using the classification and positioning network to determine whether there is fireball, smoke and calculate its bounding box according to the position and type of the above features.The explosion fireball detection method proposed by the application has a high fireball detection accuracy of more than 89%, has low requirements for hardware device performance, and is suitable for harsh field test environment.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of pattern recognition, and particularly relates to a lightweight fireball and smoke detection method. BACKGROUND

[0002] With the continuous innovation and development of weapon equipment, some powerful warheads will produce a large range of fireball and smoke when exploding, and the heat released by the high-temperature fireball in an instant will cause serious thermal damage to the enemy target. At the same time, the toxic smoke produced by the explosion will also cause certain damage to the combat personnel. Therefore, accurately identifying and detecting the explosion fireball and smoke is the basic work for understanding the performance of the warhead and evaluating its damage effect. In addition, in the field of fire prevention and disaster relief, timely detection of the flame and smoke produced by the fire is also of great significance to reduce social and ecological losses. At present, there are more public reports on flame detection, such as:

[0003] Lujing et al. proposed a flame detection algorithm based on salient object detection in the document "Lujing, Tan Jin-gang, Zhang Zhi-yi, Chen Ming, He Wei. Improved flame detection algorithm based on salient object detection [J]. Laser and Optoelectronics Progress, 2022, 59(04): 0410012." A double-branch ResNet in parallel intersection is used as a feature extraction module, and a bidirectional gold tower structure is introduced to fuse shallow texture information and deep semantic information, which better obtains the flame segmentation result. Zhao Yuanyuan et al. proposed a video image flame real-time detection algorithm based on improved Yolo-v3 in the document "Zhao Yuanyuan, Zhu Jun, Xie Yankun, Li Weilian, Guo Yukun. Improved Yolo-v3 video image flame real-time detection algorithm [J]. Journal of Wuhan University, 2021, 56(03): 326-334." The multi-scale detection network and prior frame size in the original Yolo algorithm are optimized, and the network's recognition ability for small flame areas is strengthened.

[0004] It can be found that the current mainstream fireball / flame detection is based on deep learning, although the detection accuracy is high, but the neural network structure is too complex, which leads to high requirements for hardware. At the same time, there are outstanding contradictions such as high precision requirement for experimental field detection of explosion fireball and smoke, poor protection conditions, and low detection hardware configuration. SUMMARY

[0005] In order to overcome the shortcomings of the prior art, the present application provides a lightweight fireball and smoke detection method based on Token pyramid and Transformer framework, first, a fireball and smoke detection network based on Token pyramid and Transformer framework is constructed; then the on-site test image is normalized and input into the detection network; then the input end of the detection network is used to extract the features of different levels of the image; next, the feature extraction network is used to fuse the features in different levels to obtain the discriminative features; finally, the classification and positioning network is used to judge whether there is fireball and smoke according to the position and type of the above-mentioned features and calculate the boundary box. The explosion fireball detection method provided by the present application has a fireball detection accuracy of more than 89%, has low requirements on the performance of hardware devices, and is suitable for harsh field test environment.

[0006] The technical solution adopted by the present application to solve its technical problems comprises the following steps:

[0007] Step 1: Interpolate the input image I to 640*640, and perform pixel normalization processing on it;

[0008] Step 2: Construct a token-based image pyramid from the input RGB image Obtain the local feature map {C3, C4, C5};

[0009] The Token image pyramid Token adopts the Resnet50 architecture, which contains 50 convolution modules, each module includes convolution-regularization-nonlinear function ReLu three parts, wherein {C3, C4, C5} represents the output of the 30th, 40th and 50th layers in the backbone network;

[0010] C3, C4, C5 = Token(I)

[0011] Wherein,

[0012] Step 3: Construct a visual transformer extractor T to output scale perception semantic information P5;

[0013] The visual transformer extractor T is stacked by 3 Transformer modules, as formula (2);

[0014] Each Transformer module is composed of a multi-head attention module attention, a previous item transmission module FFM and a residual connection, as formula (3);

[0015] The feature map x, i.e. C5 obtained in step 2, is input, first passes through attention, FFM and residual connection to obtain the output of the first layer of the Transformer module, i.e. Transformer(x); the output of the last layer is taken as the input of the next layer, and so on, to obtain the output of the third layer of the Transformer module, i.e. T(x); when the input x is equal to C5, T(x) = P5;

[0016] P5 = T(C5) (1)

[0017] T(x) = Transformer(Transformer(Transformer(x))) (2)

[0018] Transformer(x) = FFM(x + attention(x)) + x + attention(x) (3)

[0019] Step 4: The recognition semantics of small targets are injected into the corresponding scale feature maps to obtain a series of enhanced feature representations {P6, P7};

[0020] P4 and C3 are taken as inputs, and a interpolation convolution module is used to obtain a feature map P3 with a larger size; after obtaining the features at different levels {P5, P6, P7}, the features are taken as inputs of a detection head;

[0021] Step 5: The classification and positioning network determines whether there is an explosive fireball, smoke and calculates the bounding box according to the position and type of the above-mentioned features;

[0022] The classification and positioning network detection head is composed of an upper branch and a lower branch, wherein the upper branch is responsible for classification and center point calculation tasks, and the lower branch is responsible for bounding box variable regression tasks; the upper branch is composed of 4 convolution modules with a step of 1, which does not change the resolution of the image, and finally outputs a matrix and The lower branch is composed of 4 convolution modules with a step of 1, and finally outputs a matrix The matrix A represents the probability that a pixel belongs to a certain class, the matrix B represents the probability that a pixel belongs to a flame center point, and the matrix C represents the distance of a pixel to a flame bounding box.

[0023] The beneficial effects of the present application are as follows:

[0024] The fireball detection method provided by the present application has a high fireball detection accuracy of 89% or more, has low requirements on the performance of hardware devices, and is suitable for harsh field test environments. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 The present application is based on a Token pyramid and a lightweight fireball and smoke detection network structure schematic diagram of a Transformer.

[0026] Figure 2 The fireball and smoke detection result diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0027] The present application will be further described below in combination with the drawings and embodiments.

[0028] The present application proposes a lightweight fireball and smoke detection method based on Token pyramid and Transformer framework, which can efficiently detect the fireball and smoke generated by explosion, while reducing the demand for hardware environment, so as to be suitable for the detection of explosion fireball and smoke in field test.

[0029] A lightweight fireball and smoke detection method based on Token pyramid and Transformer framework, comprising the following steps:

[0030] Step 1: Interpolate the input image J to 640*640, and perform pixel normalization processing on it;

[0031] Step 2: Construct a token-based image pyramid from the input RGB image Obtain the local feature map {C3, C4, C5};

[0032] The Token image pyramid Token adopts the Resnet50 architecture, which contains 50 convolution modules, each module includes convolution-regularization-nonlinear function ReLu three parts, wherein {C3, C4, C5} represents the output of the 30th, 40th and 50th layers in the backbone network;

[0033] C3, C4, C5 = Token(I)

[0034] Wherein,

[0035] Step 3: Construct a visual transformer extractor T to output scale perception semantic information P5;

[0036] The visual transformer extractor T is stacked by 3 Transformer modules, as formula (2);

[0037] Each Transformer module is composed of a multi-head attention module attention, a previous item transmission module FFM and a residual connection, as formula (3);

[0038] The feature map x, that is, C5 obtained in step 2, is input, first passes through attention, FFM and residual connection to obtain the output of the first layer of the Transformer module, that is, Transformer(x); the output of the last layer is taken as the input of the next layer, and the above is repeated to obtain the output of the third layer of the Transformer module, that is, T(x); when the input x is equal to C5, T(x) = P5;

[0039] P5 = T(C5) (1)

[0040] T(x) = Transformer(Transformer(Transformer(x))) (2)

[0041] Transformer(x) = FFM(x + attention(x)) + x + attention(x) (3)

[0042] Step 4: The recognition semantics of small targets are injected into the corresponding scale feature map to obtain a series of enhanced feature representations {P6, P7};

[0043] P5 and C4 are taken as inputs, and an interpolation convolution module is used to obtain a feature map P4 with a larger size; P4 and C3 are taken as inputs, and an interpolation convolution module is used to obtain a feature map P3 with a larger size; after obtaining the features of different levels {P5, P6, P7}, the features are taken as the input of the detection head;

[0044] Step 5: The classification and positioning network determines whether there is an explosive fireball, smoke according to the position and type of the above-mentioned features, and calculates the bounding box;

[0045] The classification and positioning network detection head is composed of upper and lower branches, wherein the upper branch is responsible for classification and center point calculation tasks, and the lower branch is responsible for bounding box variable regression tasks; the upper branch is composed of 4 convolution modules with a step of 1, which does not change the resolution of the image, and finally outputs a matrix and The lower branch is composed of 4 convolution modules with a step of 1, and finally outputs a matrix The matrix A represents the probability that a certain pixel belongs to a certain class, the matrix B represents the probability that a certain pixel belongs to a flame center point, and the matrix C represents the distance of a certain pixel to a flame bounding box.

[0046] The present application proposes a lightweight fireball and smoke detection method based on Token pyramid and Transformer framework, and the technical scheme is as follows:

[0047] 1. Construct a fireball and smoke detection network based on Token pyramid and Transformer framework;

[0048] 2 The on-site test image is normalized and input into the detection network;

[0049] 3 The features of different levels of the image are extracted using the input end of the detection network;

[0050] 4 The features in different levels are fused using the feature extraction network to obtain discriminative features;

[0051] 5 The classification and positioning network is used to determine whether there is a fireball or smoke according to the position and type of the above-mentioned features and calculate the bounding box

[0052] Specifically:

[0053] 1: The input image is interpolated to 640*640 size, and after pixel normalization, it is input into the backbone network.

[0054] 2: Construct a token-based image pyramid from the input RGB image Local feature maps {C3, C4, C5} are obtained. The token image pyramid Token uses the Resnet50 [1] architecture, which contains 50 convolution modules, each module including convolution-regularization-nonlinear function ReLu three parts, as Figure 1 shown, the blue part represents the backbone network, where {C3, C4, C5} represents the output of the 30th, 40th and 50th layers in the backbone network.

[0055] C3, C4, C5 = Token(I),

[0056] where,

[0057] 3: Construct a visual transformer extractor T to output scale-aware semantic information P5, as formula (1). The visual transformer extractor T is stacked by 3 Transformer modules, as formula (2). Each Transformer module is composed of a multi-head attention module attention [2], a front transfer module FFM [2] and a residual connection, as formula (2). Specifically, we input a certain feature map x (here x is equivalent to C5) first through attention, FFM and residual connection to obtain the output of the first layer Transformer module Transformer(x). The output of the last layer is taken as the input of the next layer, and so on, that is, the output of the third layer Transformer module T(x) can be obtained. When the initial input x is equal to C5, T(x) = P5.

[0058] P5 = T(C5) (1)

[0059] T(x) = Transformer(Transformer(Transformer(x))) (2)

[0060] Transformer(x) = FFM(x + attention(x)) + x + attention(x) (3)

[0061] 4: The recognition semantics for small targets are injected into the corresponding scale feature maps to obtain a series of enhanced feature representations {P6, P7}. Specifically, P5 and C4 are taken as inputs to obtain a larger size feature map P4 through an interpolation convolution module; P4 and C3 are taken as inputs to obtain a larger size feature map P3 through an interpolation convolution module. After obtaining the features at different levels {P5, P6, P7}, the features are taken as inputs of the detection head.

[0062] 5: The classification and positioning network determines whether there is an explosive fireball, smoke according to the position and type of the above-mentioned features, and calculates the bounding box. The classification and positioning network detection head is composed of an upper branch and a lower branch, wherein the upper branch is responsible for classification and center point calculation, and the lower branch is responsible for bounding box variable regression. The upper branch is composed of 4 convolution modules with a step size of 1, which does not change the resolution of the image, and finally outputs a matrix and The lower branch is composed of 4 convolution modules with a step size of 1, and finally outputs a matrix The matrix A represents the probability that a certain pixel belongs to a certain class, the matrix means that a certain pixel belongs to the center point, and the matrix means that the model predicts four variables on each pixel. As shown in Figure 2 .

[0063] The test results of the present application and the existing detection model are shown in Table 1, and it can be seen that the present application is superior to the existing model in various aspects.

[0064] Table 1 Comparison of detection performance of the model proposed in the present application and the existing model on the data set

[0065]

Claims

1. A lightweight fireball and smoke detection method based on the Token Pyramid and Transformer framework, characterized in that, Includes the following steps: Step 1: Input image Interpolate to 640*640 and then perform pixel normalization. Step 2: Construct a token-based image pyramid from the input RGB image Obtain local feature maps ; Token Image Pyramid It adopts the ResNet50 architecture and contains 50 convolutional modules. Each module consists of three parts: convolution, regularization, and the ReLU nonlinear function. This represents the output of layers 30, 40, and 50 in the backbone network; in, , , ; Step 3: Construct the visual transformer extractor Output scale-aware semantic information ; Visual Transformer Extractor It consists of three stacked Transformer modules, as shown in equation (2); Each Transformer module consists of a multi-head attention module. Precedence Transfer Module It is composed of the residual and the equation, as shown in formula (3); feature map That is, the result obtained in step 2 Input, first through , And the residual connection obtains the output of the first-layer Transformer module. The output of the previous layer is used as the input of the next layer, and so on, to obtain the output of the third-layer Transformer module. When the input equal hour, ; Step 4: The semantic recognition of small targets is injected into a feature map of appropriate size to obtain a series of enhanced feature representations. ; by and The input is passed through an interpolation convolution module to obtain a larger feature map. ;by and The input is passed through an interpolation convolution module to obtain a larger feature map. After obtaining features at different levels These features are used as input to the detection head; Step 5: The classification and localization network determines whether there is an explosion fireball or smoke based on the location and type of the above features and calculates the bounding box; The classification and localization network detection head consists of two branches: the upper branch is responsible for classification and center point calculation, and the lower branch is responsible for bounding box variable regression. The upper branch consists of four convolutional modules with a stride of 1, which do not change the image resolution, and finally outputs a matrix. and The lower branch consists of four convolutional modules with a stride of 1, and the final output matrix is... ;matrix A matrix representing the probability that a pixel belongs to a certain category. The matrix represents the probability that a pixel belongs to the center of the flame. C This represents the distance from a given pixel to the bounding box of the flame.

Images