A fire image smoke removal method based on an improved Cycle-Dehaze neural network

Abstract

The application provides a fire image smoke removal method based on an improved Cycle-Dehaze neural network, which adopts two kinds of brightness conversion functions to construct a virtual exposure image for two sets of pixel gray value of input images with overexposure and underexposure, and then carries out multi-exposure image fusion through high and low frequency information, effectively solving the problems of uneven image illumination and ghost image in fire background imaging. In addition, on the basis of the traditional Cycle-Dehaze neural network, a dynamic dense residual block for smoke feature enhancement is designed, and a CBAM attention mechanism is introduced, so that the division accuracy of relevant and irrelevant domains of the image is improved. At the same time, in order to better solve the color distortion and texture blur problems existing in the current neural network smoke removal image reconstruction, a color loss function is introduced on the basis of the original loss function, the color recognition ability in the SF-Cycle-Dehaze generator and discriminator is enhanced, and the smoke removal image reconstruction ability of the current mainstream model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of digital image processing, and particularly relates to a forest fire single image smoke removal method based on an improved Cycle-Dehaze neural network. BACKGROUND

[0002] Forest fire prevention and control is an extremely important link in environmental governance and social emergency systems, which not only has a great impact on the forest environment, but also involves important interests related to people's lives and national property. Therefore, accurately grasping the development and evolution trend of the fire, accurately positioning the disaster range and the position of the personnel to be rescued can greatly improve the efficiency of forest fire warning and emergency, and can provide more effective auxiliary decision support for forest fire prevention and disaster reduction.

[0003] The early and middle stages of fire often accompany the generation of a large amount of smoke, and the accompanying heat wave, turbulence, smoke particles absorbing and reflecting light bring great imaging difficulties to the photographic equipment such as unmanned aerial vehicle on-board camera, which not only causes the brightness of the image to become dark, but also easily causes the phenomenon of incomplete detail texture and color distortion. In addition, the unstable flight process of the unmanned aerial vehicle, camera shaking and other factors can easily cause ghosting phenomenon of the image, produce blur and shadow, and further increase the difficulty of image processing and target recognition.

[0004] The current main image smoke removal methods mainly include three research directions of "based on physical model", "based on traditional image processing technology" and "based on learning method". The algorithm based on physical model removes smoke in reverse by simulating the physical process of smoke in the optical transmission process, but this method often needs a large amount of computing resources and deep understanding of physics, and the adaptive effect is poor in the complex forest scene. The algorithm based on traditional image processing technology includes dark channel prior algorithm, color attenuation prior algorithm proposed by ZHU, nonlinear sinusoidal function improved algorithm based on dark channel designed by Arigela and the like, although each algorithm has achieved certain success, but this kind of method based on manual prior knowledge cannot well solve all problems, and cannot effectively solve the image distortion and texture blur problems caused in the image reconstruction of smoke removal. The learning-based method often uses convolutional neural network or generative adversarial network model, and has the ability of removing smoke through a large amount of training. For example, CAI proposes an end-to-end dehaze model DehazeNet based on convolutional neural network, Yi uses a multi-scale topological network to reconstruct the smoke-free image, ENGIN proposes an improved dehaze network Cycle-dehaze based on a recurrent generative adversarial network, and the like. However, the above neural network models cannot well solve the uneven illumination and color distortion of the image in the forest fire background, and cannot well correct the image ghost caused by the shaking or continuous flight of the unmanned aerial vehicle, which seriously affects the training and test results of the model. Due to the complexity of forest fire behavior and the uncertainty of photography environment, the current mainstream smoke removal models have not reached the real universal application, and still have many defects. SUMMARY

[0005] The application provides a single image smoke fast removal method based on an improved Cycle-Dehaze neural network, which solves the defects of large smoke shielding range, difficult fire feature extraction and low image processing efficiency in forest fire monitoring in the prior art.

[0006] The technical scheme of the application is as follows: a forest fire image smoke removal method based on an improved Cycle-Dehaze neural network, comprising the following steps:

[0007] Step 1, collect similar outdoor scene fire smoke-containing images and clear smoke-free images with information correlation as network training data set, and pretreat all images;

[0008] Step 2, construct and train a fire scene image smoke removal neural network, specifically comprising the following steps:

[0009] Step 2.1, the preprocessed smoke-containing image is subjected to continuous bicubic down-sampling operation to obtain a low-resolution smoke-containing fire scene image F';

[0010] Step 2.2, construct SF-Cycle-Dehaze neural network, and input F' into SF-Cycle-Dehaze neural network;

[0011] The SF-Cycle-Dehaze neural network is based on the improvement of the Cycle-Dehaze neural network, which consists of two generators (G, F) and two discriminators (P x , P y ), wherein X is set as a smoke-containing image, Y is a clear smoke-free image, G is a generator that realizes the domain mapping from X→Y, F is a generator that realizes the mapping from Y→X, (P x , P y ) are discriminators corresponding to the domain, which are used to judge whether the image is true or false, that is, to identify the de-smoking efficiency of the network;

[0012] Step 2.3, input the smoke-containing image F' into the generator G to generate a smoke-free image, and then perform authenticity judgment through the discriminator P x ;

[0013] Step 2.4, generate a smoke-containing image through the generator F after pre-processing the clear smoke-free image, and perform authenticity judgment through the discriminator P y , the smoke-containing image sample and the clear smoke-free image sample are trained in a loop, and the neural network parameters are adjusted according to the discriminator results, so as to improve the final de-smoking efficiency of the neural network;

[0014] Step 3, in the real processing process, the smoke-containing image of the image pre-processed fire field is down-scaled to a low-resolution fire field smoke-containing image by bicubic, and is input into the trained SF-Cycle-Dehaze neural network, after the image is processed by the neural network, it is subjected to simple up-sampling of Laplace pyramid, and is restored to a high-resolution fire field smoke-free image with the same resolution size as the original image, and is output as the final result.

[0015] Further, the specific implementation steps of the pre-processing in step 1 are as follows:

[0016] Step 1.1, map the input single unprocessed image from the RGB color space to the HSV color space, separate and obtain the hue component image H, the brightness component image V and the saturation component image S of the image, and perform virtual multi-exposure processing on the brightness component image V, and the specific implementation method is as follows:

[0017] Step 1.1.1, obtain the pixel gray value set of overexposure and underexposure in the input brightness component image V, and define the two gray value sets in the image as:

[0018]

[0019] Q1 is the set of pixel gray value of overexposure in the image, Q2 is the set of pixel gray value of underexposure in the image, V(x) represents the total set of pixels of the input image, J(x) is the illumination component map of the image V, σ1 and σ2 are the gray threshold values for distinguishing overexposure and underexposure in the forest fire scene;

[0020] Step 1.1.2, let m be the exposure ratio of the image, E be the original exposure of the luminance component image V, E' be the different exposure, then E' = mE, for the luminance component image V, two optimal exposure ratios m chog and m chof are calculated for the pixel set of the bright area and the dark area respectively based on the calculation principle of maximum image information entropy and average gradient; i chog chof

[0021] Image information entropy:

[0022] Maximum average gradient:

[0023]

[0024] Optimal exposure ratio:

[0025] In the above formula, J is the total number of image gray levels, p i represents the probability of the gray level of a certain area in the image I, M*N is the image size, I(x, y) is the pixel value of the image at the pixel point (x, y);

[0026] Step 1.1.3, for the pixel gray value set of the bright area and the dark area in the luminance component image V to be processed, define two luminance conversion functions g and f for generating virtual exposure images V1 and V2 according to the exposure ratio m chog and m chof

[0027]

[0028] Among them, V1 is the enhanced exposure image, V2 is the weakened exposure image, considering that the luminance conversion function will produce truncation when the input image luminance value is large, resulting in loss of bright area, and blind exposure to the area with small input image luminance value will lose the original detail texture features, therefore, the enhanced exposure luminance conversion function g and the weakened exposure luminance conversion function f are defined in detail:

[0029]

[0030]

[0031] In the above formula, (γ, ε) is the threshold separation point of the strong exposure curve, (γ', ε') is the threshold separation point of the weak exposure curve, mchog with m chof are the two best exposure ratios obtained in step 1.1.2; a, b are the conventional brightness conversion curve parameters, and parameters a' and b' are obtained by jointly solving the brightness function curve through the (γ, ε) point and the (1, 1) point;

[0032] Step 1.2, the untreated smoke-containing fire image is respectively substituted into the two brightness conversion functions g and f for the set of overexposed elements and the set of underexposed elements of the image, to perform enhanced exposure processing and weakened exposure processing on the brightness component image V, to obtain an enhanced exposure image V1 and a weakened exposure image V2;

[0033] Step 1.3, for the input multi-exposure image, first realize the alignment of the multi-exposure image in space, and then perform multi-image low-frequency and high-frequency information fusion by using the weight average-based method and the Laplacian pyramid-based method, respectively;

[0034] Step 1.4, after the fusion of the multi-virtual exposure images, a smoke-containing image after image preprocessing is obtained.

[0035] Further, the structures of the two generators in SF-Cycle-Dehaze are the same, and each generator is composed of a Layer1 layer with a convolution kernel size of 9*9 and a step of 1; two Layer2 and Layer3 layers with a convolution kernel size of 3*3 and a step of 3; two SF feature enhancement dense residual blocks SRDB containing dynamic convolution kernels; a Deconvolution layer with a convolution kernel size of 3*3 and a step of 1; a Deconvolution layer with a convolution kernel size of 3*3 and a step of 3, and a Layer4 layer with a convolution kernel size of 9*9 and a step of 1.

[0036] Further, the structures of the two discriminators in SF-Cycle-Dehaze are the same, and each discriminator is composed of two "convolution and activation function LeakREU" layers, a skip connection layer, two "convolution and activation function LeakREU" layers, and a "convolution and activation function Sigmoid" layer.

[0037] Further, the SF feature enhancement dense residual block adopts a mechanism in which a dynamic dense residual block and a CBAM attention mechanism module are coupled, the dynamic dense residual block is composed of three dynamic convolution submodules and a dynamic convolution layer, and the CBAM attention mechanism module is composed of a CAM channel attention module and a SAM spatial attention module.

[0038] Specifically, the dynamic convolution submodule contains a dynamic convolution layer, a BN layer and a ReLU activation function layer, and the process of dynamic convolution can be simply represented as:

[0039] A i= sigmoid(f(S i ))

[0040]

[0041] A is a weight acquisition operation, S is a tensor obtained after the image is subjected to Layer3 static convolution, S i is the feature map of the i-th position of the tensor, f is a nonlinear transformation function realized by a full connection layer, C is a dynamic convolution operation, W(x) represents the convolution kernel parameters adaptively generated by the sample, K represents the number of weights, A k (x) represents the k-th attention weight, and W k represents the k-th convolution kernel parameter.

[0042] Further, the loss function used in the cycle adversarial training process includes an adversarial loss function, a cycle consistency error function and a color loss function, and the mathematical model is:

[0043] L′ total = L GAN (G, P y , X, Y) + L GAN (F, P x , Y, X) + θL cyc (G, F) + δL SSIM

[0044] In the above formula, L cyc (G, F), L GAN (F, P x , X, Y), L GAN (G, P y , X, Y) are the loss functions of the traditional cycle-dehaze neural network; wherein L GAN (F, P x , X, Y) and L GAN (G, P y , X, Y) represent two adversarial loss functions of SF-Cycle-Dehaze, L cyc (G, F) is a cycle consistency error function, and L SSIM represents a color loss function; wherein X and Y in the loss function represent a real image set, X is a real smoke-containing training sample, and Y is a real smoke-free training sample, θ and δ are the weights of the cycle consistency loss and the color loss function.

[0045] Further, the color loss function L SSIM is represented as follows;

[0046] L SSIM = ((I(x, y)) α *(C(x, y))β *(S(x, y)) γ )

[0047] wherein:

[0048] In the above formula, μ represents the mean, σ represents the standard deviation, x and y are the pixel coordinates of the image, σ xy represents the covariance of x and y, c1-c6 are constants to prevent the denominator from being 0, I(x, y), C(x, y), S(x, y) are the brightness information, contrast information and structure information of the image respectively, and α, β, γ are weight coefficients for balancing the importance of brightness, contrast and structure.

[0049] Further, α = 0.8, β = 0.06, and γ = 0.05.

[0050] Further, θ = 0.004 and δ = 0.7.

[0051] The fire smoke removal method based on the SF-Cycle-Dehaze neural network provided by the application fuses non-learning image processing technology and learning image processing technology and is composed of an image preprocessing unit and an image learning processing unit. The method uses two brightness conversion functions to construct a virtual exposure image for two sets of pixel gray value of input images with overexposure and underexposure, and then performs multi-exposure image fusion through high and low frequency information, effectively solving the problems of uneven image illumination and ghost image in the imaging of fire scene background, greatly improving the training efficiency of the image learning processing unit, and enhancing the image smoke removal effect. In addition, the image learning processing unit of the application introduces a dynamic dense residual block (SRDB) designed for smoke feature enhancement on the basis of the traditional Cycle-Dehaze neural network, and introduces a CBAM attention mechanism, which improves the partitioning accuracy of the relevant and irrelevant domains of the image. At the same time, in order to better solve the color distortion and texture blur problems existing in the current neural network smoke removal image reconstruction, the application introduces a color loss function on the basis of the original loss function, enhances the color recognition ability of the SF-Cycle-Dehaze generator and discriminator, and improves the smoke removal image reconstruction ability of the current mainstream model. Finally, in order to realize the high-speed response demand under the background of forest fire warning and emergency, the image will be scaled down by bicubic before entering the non-learning processing unit, and will be restored after being up-sampled by the Laplacian pyramid after processing. The image is processed at a low resolution of 256*256, which greatly shortens the time of removing smoke from the fire scene image and saves valuable time for fire research and judgment.

[0052] The present application realizes real-time acquisition of forest fire disaster situation, provides strong support for feature extraction of fire analysis model, and helps accurate control and trend prediction of forest fire development trend, so as to prevent and rescue measures, and maximize reduce casualties and property losses. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the present application or prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0054] Figure 1 is the flowchart of the improved Cycle-Dehaze neural network smoke removal method provided by the present application in the fire field;

[0055] Figure 2 is the schematic diagram of the image preprocessing method combining virtual multi-exposure technology and pyramid high-low frequency image fusion technology provided by the present application;

[0056] Figure 3 is the SF-Cycle-Dehaze neural network generator network structure schematic diagram provided by the present application;

[0057] Figure 4 is the SF-Cycle-Dehaze neural network generator network parameter schematic diagram provided by the present application;

[0058] Figure 5 is the SF-Cycle-Dehaze neural network discriminator network structure schematic diagram provided by the present application;

[0059] Figure 6 is the feature enhancement dense residual block (SRDB) structure schematic diagram provided by the present application;

[0060] Figure 7 is the comparison diagram of the image generation effect of the improved Cycle-Dehaze based image generation method of the present application and the prior art. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application.

[0062] In view of the defects of color distortion, uneven illumination, blurred texture details and the like of the existing fire scene image smoke removal model, the application provides a single image smoke removal method based on an improved Cycle-Dehaze neural network, so as to solve the defects of large smoke shielding range, poor fire feature extraction effect and low image processing efficiency in forest fire monitoring in the prior art.

[0063] The application provides an image preprocessing method combining virtual multi-exposure technology and pyramid high-low frequency image fusion technology and a fire scene smoke removal model (Smoke-Focused-Cycle-Dehaze, hereinafter referred to as SF-Cycle-Dehaze) based on an improved Cycle-Dehaze neural network. Figure 1 The application provides an image preprocessing method combining virtual multi-exposure technology and pyramid high-low frequency image fusion technology, which comprises the following steps:

[0064] Step 1: using a UAV, a forest tower, an optical remote sensing satellite and the like to obtain a target forest fire real-time smoke-containing image, and inputting the image into an image preprocessing unit.

[0065] Step 2: mapping the input single unprocessed image from an RGB color space to an HSV color space, separating and obtaining a hue component image H, a brightness component image V and a saturation component image S of the image, and performing self-designed virtual multi-exposure processing on the brightness component image V. The specific implementation comprises the following substeps:

[0066] Step 2.1: obtaining a set of pixel gray scale values of overexposure and underexposure in the input brightness component image V, and defining the two sets of gray scale values in the image as:

[0067]

[0068] Q1 is a set of pixel gray scale values of overexposure in the image, Q2 is a set of pixel gray scale values of underexposure in the image, V(x) represents a total set of pixels of the input image, J(x) is an illumination component image of the image V, and sigma1 and sigma2 are gray scale threshold values for distinguishing overexposure and underexposure in a forest fire scene. According to experimental effect comparison: when sigma1 is 0.8 and sigma2 is 0.25, the pixel distinguishing effect is the best.

[0069] Step 2.2: setting m as an image exposure ratio, E as an original exposure amount of the brightness component image V, and E' as different exposure amounts, so that E' = mE. According to the calculation principle based on image information entropy and average gradient maximization, two optimal exposure ratios m chog and m chof of the pixel sets of bright areas and dark areas are calculated.

[0070] Image information entropy:

[0071] Average gradient maximization:

[0072]

[0073] Optimal exposure ratio:

[0074] In the above formula, J is the total number of image gray levels, p i represents the probability of the gray level appearing in a certain region of the image I, M*N is the image size, and I(x, y) is the pixel value of the image at pixel point (x, y).

[0075] Step 2.3: For the pixel gray value set of the brighter region and the darker region in the to-be-processed luminance component image V, define the luminance conversion functions g and f for generating the virtual exposure images V1 and V2 according to the exposure ratios m chog and m chof

[0076]

[0077] where V1 is the enhanced exposure image, and V2 is the weakened exposure image. Considering that the luminance conversion function will produce truncation and cause loss of bright areas when the input image luminance value is large, and that blind exposure to the region with small input image luminance value will lose the original detail texture features, the enhanced exposure luminance conversion function g and the weakened exposure luminance conversion function f are defined in detail as follows:

[0078]

[0079]

[0080] In the above formula, (γ, ε) is the strong exposure curve threshold separation point, and (γ', ε') is the weak exposure curve threshold separation point. m chog and m chof are the two optimal exposure ratios obtained in step 2.1.1; a and b are conventional luminance conversion curve parameters, and the values are reserved as -0.368 and 1.147. The parameters a' and b' are obtained by jointly solving the luminance function curve through the (γ, ε) point and the (1, 1) point. Step 2.4: Substitute the luminance component image V of the unprocessed smoke-containing fire image into the two luminance mapping functions g and f, respectively, to perform enhanced exposure processing and weakened exposure processing, and obtain the enhanced exposure image V1 and the weakened exposure image V2.

[0081] ​Step 3: For the generated virtual multi-exposure image, firstly, the alignment of the multi-exposure images is realized in space, and then the existing weight average-based method and the Laplacian pyramid-based method are used respectively to fuse the low-frequency and high-frequency information of the multi-images.

[0082] Step 4: After the fusion of the multi-virtual exposure images, the image preprocessed image containing the fireworks scene is obtained, and at this time, the problems of non-uniform brightness of the image, the image edge containing shadow, and the UAV image ghosting are solved to some extent. The schematic diagram of the "image preprocessing method combining virtual multi-exposure technology and pyramid high-low frequency image fusion technology" provided above is shown in Figure 2 .

[0083] In a second aspect, the present application also provides a fire scene image smoke removal neural network based on improved Cycle-Dehaze (Smoke-Focused-Cycle-Dehaze, hereinafter referred to as SF-Cycle-Dehaze), and the specific building process of the neural network includes:

[0084] Step 1: Collect similar outdoor scene fire smoke-containing images and clear smoke-free images with information correlation as network training data sets, the data sources include online open source picture databases and self-built picture data sets, and all pictures are subjected to the above-mentioned non-learning preprocessing process to obtain a plurality of training images.

[0085] The SF-Cycle-Dehaze neural network is composed of two generators (G, F) and two discriminators (P x , P y ). Among them, X is assumed to be a smoke-containing image region, Y is a clear image region, G is a generator that realizes the domain mapping from X→Y, F is a generator that realizes the mapping from Y→X, (P x , P y ) are discriminators corresponding to the domain, which are used to judge whether the image is true or false, that is, to identify the smoke removal efficiency of the network. The training process of the network is composed of smoke-containing image and smoke-free image adversarial training, and since the smoke-containing training and smoke-free training processes are the same, the structures of the discriminators and the generators used are basically the same, only the use sequence is different. Therefore, only the F training process for the smoke-containing image is analyzed below.

[0086] Step 2: The image F is subjected to continuous bicubic downscaling downsampling operation to obtain a low-resolution smoke-containing fire scene image F' with a size of 256*256, so as to reduce the processing complexity of the image in the learning processing unit and save valuable time for fire research and judgment. When the size of the input image is uncertain, the SF-Cycle-Dehaze neural network will first scale it to a suitable size, and then perform three downsampling operations to obtain a size of 256*256.

[0087] Specifically, when the size of the input image is greater than 256*256, SF-Cycle-Dehaze will scale it to a slightly larger scale than 256*256, for example, 512*512. Then, the network will perform adaptive number of downsampling operations to obtain an image of 256*256 size. In this process, the network uses a step convolution downsampling method, which can effectively reduce the resolution of the image and reduce the size of the image. When the input image is less than 256*256, the network will first perform upsampling operation to obtain a slightly larger image, and then perform adaptive number of downsampling operations to obtain an image of 256*256 size. In this process, the SF-Cycle-Dehaze network uses a bilinear interpolation upsampling method, which can expand the size of the image to the required size. Then, step convolution is used for downsampling operation to obtain an image of 256*256 size. Finally, F' is input into the SF-Cycle-Dehaze neural network.

[0088] Preferably, the SF-Cycle-Dehaze neural network generator structure diagram is as shown in Figure 3 The SF-Cycle-Dehaze generator is composed of a Layer1 layer with a convolution kernel size of 9*9 and a step of 1; two Layer2 and Layer3 layers with a convolution kernel size of 3*3 and a step of 3; two SF feature enhancement dense residual blocks (SRDB) containing dynamic convolution kernels; a Deconvolution layer with a convolution kernel size of 3*3 and a step of 1; a Deconvolution layer with a convolution kernel size of 3*3 and a step of 3; and a Layer4 layer with a convolution kernel size of 9*9 and a step of 1. The generator parameter diagram is as shown in Figure 4 .

[0089] Preferably, the SF-Cycle-Dehaze neural network discriminator network structure diagram is as shown in Figure 5 The SF-Cycle-Dehaze discriminator is composed of two "Cov+LeakREU" layers (convolution layer and activation function layer), a skip connection layer, two "Cov+LeakREU" layers (convolution layer and activation function layer), and a "Cov+Sigmoid" layer (convolution layer and activation function layer). The discriminator is used to judge whether the input image is a generated image or a real image, so as to make the image more realistic.

[0090] Step 3: Obtain the low-resolution smoke-containing image F' output from the image preprocessing unit, input into the SF-Cycle-Dehaze generator G, and the image will undergo convolution and activation operations of Layerl, Layer2, and Layer3 described in step 2, and then be input into the SF feature enhancement dense residual block.

[0091] Specifically, the SF (Smoke-Focused) feature enhancement dense residual block (SRDB) is the core module of SF-Cycle-Dehaze, which realizes cross-layer fusion of image features and precise distinction between image smoke irrelevant and relevant domains by adopting a mechanism coupling dynamic dense residual block and CBAM attention mechanism module. A structure diagram of the feature enhancement dense residual block (SRDB) is shown in FIG. 1. The dynamic dense residual block is composed of three dynamic convolution sub-modules and a dynamic convolution layer, while the CBAM attention mechanism module is composed of a CAM channel attention module and a SAM spatial attention module. Figure 6

[0092] More specifically, the dynamic convolution sub-module contains a dynamic convolution layer, a BN layer and a ReLU activation function layer. The dynamic convolution layer operates on the basis of not increasing the network depth and width, only increasing the model complexity by a small amount, so as to dynamically adjust the convolution kernel for different samples to adapt to the sample diversity. For complex and variable smoke scenes, dynamic convolution can assign different weights to smoke images with uneven smoke distribution, adaptively generate and correct the corresponding convolution kernel size, enhance the flexibility and generalization ability of the model, and reduce the number of neural network input parameters, improve the network calculation efficiency, and save time cost. The process of dynamic convolution can be simply represented as:

[0093] The process of dynamic convolution can be simply represented as:

[0094] A i =sigmoid(f(S i ))

[0095]

[0096] A above is a weight acquisition operation, S is a tensor obtained after the image is subjected to Layer3 static convolution, S i is the feature map of the i-th position of the tensor, f is a nonlinear transformation function realized by a full connection layer, C is a dynamic convolution operation, W(x) represents a sample-adaptive generated convolution kernel parameter, K represents the number of weights, A k (x) represents the k-th attention weight, and W k represents the k-th convolution kernel parameter.

[0097] The dynamic convolution layer effectively avoids the drawbacks of the traditional convolution layer adjusting the mapping relationship according to the input, and improves the feature extraction performance.

[0098] Step 4: Obtain the output of the image in the dynamic dense residual block and input it into the CBAM attention mechanism module.

[0099] ​Step 5: The image at the output of the two SF feature enhancement dense residual blocks (SRDBs) is input into two Deconvolution layers and a Layer4 layer for deconvolution operation. After completion, the image is output to the generator module, at which time the image is a smoke-free image generated by the generator.

[0100] Step 6: The image enters the discriminator module P x . The first, second, fourth, and fifth layers of the discriminator use convolution to extract the output image features of the image, and the sixth layer outputs a judgment result using a sigmoid function. The judgment is whether the generated image is truly consistent with the real sample distribution. If it is a real image, the output is 1, if it is a generated image, the output is 0, and when the network reaches dynamic balance, the output is 0.5;

[0101] The above is the training process of the smoke-containing image sample, which uses the generator G and the discriminator P x . The training process of the smoke-free image is: the smoke-free image is generated by the generator F to generate a smoke-containing image, and the authenticity is judged by the discriminator P y . In summary, the smoke-containing sample and the smoke-free sample are trained in a cycle, and the network parameters are continuously adjusted according to the discriminator results, thereby improving the final smoke removal efficiency of the neural network.

[0102] Step 7: The loss function of SF-Cycle-Dehaze realizes the combination of adversarial loss, cycle consistency error function and color loss function. Preferably, in order to better overcome the influence of high temperature heat waves and turbulence on the color of the image in the fire scene background and the color distortion problem brought by the traditional Cycle-Dehaze network image reconstruction, the color loss function L SSIM is particularly introduced in the discriminator:

[0103] L SSIM = ((I(x, y)) α *(C(x, y)) β *(S(x, y)) γ )

[0104] Wherein:

[0105] In the above formula, μ represents the mean, σ represents the standard deviation, x and y are the pixel coordinates of the image, σ xy represents the covariance of x and y, c1-c6 are constants to prevent the denominator from being 0, I(x, y), C(x, y), and S(x, y) are the brightness information, contrast information, and structure information of the image respectively, and α, β, and γ are weight coefficients for balancing the importance of brightness, contrast, and structure. After 20,000 rounds of training, the batch results are compared, and when α = 0.8, β = 0.06, and γ = 0.05, the color preservation effect is better.

[0106] The overall loss function of the SF-Cycle-Dehaze neural network is composed of an adversarial loss function, a cycle-consistency error function and a color loss function, and the mathematical model is:

[0107] L′ total =L GAN (G,P y ,X,Y)+L GAN (F,P x ,Y,X)+θL cyc (G,F)+δL SSIM

[0108] In the above formula, L cyc (G,F), L GAN (F,P x ,X,Y), L GAN (G,P y ,X,Y) are the loss functions of the traditional cycle-dehaze neural network. Among them, L GAN (F,P x ,X,Y) and L GAN (G,P y ,X,Y) represent two adversarial loss functions of SF-Cycle-Dehaze, L cyc (G,F) is the cycle-consistency error function. Among them, X and Y in the loss function represent the real image set, X is the real smoke-containing training sample, and Y is the real smoke-free training sample. θ and δ are the weights of the cycle-consistency loss and the color loss function. The loss function parameter values are: θ = 0.004, δ = 0.7.

[0109] In summary, in the real processing process, the SF-Cycle-Dehaze neural network will obtain the smoke-containing fire scene image F after image preprocessing, and perform bicubic downscaling to obtain a low-resolution fire scene smoke-containing image with a resolution of 256*256, in order to improve the image processing speed and strive for rescue time. After the image is processed by the neural network, it will be simply up-sampled by Laplacian pyramid to restore it to a high-resolution smoke-free fire scene image with the same resolution as the original image, and output as the final result.

[0110] More specifically, a low-resolution smoke-free image F" with a resolution of 256*256 is obtained, and a simple upsampling operation of the Laplacian pyramid is performed. In order to reduce the decline of image quality in the process of downscaling and upscaling, the present application follows the traditional Cycle-Dehaze which uses the Laplacian pyramid created by the high-resolution hazy image. In order to obtain a high-resolution smoke-free image, the low-resolution smoke-free image is used to change the top layer of the Laplacian pyramid, and the Laplacian upgrade process is performed as usual. Finally, a high-resolution smoke-free image with the same size as the original image is obtained.

[0111] In a third aspect, the present application also provides a computer program product comprising a computer program which, when executed by a processor, can completely implement the forest fire image smoke removal method as described above.

[0112] Experimental analysis:

[0113] Experimental environment configuration:

[0114] CPU: Intel Core i7-8700K@3.7GHz; GPU: NVIDIA GeForce GTX 1080 Ti 11GB GDDR5X; operating system: Ubuntu 18.04 LTS; deep learning framework: PyTorch 1.9.0 image processing library: OpenCV 4.5.2; other dependencies: NumPy, SciPy, Matplotlib, etc.

[0115] Experimental data set and training parameters:

[0116] Four data sets are used for network training, including the public RESIDE SOTS outdoor haze image data set (a total of 1265 images), the outdoor image part of the IHAZY public data set, the outdoor image part of the OHAZY public data set, and the FLAME forest fire data set based on aerial images opened by the University of Northern Arizona, wherein 80% is used for training, 10% is used for verification, and 10% is used for testing.

[0117] Experimental parameters: training parameters: learning rate is 0.0001, momentum is 0.9, training rounds are 20000 times, batch size is 16, model structure: SF-Cycle-Dehaze neural network model;

[0118] Experimental result comparison:

[0119] The application adopts peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to evaluate the image quality after removing the smoke. In order to evaluate the performance of the algorithm, the algorithm is compared with some mainstream dehazing algorithm models, and the comparison algorithm is De-hazeNet, traditional Cycle-Dehaze and Cycle-GAN. The comparison results are shown in the following table.

[0120] Table 1 is the comparison of PSNR / SSIM evaluation indexes of IHAZY data set:

[0121]

[0122] Table 2 is the comparison of PSNR / SSIM evaluation indexes of OHAZY data set:

[0123]

[0124] Table 3 is the comparison of PSNR / SSIM evaluation indexes of SOTS outdoor part:

[0125]

[0126] Table 4 is the comparison of PSNR / SSIM evaluation indexes of FLAME data set:

[0127]

[0128] Experimental summary:

[0129] The obtained De-hazeNet outdoor de-smoke image has different degrees of smoke residue, and the saturation is higher than that of the original image; although the traditional Cycle-dehaze algorithm has good indoor de-smoke effect, the training effect on the outdoor data set is general, there are still a small amount of smoke residue, and there are color distortion and edge shadow phenomenon; and the saturation of the Cycle-Gan de-smoke image is too high, and in the training of the image containing a large amount of sky and the image containing dense leaves, there are image distortion and color distortion phenomenon. The image preprocessing unit of the algorithm eliminates the shadow and enhances the uneven illumination area. After the SF-Cycle-Dehaze neural network de-smoke, the problems of the above algorithms are effectively improved, and the texture details and color retention are better than the above algorithms. The image generation effect comparison diagram of the image generation method based on SF-Cycle-Dehaze and the prior art is as shown in Figure 7

[0130] ​Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0131] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0132] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks.

[0133] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0134] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A fire image smoke removal method based on an improved Cycle-Dehaze neural network, characterized in that, Comprise the following steps: Step 1, collect similar outdoor scene fire smoke containing image and clear smoke-free image with information correlation as network training data set, and pretreat all images; Step 2, build and train fire scene image smoke removal neural network, specifically comprising the following steps: Step 2.1, after pretreatment, the smoke containing image is obtained by continuous bicubic downscaling sampling operation, and the low resolution smoke containing fire scene image F' is obtained; Step 2.2, build SF-Cycle-Dehaze neural network, and input F' into SF-Cycle-Dehaze neural network; The SF-Cycle-Dehaze neural network is improved based on a Cycle-Dehaze neural network and consists of two generators (G, F) and two discriminators (P x , P y ), wherein X is set as a smoke-containing image, Y is a clear smoke-free image, G is a generator for realizing domain mapping from X to Y, F is a generator for realizing domain mapping from Y to X, (P x , P y ) are discriminators corresponding to the domains, and are used for judging whether an image is true or false, i.e., identifying the smoke removal efficiency of the network. Step 2.

3. Input the smoke-containing image F' to the generator G to generate a smoke-free image, and then pass it through the discriminator P x to make a reality judgment; Step 2.

4. The preprocessed clear smoke-free image is generated into a smoke image by the generator F, and is subjected to the discriminator P y The authenticity is judged, the smoke image sample and the clear smoke-free image sample are cyclically trained, the neural network parameters are continuously adjusted according to the discriminator result, and the final smoke removal efficiency of the neural network is improved. Step 3, in the real processing process, the low resolution smoke containing fire scene image is obtained by bicubic downscaling of the image pretreated smoke containing fire scene image, and is input into the trained SF-Cycle-Dehaze neural network, after the image is processed by the neural network, the high resolution smoke-free image with the original image resolution size is obtained by Laplacian pyramid simple up sampling, and is output as the final result.

2. The fire image smoke removal method based on the improved Cycle-Dehaze neural network according to claim 1, wherein: The specific implementation steps of the pretreatment in step 1 are as follows: Step 1.1, map the input single untreated image from RGB color space to HSV color space, separate and obtain the hue component image H, the brightness component image V and the saturation component image S of the image, and perform virtual multi-exposure processing on the brightness component image V, and the specific implementation method is as follows: Step 1.1.1, obtain the pixel gray value set of overexposure and underexposure in the input brightness component image V, and define the two gray value sets in the graph as: Q1 is the pixel gray value set of overexposure in the image, Q2 is the pixel gray value set of underexposure in the image, V(x) represents the total pixel set of the input image, J(x) is the illumination component graph of the image V, and sigma1 and sigma2 are the gray threshold values for distinguishing overexposure and underexposure in forest fire scene; Step 1.1.2, set m as the image exposure ratio, E as the original exposure of the luminance component image V, and E' as the different exposure, then E' = mE, for the luminance component image V, two kinds of optimal exposure ratios m of the light area and dark area pixel sets are calculated respectively based on the calculation principle of maximum image information entropy and average gradient chog and m chof ; Image information entropy: Maximum average gradient: Optimum exposure ratio: J is the total number of gray levels of the image, p i represents the probability of the gray level of a certain region in the image I, M*N is the image size, and I(x, y) is the pixel value of the image at the pixel point (x, y). Step 1.1.3, define the two luminance conversion functions g and f for generating the virtual exposure images V1, V2, for the set of light and dark pixel gray values in the image V of the luminance component to be processed, as a function of the exposure ratio m chog and m chof Step 1.1.3, define the two luminance conversion functions g and f for generating the virtual exposure images V1, V2, for the set of light and dark pixel gray values in the image V of the luminance component to be processed, as a function of the exposure ratio m Where V1 is the enhanced exposure image, V2 is the weakened exposure image, considering that the brightness conversion function will produce truncation when the input image brightness value is large, resulting in loss of bright area, and blind exposure to the area with small input image brightness value will lose the original detail texture features, so the enhanced exposure brightness conversion function g and the weakened exposure brightness conversion function f are defined in detail: g： f: In the above equation, (ε) is the strong-exposure curve threshold separation point, (′,′) is the weak-exposure curve threshold separation point, m chog and m chof are the two optimal exposure ratios obtained in step 1.1.2; a, b are the conventional luminance conversion curve parameters, and parameters a′ and b′ are obtained by jointly solving the luminance function curve through (γ,ε) and (1,1). Step 1.2, substitute the untreated smoke containing fire image into the two brightness conversion functions g and f for the overexposed element set and the underexposed element set of the image respectively, and perform enhanced exposure processing and weakened exposure processing on the brightness component image V to obtain the enhanced exposure image V1 and the weakened exposure image V2; Step 1.3, for the input multi-exposure image, first realize the alignment of the multi-exposure image in space, and then use the weight average based method and the Laplacian pyramid based method to fuse the low frequency and high frequency information of the multi-image respectively; Step 1.4, after the fusion of the multi-virtual exposure images, the image pretreated smoke containing image is obtained.

3. The fire image smoke removal method based on the improved Cycle-Dehaze neural network of claim 1, characterized in that: The structure of the two generators in SF-Cycle-Dehaze is the same, and each is composed of a Layer1 layer with a convolution kernel size of 9*9 and a step of 1; two Layer2 and Layer3 layers with a convolution kernel size of 3*3 and a step of 3; two SF feature enhancement dense residual blocks SRDB containing dynamic convolution kernels; a Deconvolution layer with a convolution kernel size of 3*3 and a step of 1; a Deconvolution layer with a convolution kernel size of 3*3 and a step of 3; and a Layer4 layer with a convolution kernel size of 9*9 and a step of 1.

4. The fire image smoke removal method based on the improved Cycle-Dehaze neural network according to claim 1, wherein: The structure of the two discriminators in SF-Cycle-Dehaze is the same, and each is composed of two "convolution and activation function LeakREU" layers, a skip connection layer, two "convolution and activation function LeakREU" layers, and a "convolution and activation function Sigmoid" layer.

5. The fire image smoke removal method based on the improved Cycle-Dehaze neural network according to claim 3, wherein: The SF feature enhancement dense residual block adopts a mechanism of coupling a dynamic dense residual block and a CBAM attention mechanism module. The dynamic dense residual block is composed of three dynamic convolution sub-modules and a dynamic convolution layer, and the CBAM attention mechanism module is composed of a CAM channel attention module and a SAM spatial attention module. Specifically, the dynamic convolution sub-module contains a dynamic convolution layer, a BN layer, and a ReLU activation function layer. The process of dynamic convolution can be simply represented as: A i = sigmoid(( i )) The above A is a weight acquisition operation, S is a tensor obtained after the image is subjected to Layer3 static convolution, S i is a feature map of the i-th position of the tensor, f is a nonlinear transformation function realized by a full connection layer, C is a dynamic convolution operation, W(x) represents a convolution kernel parameter adaptively generated by a sample, K represents a weight number, A k () represents the k-th attention weight, W k represents the k-th convolution kernel parameter. 6.The fire image smoke removal method based on the improved Cycle-Dehaze neural network according to claim 1, wherein: The loss function used in the cycle adversarial training process includes an adversarial loss function, a cycle consistency error function, and a color loss function, and the mathematical model is: L' total = L GAN (G, P y , X, Y) + L GAN (F, P x , Y, X) + θL cyc (G, F) + δL SSIM L cyc (G,F), L GAN (F,P x ,X,Y), L GAN (G,P y ,X,Y) is the loss function of the traditional cycle-dehaze neural network; wherein L GAN (F,P x ,X,Y) and L HAN (G,P y ,X,Y) represent two adversarial loss functions that make up SF-Cycle-Dehaze, L cyc (G,F) is a cycle consistency error function, L SSIM represents a color loss function; wherein X and Y in the loss function represent a real image set, X is a real smoke-containing training sample, Y is a real smoke-free training sample, and δ is the weight of the cycle consistency loss and the color loss function.

7. The method of claim 6, wherein the improved Cycle-Dehaze neural network is based on a Cycle-Dehaze neural network. Color loss function L SSIM is represented as follows; L SSIM = ((I(x, y) α * (C(x, y)) β * (S(x, y)) γ ) wherein: In the above equation, μ represents the mean, σ represents the standard deviation, x, y are pixel coordinates of the image, σ xy represents the covariance of x and y, c1-c6 are constants to prevent the denominator from being zero, I(x, y), C(x, y), S(x, y) are the brightness information, contrast information and structure information of the image respectively, and α, β, γ are weight coefficients for balancing the importance of brightness, contrast and structure. 8.The fire image smoke removal method based on the improved Cycle-Dehaze neural network according to claim 7, wherein: α=0.8、β=0.06、γ=0.

05. 9.The fire image smoke removal method based on the improved Cycle-Dehaze neural network according to claim 6, wherein: θ=0.004,δ=0.7.

Images