Image defogging method based on feature attention network

Abstract

The application provides an image defogging method based on a feature attention network, and belongs to the technical field of image processing. The method comprises the following steps: acquiring image data, that is, acquiring a pair of foggy images and non-foggy images of the same scene; and pre-processing the acquired image data to obtain pre-processed images; constructing a network model according to the pre-processed images; the network model comprises a detail enhancement branch, a main branch of an FFA-Net and a feature fusion and image reconstruction branch; inputting the foggy images into the network model to obtain reconstructed images; and adopting two kinds of loss functions to jointly constrain the training of the network. The image defogging network based on the feature attention provided in the application can restore image detail information, significantly improves the image defogging effect, and the model has good generalization ability and can adapt to new data sets and tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular, relates to an image defogging method based on a feature attention network. BACKGROUND

[0002] In hazy weather, the tiny particles in the air can affect the propagation of light, causing the intensity of reflected light to weaken, thus causing the image to have problems such as low contrast, color shift, and blurring. This makes it difficult for the image to accurately reflect the color and details of the object, seriously affecting the image quality and visual effect. Today's commonly used intelligent devices, such as surveillance cameras, autonomous driving, drones, and satellite detection, all rely on clear images to ensure normal operation. Therefore, how to remove haze and restore image color to improve quality has important research and application value.

[0003] Currently, the mainstream algorithms of image defogging technology can be divided into two categories according to their different implementation ideas: one is the method based on prior knowledge, and the other is the method based on deep learning.

[0004] The defogging algorithm based on prior knowledge usually relies on the atmospheric scattering model to describe the imaging principle of the image in the hazy environment. This kind of method uses mathematical derivation, model parameter estimation and image restoration technology to remove haze and restore clear images. Specifically, the prior knowledge method assumes that the hazy effect in the image is caused by the scattering and absorption of light in the atmosphere, and by modeling this physical phenomenon, the transmission image and atmospheric light information of the image are estimated. However, this kind of algorithm based on prior knowledge usually has certain limitations, especially when facing complex hazy scenes, its effect may be limited. Hazy images in complex environments often contain a large amount of non-uniform and changeable scattering characteristics, which may cause image distortion, blurring or unsatisfactory recovery effect in the algorithm based on prior knowledge.

[0005] With the rapid development of deep learning technology, especially the application of convolutional neural network (CNN) in large-scale image processing field has made significant progress, the image defogging algorithm based on deep learning has gradually become a research hotspot. Unlike traditional methods based on physical models, the defogging algorithm based on deep learning trains a large number of hazy images and corresponding clear images, so that the neural network can automatically learn the features of the image, and on this basis, directly restores the clear image from the input hazy image. This kind of algorithm usually shows high performance in defogging effect, which can effectively remove the influence of different degrees of haze, and restore the clarity, color and details of the image.

[0006] Although deep learning-based algorithms perform well in image defogging tasks, their application also faces some challenges. First, the training of deep learning models requires a large amount of high-quality data sets so that the network can learn effective defogging features. In addition, deep learning models usually contain a large network structure and a large number of parameters, which leads to large consumption of computing resources, slow convergence speed of the model, and long training time during the training process. For practical applications, this means that efficient computing devices and a large amount of time are needed to complete the training of the model, especially in hardware resource-limited scenarios, the high computational demand of deep learning models restricts the popularity of their application.

[0007] Feature attention network (FFA-Net) is a lightweight image defogging network with fewer parameters and computational overhead, making it suitable for real-time processing and resource-constrained devices. However, FFA-Net itself has certain performance deficiencies. First, FFA-Net has insufficient global information extraction and fusion. Second, the excessive residual skip structure of FFA-Net can cause loss of image detail information. Therefore, the present application uses FFA-Net as the backbone network, increases the self-attention mechanism and detail enhancement branch to ensure the defogging effect while ensuring the efficiency of the defogging algorithm, and significantly improves the image quality. SUMMARY

[0008] In view of the above-mentioned technical problems in the field of image defogging, such as image detail loss or image blur, and with the continuous development of deep learning, many algorithms have more and more parameters and higher computing power requirements, the present application provides an image defogging method based on a feature attention network.

[0009] The technical means adopted by the present application are as follows:

[0010] An image defogging method based on a feature attention network, comprising the following steps:

[0011] Step 1: obtaining image data, i.e., obtaining a pair of foggy images and non-foggy images of the same scene; and pre-processing the obtained image data to obtain pre-processed images;

[0012] Step 2: constructing a network model according to the pre-processed images; the network model comprises a detail enhancement branch, a backbone branch of FFA-Net, and a feature fusion and image reconstruction branch;

[0013] Step 3: inputting the foggy image into the network model to obtain a reconstructed image;

[0014] Step 4: using two loss functions to jointly constrain the training of the network.

[0015] Further, the pre-processing comprises the following steps:

[0016] The acquisition image is first center cropped, and the acquisition image is cropped to a uniform size; the cropped image is randomly rotated by N*90 degrees; wherein N=1, 2, 3, 4;

[0017] The rotated image is subjected to a standardization operation, that is, the value of each pixel is subtracted from the mean and then divided by the standard deviation, so that the value of each pixel is concentrated around 0 and has a certain value range;

[0018] The image after the standardization operation is segmented in proportion, into a test set, a validation set and a training set.

[0019] Further, the uniform size is 240*240.

[0020] Further, the preprocessed image is sent into the detail enhancement branch and the backbone branch of the FFA-Net after passing through a two-dimensional convolution channel.

[0021] Further, the detail enhancement branch first performs multi-scale feature extraction through five dilated convolutions with different hole rates, and concatenates the feature maps obtained by the dilated convolutions in the channel; then the concatenated feature map D z is subjected to channel attention and spatial attention; the outputs of the channel attention and spatial attention branches are fused and added, and the output is represented as O;

[0022] Finally, the output is enhanced through a gating mechanism, and the output is I o ; the gating mechanism includes a DW convolution and a Relu function;

[0023] Io=Relu(DW(O));

[0024] wherein DW represents a DW convolution;

[0025] The channel attention includes a global average pooling convolution GMP, a convolution layer with a convolution kernel size of 1*1, a Relu activation function and a Sigmoid activation function; the output of the channel attention is denoted as T o , and the expression is:

[0026] T o =(Relu(GMP(D z )))*D z ;

[0027] The spatial attention includes a convolution layer with a convolution kernel of 1*1 and a Sigmoid function; the output of the spatial attention is denoted as S o , and the expression is:

[0028] So = Sigmoid(F 1×1 (D z )) x D z ;

[0029] wherein, F 1×1 represents a convolution layer with a convolution kernel of 1x1.

[0030] Further, the backbone branch of the FFA-Net firstly concatenates three residual blocks Group composed of channel attention blocks CA and pixel attention blocks PA, secondly concatenates an additional global context extraction module after the Group, and finally concatenates a channel attention block CA and a pixel attention block PA.

[0031] Further, the global context enhancement module comprises a self-attention mechanism and a local attention.

[0032] The self-attention mechanism depends on query, key and value to process the input sequence, and scales the dot product attention calculation between the query and the key. The expression of the self-attention mechanism is:

[0033]

[0034] wherein, d k represents an adjustment factor; ln represents an input feature record; the output record of the self-attention mechanism is G, and the expression is:

[0035] G = Attention(ln) + ln

[0036] The local attention comprises a convolution layer with a convolution kernel of 3x3 and two convolution layers with a convolution kernel of 1x1; the branch output record of the local attention is S, and the expression is:

[0037] S = F 1×1 (F 1×1 F 3×3 ((In)) + In

[0038] In order to speed up the network training and strengthen the features, the initial input of the module is fused and connected with the final output of each branch; the output record of the global context module is N, and the expression is:

[0039] N = S + G

[0040] Further, the feature fusion and image reconstruction branch comprises a feature fusion module and reconstruction.

[0041] The feature fusion module first adjusts the channel number through a convolution layer with a 1*1 convolution kernel, then performs cross-scale fusion through a center module RepBlock of the RepVGG network, and generates two feature maps through a Sigmoid activation function to reversely adjust the output.

[0042] The reconstruction adopts two convolution layers with a 1*1 convolution kernel to adjust the image channel and generate a clear image; the output of the fusion feature module is denoted as O, the feature maps of the two input branches are denoted as X and Y respectively, and the expression is as follows: O = Sigmoid(F(X+Y)+RF(F(X+Y)))X+(1-Sigmoid(F(X+Y)+RF(F(X+Y)))Y f 1×1 1×1 1×1

[0043] 1×1

[0044] Wherein, RF represents the processing of RepBlock.

[0045] Further, in step 4, two loss functions are L1 loss function and CR loss function.

[0046] The expression of the L1 loss function is as follows:

[0047] L1 = ||H t -G t ||;

[0048] Wherein, H t represents the reconstructed image obtained by the network, and G t represents the original clear image.

[0049] The CR loss function uses the feature map output by the last layer of the VGG19 network pre-trained on the ImageNet dataset to calculate the loss, and the expression is as follows:

[0050]

[0051] Wherein, represents the feature map output by the last layer of VGG19;

[0052] The total loss function is as follows:

[0053] L = alpha L1 + beta CR;

[0054] Wherein, alpha and beta represent balance parameters, alpha = 1, and beta = 0.1.

[0055] Compared with the prior art, the present application has the following advantages:

[0056] ​​​​​​The feature attention-based image dehazing network proposed in this paper can restore image detail information and significantly improve the image dehazing effect. In addition, the model has good generalization ability and can adapt to new data sets and tasks.

[0057] The model of the present invention has few parameters and low computational overhead, fast convergence speed, and is suitable for real-time processing and resource-constrained devices;

[0058] The present invention adopts the joint loss of L1 and CR to constrain the training process of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0060] Figure 1 This is the image defogging flow chart of the present invention.

[0061] Figure 2 This is the overall network structure diagram of the present invention.

[0062] Figure 3 It is the FFA-Net backbone network of the present invention.

[0063] Figure 4 A structural diagram of the branches is enhanced for the present invention details. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0065] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and in the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0066] As shown in Figures 1-4 , the present application provides a feature attention network-based image defogging method, comprising the following steps:

[0067] Step 1: obtaining image data, i.e. obtaining a pair of foggy images and non-foggy images of the same scene; and pre-processing the obtained image data to obtain pre-processed images; the pre-processing includes the following steps: first, center cropping the obtained image, and cropping the obtained image to a uniform size; the uniform size is 240x240; the cropped image is randomly rotated by Nx90 degrees; wherein N = 1, 2, 3, 4; the rotated image is subjected to a standardization operation, i.e. the value of each pixel is subtracted from the mean value and then divided by the standard deviation, so that the value of each pixel is concentrated around 0 and has a certain value range;

[0068] The image after the standardization operation is segmented in proportion to: a test set, a validation set and a training set. After the pre-processed image passes through a two-dimensional convolution channel, it is respectively sent to a detail enhancement branch and a main branch of FFA-Net.

[0069] Step 2: constructing a network model according to the pre-processed image; the network model includes: a detail enhancement branch, a main branch of FFA-Net and a feature fusion and image reconstruction branch.

[0070] In the present application, the detail enhancement branch first extracts multi-scale features through five dilated convolutions with different hole rates, and concatenates the feature maps obtained by the dilated convolutions in the channel; then the concatenated feature map D z is subjected to channel attention and spatial attention; the outputs of the channel attention and spatial attention branches are fused and added, and the output is represented as O;

[0071] Finally, the output is enhanced through a gating mechanism, and the output is I o ; the gating mechanism includes a DW convolution and a Relu function;

[0072] I o = Relu(DW(O));

[0073] wherein DW represents a DW convolution;

[0074] The channel attention comprises: a global average pooling convolution GMP, a convolution layer with a convolution kernel size of 1*1, a Relu activation function and a Sigmoid activation function; and an output of the channel attention is denoted as T o , and an expression is as follows:

[0075] T o = (Relu(GMP(D z ))) * D z ;

[0076] The spatial attention comprises: a convolution layer with a convolution kernel of 1*1 and a Sigmoid function; and an output of the spatial attention is denoted as S o , and an expression is as follows:

[0077] S o = Sigmoid(F 1×1 (D z )) * D z ;

[0078] wherein F 1×1 represents a convolution layer with a convolution kernel of 1*1.

[0079] In the present application, the backbone branch of the FFA-Net firstly concatenates three residual blocks Group composed of a channel attention block CA and a pixel attention block PA, secondly concatenates a global context extraction module after the Group, and finally concatenates a channel attention block CA and a pixel attention block PA. The global context enhancement module comprises: a self-attention mechanism and a local attention;

[0080] The self-attention mechanism depends on query, key and value to process the input sequence, and calculates the similarity between the query and the key by scaling dot product attention. An expression of the self-attention mechanism is as follows:

[0081]

[0082] wherein d k represents an adjustment factor; ln represents an input feature; an output of the self-attention mechanism is denoted as G, and an expression is as follows:

[0083] G = Attention(ln) + ln;

[0084] The local attention includes a convolution layer with a convolution kernel of 3*3 and two convolution layers with a convolution kernel of 1*1; the branch output of the local attention is denoted as S, and the expression is:

[0085] S=F 1×1 1×1 F 3×3 ((In)))+In;

[0086] In order to accelerate network training and strengthen features, the initial input of the module is fused and connected with the final output of each branch; the output of the global context module is denoted as N, and the expression is:

[0087] N=S+G.

[0088] As preferred, the feature fusion and image reconstruction branch includes a feature fusion module and a reconstruction;

[0089] The feature fusion module first adjusts the channel number through a convolution layer with a convolution kernel of 1*1, then performs cross-scale fusion through the center module RepBlock of the RepVGG network, and reversely adjusts the two feature maps of the output through the weight generated by the Sigmoid activation function;

[0090] The reconstruction adjusts the image channel through two convolution layers with a convolution kernel of 1*1 to generate a clear image; the output of the fusion feature module is denoted as O, the feature maps of the two input branches are X and Y respectively, and the expression is: f O=Sigmoid(F 1×1 (X+Y)+RF(F 1×1 (X+Y)))X+(1-Sigmoid(F 1×1 (X+

[0091] Y)+RF(F 1×1 (X+Y))))Y;

[0092] Wherein, RF represents the processing of RepBlock.

[0093] Step 3: input the foggy image into the network model to obtain the reconstructed image;

[0094] Step 4: two loss functions are used to jointly constrain the training of the network. In step 4, the two loss functions are L1 loss function and CR loss function;

[0095] The expression of the L1 loss function is:

[0096] L1=||H t -G t ||;

[0097] Wherein, H t represents the reconstructed image obtained by the network, and G​t represents the original clear image;

[0098] The CR loss function uses the feature map output by the last layer of the VGG19 network pre-trained on the ImageNet dataset to calculate the loss, and the expression is:

[0099]

[0100] wherein, represents the feature map output by the last layer of the VGG19 network;

[0101] The total loss function is:

[0102] L = aL1 + bCR;

[0103] wherein, a and b both represent balance parameters, a = 1, b = 0.1.

[0104] The above-mentioned embodiment numbers of the application only serve for description, and do not represent the advantages and disadvantages of the embodiments. In the above-mentioned embodiments of the application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments. In the several embodiments provided in the present application, it should be understood that the disclosed technical contents can be implemented by other ways.

[0105] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the application, and not to limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. An image defogging method based on feature attention network, characterized in that: The following steps are involved: Step 1: Obtain image data, that is, obtain a pair of foggy and fog-free images of the same scene; and preprocessing the acquired image data to obtain a preprocessed image; Step 2: Construct a network model based on the preprocessed image; the network model includes: a detail enhancement branch, a main branch of FFA-Net, and a feature fusion and image reconstruction branch; Step 3: Input the foggy image into the network model to obtain a reconstructed image; Step 4: Use two loss functions to jointly constrain network training; After the preprocessed image passes through the two-dimensional convolution channel, it is sent to the detail enhancement branch and the main branch of FFA-Net respectively; the detail enhancement branch first performs multi-scale feature extraction through five dilated convolutions with different void rates, and concats the feature maps obtained after the dilated convolution on the channel; then the spliced ​​feature maps are Perform channel attention and spatial attention; the outputs of the channel attention and spatial attention branches are fused and added, and the output is expressed as ; Finally, the output is enhanced through the gating mechanism, and the output is I o ; The gating mechanism includes: a DW convolution and ReLU function; ； in, express convolution; The channel attention includes: global average pooling convolution GMP, convolution layer with convolution kernel size of 1×1, Relu activation function and Sigmoid activation function; the output of the channel attention is recorded as ， The expression is: ； The spatial attention includes: a convolution layer with a convolution kernel of 1×1 and a Sigmoid function; the output of the spatial attention is recorded as , the expression is: ； in, Indicates a convolution layer with a convolution kernel of 1×1; The main branch of the FFA-Net first connects three residual block groups consisting of a channel attention block CA and a pixel attention block PA in series, then adds a global context extraction module in series after the group group, and finally connects a channel attention block CA and a pixel attention block PA in series; The global context extraction module includes: self-attention mechanism and local attention; The feature fusion and image reconstruction branch includes: a feature fusion module and reconstruction; The feature fusion module first adjusts the number of channels through a convolution layer with a convolution kernel of 1×1, then performs cross-scale fusion through the central module RepBlock of the RepVGG network, and reversely adjusts the two output feature maps through the weights generated by the Sigmoid activation function; The reconstruction uses two convolution layers with a convolution kernel of 1×1 to adjust the image channel and generate a clear image. The output of the fusion feature module is recorded as , and the feature maps of the two input branches are X and Y respectively. Then the expression is: ； in, Indicates the processing of RepBlock.

2. The image defogging method based on feature attention network according to claim 1 is characterized in that: The pretreatment comprises the following steps: The acquired image is firstly subjected to center cropping to crop the acquired image to a uniform size; the cropped image is randomly rotated by N×90°, where N=1, 2, 3, or 4; The rotated image is normalized by subtracting the mean from the value of each pixel and dividing it by the standard deviation, so that the value of each pixel is concentrated around 0 and has a certain value range; The standardized images are divided into three parts: test set, validation set and training set.

3. The image defogging method based on feature attention network according to claim 2 is characterized in that: The uniform size is 240×240.

4. The image defogging method based on feature attention network according to claim 1, characterized in that: The self-attention mechanism relies on the query, key and value to process the input sequence, and the scaled dot product attention is used to calculate the similarity between the query and the key. The self-attention mechanism expression is: ； in, represents the regulating factor; Denotes the input feature; the output of the self-attention mechanism is denoted as G, and the expression is: ； The local attention includes: a convolution layer with a convolution kernel of 3×3 and two convolution layers with a convolution kernel of 1×1; the branch output of the local attention is recorded as , the expression is: ； In order to speed up network training and strengthen features, the initial input of the module is fused with the final output of each branch. The output of the global context extraction module is denoted as N, and the expression is: 。 5. The image defogging method based on feature attention network according to claim 1, characterized in that: In step 4, the two loss functions are: L1 loss function and CR loss function; The expression of the L1 loss function is: ； in, represents the reconstructed image obtained by the network, Indicates the original clear image; The CR loss function uses the feature map output by the last layer of the VGG19 network pre-trained on the ImageNet dataset to calculate the loss, and the expression is: ； in, Represents the feature map output by the last layer of VGG19; The total loss function is: ； in, and are equilibrium parameters, , .

Images