Image defogging method based on feature attention network

Abstract

The invention provides an image defogging method based on a feature attention network, and belongs to the technical field of image processing. Comprising the following steps: acquiring image data, namely acquiring paired foggy images and fogless images in the same scene; pre-processing the obtained image data to obtain a pre-processed image; constructing a network model according to the preprocessed image; the network model comprises a detail enhancement branch, a trunk branch of the FFA-Net and a feature fusion and image reconstruction branch; inputting a foggy image into the network model to obtain a reconstructed image; two loss functions are adopted to jointly constrain the training of the network. According to the image defogging network based on the feature attention, image detail information can be recovered, the image defogging effect is remarkably improved, and the model has good generalization ability and can adapt to new data sets and tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image defogging method based on a feature attention network. Background Art

[0002] In haze weather, tiny particles in the air affect the propagation of light, causing the intensity of reflected light to weaken, resulting in low contrast, color shift, and blurring of images. This makes it difficult for images to accurately reflect the color and details of objects, seriously affecting image quality and visual effects. Today's commonly used smart devices, such as surveillance cameras, autonomous driving, drones, and satellite detection, all rely on clear images to ensure normal operation. Therefore, how to remove haze and restore image color and improve quality has important research and application value.

[0003] At present, the mainstream algorithms of image dehazing technology can generally be divided into two categories according to their implementation ideas: one is the method based on prior knowledge, and the other is the method based on deep learning.

[0004] Dehazing algorithms based on prior knowledge usually rely on atmospheric scattering models to describe the imaging principle of images in haze environments. This type of method uses mathematical derivation, model parameter estimation and image restoration technology to remove haze and restore clear images. Specifically, the prior knowledge method assumes that the haze effect in the image is caused by the scattering and absorption of light in the atmosphere. By modeling this physical phenomenon, the transmission image and atmospheric light information of the image are estimated. However, this type of algorithm based on prior knowledge usually has certain limitations, especially when facing complex haze scenes, its effect may be relatively limited. Haze images in complex environments often contain a large number of non-uniform and varied scattering characteristics, which may cause the algorithm based on prior knowledge to have image distortion, blur or unsatisfactory restoration effects.

[0005] With the rapid development of deep learning technology, especially the significant progress made in the application of convolutional neural networks (CNN) in the field of large-scale image processing, image dehazing algorithms based on deep learning have gradually become a research hotspot. Different from the traditional physical model-based methods, the dehazing algorithms based on deep learning are trained through a large number of foggy images and corresponding clear images, so that the neural network can automatically learn the features of the image and directly restore the clear image from the input haze image on this basis. This type of algorithm usually shows high performance in dehazing effect, and can effectively remove the effects of haze to different degrees and restore the clarity, color and details of the image.

[0006] Although deep learning-based algorithms have shown excellent performance in image dehazing tasks, their application also faces some challenges. First, the training of deep learning models requires a large amount of high-quality data sets so that the network can learn effective dehazing features. In addition, deep learning models usually contain a large network structure and a large number of parameters, which leads to large consumption of computing resources during training, slow model convergence, and long training time. For practical applications, this means that efficient computing equipment and a lot of time are required to complete model training, especially in scenarios with limited hardware resources. The high computing requirements of deep learning models restrict the popularity of their applications.

[0007] Feature attention network (FFA-Net) is a lightweight image defogging network with fewer parameters and computational overhead, making it suitable for real-time processing and resource-constrained devices, but FFA-Net itself also has certain shortcomings in performance. First, FFA-Net has the problem of insufficient global information extraction and fusion. Secondly, the excessive residual jump structure of FFA-Net will cause the problem of loss of image detail information. Therefore, the present invention uses FFA-Net as the backbone network, and ensures the defogging effect while ensuring the efficiency of the defogging algorithm by adding self-attention mechanism and detail enhancement branch, which significantly improves the image quality. Summary of the invention

[0008] According to the phenomenon that image details are lost or image blur is common in the field of image dehazing, and with the continuous development of deep learning, many algorithm parameters are increasing and the computing power requirements are becoming higher and higher, an image dehazing method based on feature attention network is provided.

[0009] The technical means adopted by the present invention are as follows:

[0010] An image defogging method based on a feature attention network comprises the following steps:

[0011] Step 1: Acquire image data, that is, acquire a pair of foggy images and fog-free images of the same scene; and preprocess the acquired image data to obtain a preprocessed image;

[0012] Step 2: construct a network model based on the preprocessed image; the network model includes: a detail enhancement branch, a main branch of FFA-Net, and a feature fusion and image reconstruction branch;

[0013] Step 3: Input the foggy image into the network model to obtain a reconstructed image;

[0014] Step 4: Use two loss functions to jointly constrain network training.

[0015] Furthermore, the preprocessing comprises the following steps:

[0016] The acquired image is firstly subjected to center cropping, and the acquired image is cropped to a uniform size; the cropped image is randomly rotated N×90°; wherein N=1, 2, 3, 4;

[0017] The rotated image is standardized by subtracting the mean from the value of each pixel and dividing it by the standard deviation, so that the value of each pixel is concentrated around 0 and has a certain value range;

[0018] The standardized images are divided into test set, validation set and training set according to the proportion.

[0019] Furthermore, the uniform size is 240×240.

[0020] Furthermore, the preprocessed image is sent to the detail enhancement branch and the main branch of FFA-Net respectively after passing through the two-dimensional convolution channel.

[0021] Furthermore, the detail enhancement branch first performs multi-scale feature extraction through five dilated convolutions with different dilation rates, concatenates the feature maps obtained after the dilated convolution on the channel, and then concatenates the concatenated feature maps D z Perform channel attention and spatial attention; the outputs of the channel attention and spatial attention branches are fused and added, and the output is represented as O;

[0022] Finally, the output is enhanced through the gating mechanism, and the output is I o ; The gating mechanism includes: a DW convolution and a ReLU function;

[0023] Io = Relu(DW(O));

[0024] Among them, DW represents DW convolution;

[0025] The channel attention includes: global average pooling convolution GMP, convolution layer with convolution kernel size of 1×1, Relu activation function and Sigmoid activation function; the output of the channel attention is recorded as T o , the expression is:

[0026] T o =(Relu(GMP(D z )))×D z ;

[0027] The spatial attention includes: a convolution layer with a convolution kernel of 1×1 and a Sigmoid function; the output of the spatial attention is denoted as S o , the expression is:

[0028] So =Sigmoid(F 1×1 (D z ))×D z ;

[0029] Among them, F 1×1 Represents a convolutional layer with a convolution kernel of 1×1.

[0030] Furthermore, the trunk branch of the FFA-Net first connects in series three residual block Group groups consisting of a channel attention block CA and a pixel attention block PA, then adds a global context extraction module in series after the Group group, and finally connects in series a channel attention block CA and a pixel attention block PA.

[0031] Furthermore, the global context enhancement module includes: a self-attention mechanism and a local attention;

[0032] The self-attention mechanism relies on the query, key and value to process the input sequence, and the scaled dot product attention calculates the similarity between the query and the key; the self-attention mechanism expression is:

[0033]

[0034] Among them, d k represents the adjustment factor; ln represents the input feature; the output of the self-attention mechanism is denoted by G, and the expression is:

[0035] G = Attention(ln) + ln;

[0036] The local attention includes: a convolution layer with a convolution kernel of 3×3 and two convolution layers with a convolution kernel of 1×1; the branch output of the local attention is denoted as S, and the expression is:

[0037] S=F 1×1 (F 1×1 F 3×3 ((In)))+In;

[0038] In order to speed up network training and strengthen features, the initial input of the module is fused with the final output of each branch; the output of the global context module is denoted as N, and the expression is:

[0039] N=S+G.

[0040] Further, the feature fusion and image reconstruction branch includes: a feature fusion module and reconstruction;

[0041] The feature fusion module first adjusts the number of channels through a convolution layer with a convolution kernel of 1×1, and then performs cross-scale fusion through the central module RepBlock of the RepVGG network, and reversely adjusts the two output feature maps through the weights generated by the Sigmoid activation function;

[0042] The reconstruction uses two convolution layers with a convolution kernel of 1×1 to adjust the image channel and generate a clear image; the output of the fusion feature module is recorded as, and the feature maps of the two input branches are X and Y respectively, then the expression is: f =Sigmoid(F 1×1 (X+Y)+RF(F 1×1 (X+Y)))X+(1-Sigmoid(F 1×1 (X+

[0043] Y)+RF(F 1×1 (X+Y))))Y;

[0044] Here, RF represents the processing of RepBlock.

[0045] Furthermore, in step 4, the two loss functions are: L1 loss function and CR loss function;

[0046] The expression of the L1 loss function is:

[0047] L1=||H t -G t ||;

[0048] Among them, H t represents the reconstructed image obtained by the network, G t Indicates the original clear image;

[0049] The CR loss function uses the feature map output by the last layer of the VGG19 network pre-trained on the ImageNet dataset to calculate the loss, and the expression is:

[0050]

[0051] in, Represents the feature map output by the last layer of VGG19;

[0052] The total loss function is:

[0053] L = αL1 + βCR;

[0054] Wherein, α and β both represent equilibrium parameters, α=1, β=0.1.

[0055] Compared with the prior art, the present invention has the following advantages:

[0056] The feature attention-based image defogging network proposed in this paper can restore image detail information and significantly improve the image defogging effect. In addition, the model has good generalization ability and can adapt to new data sets and tasks.

[0057] The model of the present invention has few parameters and low computational overhead, fast convergence speed, and is suitable for real-time processing and resource-constrained devices;

[0058] The present invention adopts the common loss of L1 and CR to constrain the training process of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0060] Figure 1 This is the image defogging flow chart of the present invention.

[0061] Figure 2 This is the overall network structure diagram of the present invention.

[0062] Figure 3 It is the FFA-Net backbone network of the present invention.

[0063] Figure 4 A structural diagram of the branches is enhanced for the present invention details. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0065] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0066] like Figure 1-4 As shown, the present invention provides an image defogging method based on a feature attention network, comprising the following steps:

[0067] Step 1: Acquire image data, that is, acquire a pair of foggy images and fog-free images of the same scene; and preprocess the acquired image data to obtain the preprocessed image; the preprocessing includes the following steps: first perform center cropping on the acquired image, and crop the acquired image to a uniform size; the uniform size is 240×240; perform a random N×90° rotation on the cropped image; where N=1,2,3,4; perform a standardization operation on the rotated image, that is, subtract the mean from the value of each pixel, and then divide it by the standard deviation, so that the value of each pixel is concentrated around 0 and has a certain value range;

[0068] The images after the normalization operation are divided into the test set, validation set and training set according to the proportion. After the preprocessed images pass through the two-dimensional convolution channel, they are sent to the detail enhancement branch and the main branch of FFA-Net respectively.

[0069] Step 2: Build a network model based on the preprocessed image; the network model includes: detail enhancement branch, FFA-Net trunk branch, and feature fusion and image reconstruction branch.

[0070] In this application, the detail enhancement branch first performs multi-scale feature extraction through five dilated convolutions with different dilation rates, concatenates the feature maps obtained after the dilated convolution on the channel, and then concatenates the concatenated feature maps D z Perform channel attention and spatial attention; the outputs of the channel attention and spatial attention branches are fused and added, and the output is represented as O;

[0071] Finally, the output is enhanced through the gating mechanism, and the output is I o ; The gating mechanism includes: a DW convolution and a ReLU function;

[0072] I o =Relu(DW(O));

[0073] Among them, DW represents DW convolution;

[0074] Channel attention includes: global average pooling convolution GMP, convolution layer with convolution kernel size of 1×1, Relu activation function and Sigmoid activation function; the output of channel attention is recorded as T o , the expression is:

[0075] T o =(Relu(GMP(D z )))×D z ;

[0076] Spatial attention includes: a convolution layer with a convolution kernel of 1×1 and a Sigmoid function; the output of spatial attention is denoted as S o , the expression is:

[0077] S o =Sigmoid(F 1×1 (D z ))×D z ;

[0078] Among them, F 1×1 Represents a convolutional layer with a convolution kernel of 1×1.

[0079] In this application, the main branch of FFA-Net first connects three residual block groups consisting of channel attention blocks CA and pixel attention blocks PA in series, then adds a global context extraction module in series after the group group, and finally connects a channel attention block CA and pixel attention block PA in series. The global context enhancement module includes: self-attention mechanism and local attention;

[0080] The self-attention mechanism relies on the query, key and value to process the input sequence, and the scaled dot product attention calculates the similarity between the query and the key; the self-attention mechanism expression is:

[0081]

[0082] Among them, d k represents the adjustment factor; ln represents the input feature; the output of the self-attention mechanism is denoted by G, and the expression is:

[0083] G = Attention(ln) + ln;

[0084] The local attention consists of a convolutional layer with a 3×3 kernel and two convolutional layers with a 1×1 kernel. The branch output of the local attention is denoted as S, and the expression is:

[0085] S=F 1×1 (F 1×1 F 3×3 ((In)))+In;

[0086] In order to speed up network training and strengthen features, the initial input of the module is fused with the final output of each branch; the output of the global context module is denoted as N, and the expression is:

[0087] N=S+G.

[0088] Preferably, the feature fusion and image reconstruction branch includes: a feature fusion module and reconstruction;

[0089] The feature fusion module first adjusts the number of channels through a convolution layer with a convolution kernel of 1×1, and then performs cross-scale fusion through the central module RepBlock of the RepVGG network, and reversely adjusts the two output feature maps through the weights generated by the Sigmoid activation function;

[0090] Reconstruction uses two convolution layers with a convolution kernel of 1×1 to adjust the image channel and generate a clear image; the output of the fusion feature module is recorded as, and the feature maps of the two input branches are X and Y respectively, then the expression is: f =Sigmoid(F 1×1 (X+Y)+RF(F 1×1 (X+Y)))X+(1-Sigmoid(F 1×1 (X+

[0091] Y)+RF(F 1×1 (X+Y))))Y;

[0092] Here, RF represents the processing of RepBlock.

[0093] Step 3: Input the foggy image into the network model to obtain the reconstructed image;

[0094] Step 4: Use two loss functions to jointly constrain the network training. In step 4, the two loss functions are: L1 loss function and CR loss function;

[0095] The expression of L1 loss function is:

[0096] L1=||H t -G t ||;

[0097] Among them, H t represents the reconstructed image obtained by the network, Gt Indicates the original clear image;

[0098] The CR loss function uses the feature map output by the last layer of the VGG19 network pre-trained on the ImageNet dataset to calculate the loss, and the expression is:

[0099]

[0100] in, Represents the feature map output by the last layer of VGG19;

[0101] The total loss function is:

[0102] L = αL1 + βCR;

[0103] Wherein, α and β both represent equilibrium parameters, α=1, β=0.1.

[0104] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image defogging method based on feature attention network, characterized in that: The following steps are involved: Step 1: Obtain image data, that is, obtain a pair of foggy images and fog-free images of the same scene; and preprocessing the acquired image data to obtain a preprocessed image; Step 2: construct a network model based on the preprocessed image; the network model includes: a detail enhancement branch, a trunk branch of FFA-Net, and a feature fusion and image reconstruction branch; Step 3: Input the foggy image into the network model to obtain a reconstructed image; Step 4: Use two loss functions to jointly constrain network training.

2. According to claim 1, the image defogging method based on feature attention network is characterized in that: The pre-processing comprises the following steps: The acquired image is firstly subjected to center cropping, and the acquired image is cropped to a uniform size; the cropped image is randomly rotated N×90°; wherein N=1, 2, 3, 4; The rotated image is standardized by subtracting the mean from the value of each pixel and dividing it by the standard deviation, so that the value of each pixel is concentrated around 0 and has a certain value range; The standardized images are divided into three parts: test set, validation set and training set.

3. The image defogging method based on feature attention network according to claim 2 is characterized in that: The uniform size is 240×240.

4. The image defogging method based on feature attention network according to claim 1, characterized in that: The preprocessed image is sent to the detail enhancement branch and the main branch of FFA-Net respectively after passing through the two-dimensional convolution channel.

5. The image defogging method based on feature attention network according to claim 1 or 4, characterized in that: The detail enhancement branch first performs multi-scale feature extraction through five dilated convolutions with different dilation rates, concatenates the feature maps obtained after the dilated convolution on the channel, and then concatenates the concatenated feature maps D z Perform channel attention and spatial attention; the outputs of the channel attention and spatial attention branches are fused and added, and the output is represented as O; Finally, the output is enhanced through the gating mechanism, and the output is I o ; The gating mechanism includes: a DW convolution and a ReLU function; and o =Relu(DW(O)); Among them, DW represents DW convolution; The channel attention includes: global average pooling convolution GMP, convolution layer with convolution kernel size of 1×1, Relu activation function and Sigmoid activation function; the output of the channel attention is recorded as T o , the expression is: T o (Relu(GMP(D). z )))×D z 100. The spatial attention includes: a convolution layer with a convolution kernel of 1×1 and a Sigmoid function; the output of the spatial attention is denoted as S o , the expression is: S o =Sigmoid(F 1×1 (D z ))×D z ; Among them, F 1×1 Represents a convolutional layer with a convolution kernel of 1×1.

6. The image defogging method based on feature attention network according to claim 1 or 4, characterized in that: The trunk branch of the FFA-Net first connects three residual block Group groups consisting of channel attention blocks CA and pixel attention blocks PA in series, then adds a global context extraction module in series after the Group group, and finally connects a channel attention block CA and a pixel attention block PA in series.

7. The image defogging method based on feature attention network according to claim 1, characterized in that: The global context enhancement module includes: a self-attention mechanism and a local attention mechanism; The self-attention mechanism relies on the query, key and value to process the input sequence, and the scaled dot product attention calculates the similarity between the query and the key; the self-attention mechanism expression is: Among them, d k represents the adjustment factor; ln represents the input feature; the output of the self-attention mechanism is denoted by G, and the expression is: G = Attention(ln) + ln; The local attention includes: a convolution layer with a convolution kernel of 3×3 and two convolution layers with a convolution kernel of 1×1; the branch output of the local attention is denoted as S, and the expression is: S=F 1×1 (F 1×1 F 3×3 ((In)))+In; In order to speed up network training and strengthen features, the initial input of the module is fused with the final output of each branch; the output of the global context module is denoted as N, and the expression is: N=S+G.

8. The image defogging method based on feature attention network according to claim 1, characterized in that: The feature fusion and image reconstruction branch includes: a feature fusion module and reconstruction; The feature fusion module first adjusts the number of channels through a convolution layer with a convolution kernel of 1×1, and then performs cross-scale fusion through the central module RepBlock of the RepVGG network, and reversely adjusts the two output feature maps through the weights generated by the Sigmoid activation function; The reconstruction uses two convolution layers with a convolution kernel of 1×1 to adjust the image channel and generate a clear image; the output of the fusion feature module is recorded as, and the feature maps of the two input branches are X and Y respectively, then the expression is: f =Sigmoid(F 1×1 (X+Y)+RF(F 1×1 (X+Y)))X+(1-Sigmoid(F 1×1 (X+Y)+RF(F 1×1 (X+Y))))Y; Here, RF represents the processing of RepBlock.

9. The image defogging method based on feature attention network according to claim 1, characterized in that: In step 4, the two loss functions are: L1 loss function and CR loss function; The expression of the L1 loss function is: L1=||H t -G t ||; Among them, H t represents the reconstructed image obtained by the network, G t Indicates the original clear image; The CR loss function uses the feature map output by the last layer of the VGG19 network pre-trained on the ImageNet dataset to calculate the loss, and the expression is: in, Represents the feature map output by the last layer of VGG19; The total loss function is: L = αL1 + βCR; Wherein, α and β both represent equilibrium parameters, α=1, β=0.1.

Images