Image defogging method, device and equipment
By introducing residual double attention fusion module into the image defog removal model, combining channel and pixel attention module, the problem of poor fog removal effect in the existing technology is solved, and better image defog removal effect and model training are achieved. efficiency.
Patent Information
- Application Number
- CN202111091274.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-17
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2041-09-17
AI Technical Summary
The existing image defog removal algorithm is based on deep learning and cannot effectively distinguish between thin and dense fog, close-up and long-range pixel weights, resulting in poor fog removal and loss of image details.
A pre-trained defogging model is adopted, which includes multiple residual groups, which are composed of a series of residual double attention fusion modules, which include residual blocks, channel attention modules and pixel attention modules. Through the fusion of these modules, the enhancement and global dependence of the channel graph are achieved, and pixel features are obtained, the number of parameters is reduced, and the defog removal effect is improved.
Improve the fog removal effect, retain the details of the image, simplify the complexity of the model, and improve the model training efficiency.
Smart Images

Figure CN113947537B_ABST
Abstract
Description
Technical Field
[0001] The present application mainly relates to the field of image processing technology, and more specifically to an image defogging method, device and equipment. Background Art
[0002] In recent years, with the development of society, haze has become a current atmospheric phenomenon and is ubiquitous in computer vision. Due to the presence of numerous suspended particles, light is reflected during propagation, resulting in blurry outdoor images, color distortion, and reduced contrast. Advanced image processing tasks, such as target detection, target recognition, and industrial Internet of Things, all require clear images as input. Foggy images will affect the quality and robustness of subsequent advanced tasks; therefore, as a preprocessing image task, image dehazing is a classic image processing problem and has always been a hot topic for researchers.
[0003] At present, image dehazing algorithms are mainly based on deep learning methods. The deep learning-based dehazing network treats channel and pixel features equally, but haze is non-homogeneous. For example, for thin fog and thick fog, the pixel weights of near and far views should be significantly different. Therefore, the dehazing network treats channel and pixel features equally, resulting in poor dehazing effect. The dehazed image still inevitably retains the deep haze of the image and loses the details of the image. Summary of the invention
[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide an image defogging method, device and equipment.
[0005] In a first aspect, the present invention provides an image defogging method, the method comprising:
[0006] Obtain a foggy image to be processed;
[0007] Inputting the foggy image to be processed into a pre-trained defogging model to obtain a fog-free image corresponding to the foggy image to be processed;
[0008] It is characterized in that the pre-trained dehazing model includes multiple residual groups, the residual group includes several serially connected residual dual attention fusion modules, the residual dual attention fusion module includes a residual block, a first convolutional layer, a channel attention module, a pixel attention module and a second convolutional layer, the output of the residual block is connected to the input of the channel attention module and the pixel attention module respectively through the first convolutional layer, the output of the channel attention module and the pixel attention module is fused and then output processed, so as to realize that the output of the residual dual attention fusion module obtains pixel features while enhancing global dependencies in each channel map.
[0009] Furthermore, the defogging model includes three residual groups, and the three residual groups are connected within a channel according to the output from back to front.
[0010] Furthermore, the residual group includes three residual dual attention fusion modules.
[0011] Furthermore, the outputs of the channel attention module and the pixel attention module are added element by element to the input of the residual block and then input into the second convolutional layer for fusion to obtain the output of the residual dual attention fusion module.
[0012] Furthermore, the defogging model also includes a feature extraction convolution layer, a channel attention module, a pixel attention module and an output convolution layer. The foggy image to be processed enters the residual group after the features are extracted by the feature extraction convolution layer. After being processed by the residual group, it enters the channel attention module, the pixel attention module and the output convolution layer in turn for processing to obtain output features. The output features are added element by element to the foggy image to be processed to obtain a fog-free image.
[0013] Furthermore, the training of the defogging model includes:
[0014] Obtain the RESIDE dataset, and randomly select 6000 pairs of foggy and non-fog images from the RESIDE dataset to construct a training sample set;
[0015] The pre-established neural network is trained using the training sample set.
[0016] Furthermore, the loss function L of the neural network is expressed as:
[0017]
[0018] In the formula, N is the number of training samples, is the real clear image of the i-th training sample, is the haze-free image estimated by the neural network for the i-th training sample.
[0019] In a second aspect, the present invention further provides an image defogging device, the device comprising:
[0020] A foggy image acquisition module to be processed, used to acquire the foggy image to be processed;
[0021] A defogging processing module, used for inputting the image to be defogged into a defogging model for processing, and outputting a fog-free image corresponding to the foggy image to be processed;
[0022] The dehazing model includes multiple residual groups, each of which includes several residual dual attention fusion modules connected in series, and each of which includes a residual block, a first convolutional layer, a channel attention module, a pixel attention module and a second convolutional layer. The output of the residual block is connected to the input of the channel attention module and the pixel attention module respectively via the first convolutional layer, and the output of the residual block is connected to the input of the channel attention module and the pixel attention module respectively via the first convolutional layer. The output of the channel attention module and the pixel attention module are fused to realize that the output of the residual dual attention fusion module obtains pixel features while enhancing global dependencies in each channel map.
[0023] Furthermore, the outputs of the channel attention module and the pixel attention module are added element by element to the input of the residual block and then input into the second convolutional layer for fusion to obtain the output of the residual dual attention fusion module.
[0024] In a third aspect, the present invention further provides a device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image dehazing method described in any one of the first aspects is implemented.
[0025] Compared with the prior art, the present invention has the following beneficial effects:
[0026] First, the present invention improves the convolutional neural network with a fixed receptive field, and takes the residual dual attention fusion module as the basic module. The residual dual attention fusion module is composed of the fusion of the residual block, the channel attention module and the pixel attention module. The relevant features of different channel maps are combined, and each channel map enhances the global dependency and obtains the pixel features at the same time, which reduces the number of parameters while retaining better details and improves the defogging effect.
[0027] Second, the present invention adopts an end-to-end dehazing network, and only three residual dual attention fusion modules are set inside the residual group, which reduces the complexity of the model and improves the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 is a flow chart of an image defogging method provided by an embodiment of the present invention;
[0029] Figure 2 is a schematic diagram of the structure of a defogging model provided by an embodiment of the present invention;
[0030] Figure 3 is a schematic diagram of a channel attention module provided by an embodiment of the present invention;
[0031] Figure 4 is a schematic diagram of a pixel attention module provided by an embodiment of the present invention;
[0032] Figure 5 is a schematic diagram of a residual group provided by an embodiment of the present invention;
[0033] Figure 6 is a schematic diagram of a residual dual attention fusion module provided by an embodiment of the present invention;
[0034] Figure 7 This is a comparison chart of the defogging effects of the image defogging method provided by an embodiment of the present invention and other methods. DETAILED DESCRIPTION
[0035] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0036] Embodiment 1:
[0037] like Figure 1 As shown, the present invention provides an image dehazing method. The method establishes an image dehazing model based on a neural network, but improves the convolutional neural network with a fixed receptive field, and takes the residual dual attention fusion module as the basic module. The residual dual attention fusion module is composed of a residual block and a fusion of channel and pixel attention modules.
[0038] Specifically, Figure 2 As shown, the defogging model includes 1 feature extraction convolution layer, 3 residual groups, a channel attention module, a pixel attention module and 2 output convolution layers. The 3 residual groups are connected within the channel according to the output from back to front; the foggy image to be processed enters the residual group after the feature extraction convolution layer extracts the features, and after being processed by the residual group, it enters the channel attention module, the pixel attention module and the 2 output convolution layers in turn to obtain the output features, and the output features are added element by element to the foggy image to be processed to obtain a fog-free image.
[0039] like Figure 3 As shown in the figure, the channel attention module is input by the input feature into the global average pooling, convolution layer, ReLU activation function, convolution layer, Sigmoid activation function, and the output weight is multiplied element by element with the input feature to obtain the output of the channel attention module. The expression is as follows:
[0040]
[0041]
[0042]
[0043] In the formula, Z c(x, y) represents the input Z of the c channel at the position (x, y) c The pixel value of , c∈{R, G, B}; after global average pooling, the dimension of the feature map changes from C×H×W to C×1×1; δ is the ReLU activation function, σ is the Sigmoid activation function, It is element-by-element multiplication; the channel attention module input F c To the channel attention module output The mapping function between CAB .
[0044] The first convolution layer of the channel attention module uses 8 convolution kernels of size 1*1, and the second convolution layer uses 64 convolution kernels of size 1*1.
[0045] like Figure 4 As shown in the figure, the pixel attention module is composed of the input feature input convolution layer, ReLU activation function, convolution layer, Sigmoid activation function, and the output weight is multiplied element by element with the input feature to obtain the output of the pixel attention module. Its expression is as follows:
[0046] F PA =σ(Conv(δ(Conv(F)))) (4)
[0047]
[0048] F PAB =H PAB (F) (6)
[0049] Where F PA is the feature weight of the output, the dimension changes from C×H×W to 1×H×W, the pixel attention module input F to the channel attention module output F PAB The mapping function between PAB .
[0050] The first convolution layer of the pixel attention module uses 8 convolution kernels of size 1*1, and the second convolution layer uses 1 convolution kernel of size 1*1. The other convolution layers use 64 convolution kernels of size 3*3.
[0051] like Figure 5 As shown in the figure, the residual group includes three serially connected residual dual attention fusion modules and one convolutional layer. The input features are input into the residual dual attention fusion module and the convolutional layer. The output result and the input features are added element by element to obtain the output of the residual group. The output expression of the residual group is as follows:
[0052] F g,m =H RDAFM (F g,m-1 ) (7)
[0053]
[0054] F g =H RG (F g,0 ) (9)
[0055] In the formula, F g,m-1 and F g,m The distribution is the input and output of the mth residual dual attention fusion module in the gth residual group, g = 1, 2, 3, m = 1, 2, 3; the input F of the residual dual attention fusion module g,m-1 To the output F of the residual dual attention fusion module g,m The mapping function between RDAFM ; Input F of the residual group g,0 Output F to the residual group g The mapping function between RG .
[0056] like Figure 6 As shown, the residual dual attention fusion module includes a residual block, a first convolutional layer, a channel attention module, a pixel attention module and a second convolutional layer. The residual block includes a convolutional layer and a ReLU activation function. The output of the residual block is connected to the input of the channel attention module and the pixel attention module respectively through the first convolutional layer. The output fusion of the channel attention module and the pixel attention module is used to realize the output of the residual dual attention fusion module and the input of the residual block after element-by-element addition and input into the second convolutional layer for fusion, so as to obtain the output of the residual dual attention fusion module, so that the output of the residual dual attention fusion module obtains pixel features while enhancing the global dependency in each channel map. The output of the residual dual attention fusion module is expressed as follows:
[0057]
[0058] F * =Conv(F RB ) (11)
[0059]
[0060] F RDAFM =H RDAFM (F) (13)
[0061] in, is element-by-element addition, F RB is the output of the residual block, F * It is the input of the attention module, from the input F of the residual dual attention fusion module to the output F of the residual dual attention fusion moduleRDAFM The mapping function between RDAFM .
[0062] The training of the defogging model includes the following steps: obtaining the RESIDE dataset, randomly selecting 6000 pairs of foggy images and fog-free images from the RESIDE dataset to construct a training sample set, using the training sample set to train the neural network to obtain the defogging model, and when using it, obtaining the foggy image to be processed and inputting it into the defogging model to obtain a fog-free image.
[0063] The loss function L of the neural network is expressed as:
[0064]
[0065] Where N is the number of training samples, is the real clear image of the i-th training sample, is the haze-free image estimated by the neural network for the i-th training sample.
[0066] In the neural network, the Adam optimizer is used to initialize the network weight parameters, where the default values of β1 and β2 are 0.9 and 0.999 respectively; the initial learning rate α is set to 1×10 -4 , use the cosine annealing strategy to update the learning rate and adjust the learning rate from the initial value to 0:
[0067]
[0068] In the formula, T is the total number of batches, α is the initial learning rate, t is the current batch, and α t is the learning rate for adaptive updates.
[0069] For each sample image in the training set of the input dehazing network model, the total loss of the difference between the real clear image and the dehazing restored by the network is first calculated using forward propagation, and then the weight parameters are updated according to the Adam optimizer; a total of 1×10 5 Steps, every 200 steps as a batch, a total of 500 batches, repeat the above steps until the set maximum step size is reached, and the trained defogging network model is obtained, whose expression is:
[0070] F0=Conv(I) (16)
[0071] F g =H RG (F g-1 ) (17)
[0072]
[0073]
[0074] Where I is the foggy input image, F g-1 and F g are the input and output of the g-th residual group, g = 1, 2, 3; Indicates the connection operation within the channel. is the restored output image.
[0075] Embodiment 2:
[0076] In this embodiment, an image defogging device is also provided, the device comprising:
[0077] An image acquisition module, used for acquiring a foggy image to be processed;
[0078] An image defogging module is used to input the image to be defogged into a defogging model for processing, and output a fog-free image corresponding to the foggy image to be processed;
[0079] The dehazing model includes multiple residual groups, each of which includes several serially connected residual dual attention fusion modules. The residual dual attention fusion module includes a residual block, a first convolutional layer, a channel attention module, a pixel attention module and a second convolutional layer. The output of the residual block is connected to the input of the channel attention module and the pixel attention module respectively via the first convolutional layer. The output of the residual block is connected to the input of the channel attention module and the pixel attention module respectively via the first convolutional layer. The output of the channel attention module and the pixel attention module are added to the input of the residual block element by element and then input into the second convolutional layer for fusion to obtain the output of the residual dual attention fusion module, which is used to realize that the output of the residual dual attention fusion module obtains pixel features while enhancing global dependencies in each channel map.
[0080] Embodiment 3:
[0081] This embodiment also provides a device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the image defogging method described in Embodiment 1 is implemented when the processor executes the computer program.
[0082] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.
[0083] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0084] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0085] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0086] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A method for defogging an image, the method comprising: Obtain a foggy image to be processed; Inputting the foggy image to be processed into a pre-trained defogging model to obtain a fog-free image corresponding to the foggy image to be processed; The method is characterized in that the pre-trained dehazing model includes a plurality of residual groups, the residual group includes a plurality of residual dual attention fusion modules connected in series, the residual dual attention fusion module includes a residual block, a first convolutional layer, a channel attention module, a pixel attention module, and a second convolutional layer, the output of the residual block is respectively connected to the input of the channel attention module and the pixel attention module through the first convolutional layer, the output of the channel attention module and the pixel attention module are fused and then output processed, so as to realize that the output of the residual dual attention fusion module obtains pixel features while enhancing global dependency in each channel map; Among them, the channel attention module is input by the input feature global average pooling, convolution layer, ReLU activation function, convolution layer, Sigmoid activation function, and the output weight and input feature are multiplied element by element to obtain the output of the channel attention module. Its expression is as follows: In the formula, Representatives in Positional Channel Input The pixel value of ; After global average pooling, the dimension of the feature map is becomes ; is the ReLU activation function, is the Sigmoid activation function, Is element-by-element multiplication; channel attention module input To the channel attention module output The mapping function between ; The pixel attention module is composed of the input feature input convolution layer, ReLU activation function, convolution layer, Sigmoid activation function, and the output weight is multiplied element by element with the input feature to get the output of the pixel attention module. Its expression is as follows: In the formula is the output feature weight, and the dimension is becomes , pixel attention module input To the channel attention module output The mapping function between .
2. The image defogging method according to claim 1, characterized in that: The defogging model includes three residual groups, and the three residual groups are connected within a channel according to the output from back to front.
3. The image defogging method according to claim 1, characterized in that: The residual group includes 3 residual dual attention fusion modules.
4. The image defogging method according to claim 1, characterized in that: The outputs of the channel attention module and the pixel attention module are added element by element to the input of the residual block and then input into the second convolutional layer for fusion to obtain the output of the residual dual attention fusion module.
5. The image defogging method according to claim 1, characterized in that: The defogging model also includes a feature extraction convolution layer, a channel attention module, a pixel attention module and an output convolution layer. The foggy image to be processed enters the residual group after the features are extracted by the feature extraction convolution layer. After being processed by the residual group, it enters the channel attention module, the pixel attention module and the output convolution layer in turn for processing to obtain output features. The output features are added element by element to the foggy image to be processed to obtain a fog-free image.
6. The image defogging method according to claim 1, characterized in that: The training of the defogging model includes: Obtain the RESIDE dataset, and randomly select 6000 pairs of foggy and non-fog images from the RESIDE dataset to construct a training sample set; The pre-established neural network is trained using the training sample set.
7. The image defogging method according to claim 6, characterized in that: The loss function L of the neural network is expressed as: In the formula, is the number of training samples, For the The real clear images of training samples, For the The haze-free images estimated by the neural network are obtained by training samples.
8. An image defogging device, comprising: An image acquisition module, used for acquiring a foggy image to be processed; An image defogging module is used to input the image to be defogged into a defogging model for processing, and output a fog-free image corresponding to the foggy image to be processed; The defogging model includes a plurality of residual groups, the residual group includes a plurality of residual dual attention fusion modules connected in series, the residual dual attention fusion module includes a residual block, a first convolution layer, a channel attention module, a pixel attention module and a second convolution layer, the output of the residual block is respectively connected to the input of the channel attention module and the pixel attention module via the first convolution layer, the output of the residual block is respectively connected to the input of the channel attention module and the pixel attention module via the first convolution layer, the output of the channel attention module and the pixel attention module are fused, so as to realize that the output of the residual dual attention fusion module enhances the global dependency in each channel map while obtaining pixel features; The channel attention module is input by the input feature into the global average pooling, convolution layer, ReLU activation function, convolution layer, Sigmoid activation function, and the output weight is multiplied element by element with the input feature to obtain the output of the channel attention module. The expression is as follows: In the formula, Representatives in Positional Channel Input The pixel value of ; After global average pooling, the dimension of the feature map is becomes ; is the ReLU activation function, is the Sigmoid activation function, Is element-by-element multiplication; channel attention module input To the channel attention module output The mapping function between ; The pixel attention module is composed of the input feature input convolution layer, ReLU activation function, convolution layer, Sigmoid activation function, and the output weight is multiplied element by element with the input feature to get the output of the pixel attention module. Its expression is as follows: In the formula is the output feature weight, and the dimension is becomes , pixel attention module input To the channel attention module output The mapping function between .
9. The image defogging device according to claim 8, characterized in that: The outputs of the channel attention module and the pixel attention module are added element by element to the input of the residual block and then input into the second convolutional layer for fusion to obtain the output of the residual dual attention fusion module.
10. A device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image defogging method according to any one of claims 1 to 7 is implemented.
