Image defogging method based on mixed attention encoder

Abstract

The invention provides an image defogging method based on a hybrid attention encoder. The image defogging method comprises the following steps: performing feature extraction on an input fog image by adopting the hybrid attention encoder; then matching the extracted fog image features with discrete codebook prior; then, the matched features are decoded; and finally, carrying out feature embedding on the decoder features, and gradually decoding to obtain a fog-free result. According to the method, a hybrid attention encoder based on Transform and a channel attention mechanism is proposed to enhance the expression ability of a network for local information; meanwhile, discrete codebook priori is introduced to further enhance texture details of a defogging result, and a binary modulation mechanism BMM is proposed to guide a priori matching process.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision, deep learning, etc., and specifically relates to an image defogging method based on a hybrid attention encoder. Background Art

[0002] Images taken in a haze environment are prone to color and information distortion. In the image restoration problem, how to uncover the fog in the image and obtain the information hidden behind it is an important task. Through defogging, the clarity and visibility of the image are improved, which has positive practical significance in dealing with the haze problem.

[0003] Traditional defogging methods based on priors and defogging methods based on deep learning are the two basic methods in the field of image defogging. Defogging methods based on priors estimate parameters in the physical scattering model to perform defogging. Among them, the more famous one is the dark channel defogging algorithm, which has achieved good results in image defogging. Some methods based on deep learning use neural networks to estimate physical parameters to perform defogging, while others do not estimate physical parameters and directly learn the mapping relationship from foggy images to clear images to perform defogging.

[0004] Dehazing methods based on deep learning have gradually taken a leading position in the field of image dehazing. Among them, Transformer performs very well in image dehazing. For example, Dehamer introduces the prior related to haze density into Transformer for dehazing by transferring the perceptual 3D position embedding module. TransRA proposes a dual-branch neural network that integrates Transformer and residual attention for single remote sensing image dehazing. However, despite the excellent performance of Transformers in processing global information, they still have some inherent problems, the most notable of which is the lack of local fitting ability. There are still challenges when directly applying it to image dehazing tasks. In image dehazing, local fitting ability plays a vital role in dealing with non-uniform haze in certain areas. Summary of the invention

[0005] Considering that Transformer can provide long-distance feature dependencies through cascaded self-attention to achieve image dehazing tasks, however, due to its lack of local fitting ability, the Transformer-based dehazing method is insufficient in expressing local details of the image.

[0006] To address the above problems, this paper proposes a hybrid attention encoder based on Transformer and channel attention mechanism to enhance the network's ability to express local information. At the same time, a discrete codebook prior is introduced to further enhance the texture details of the dehazing result, and a binary modulation mechanism BMM is proposed to guide the prior matching process.

[0007] Among them, a hybrid attention encoder (HAE) based on the standard Swin Transformer structure is proposed, which combines self-attention and channel attention mechanisms. The former can process global information, while the latter has local fitting capabilities. The overlapping block spatial attention mechanism is introduced to enhance the interaction between adjacent window features.

[0008] Thanks to the above improvements, the defogging framework provided by the present invention can further improve the performance of the Transformer structure in defogging tasks and show a high degree of robustness in complex real-life haze scenes.

[0009] The technical solution specifically adopted by the present invention to solve the technical problem is:

[0010] An image defogging method based on a hybrid attention encoder: a hybrid attention encoder is used to extract features from an input fog image; the extracted fog image features are then matched with a discrete codebook prior; after that, the matched features are decoded; finally, the decoder features are feature embedded and gradually decoded to obtain a fog-free result.

[0011] Furthermore, the hybrid attention encoder is based on Transformer and channel attention mechanism, introduces channel attention on the basic structure of swin-transformer, and adopts a parallel attention module PAB composed of a channel attention module and a window multi-head self-attention module to replace the window multi-head self-attention module.

[0012] Furthermore, after acquiring the image, the hybrid attention encoder is used to perform multi-scale feature extraction to obtain the final encoder features: for a given image input H, W, C h Respectively represent the length, width, and number of channels of the input; first, a convolutional layer and several residual layers are used to extract shallow features, and we get C S represents the number of channels of the intermediate features; then, the encoder features are obtained through a deep feature extraction module consisting of four residual hybrid attention group modules RHAG Among them, each residual hybrid attention group module RHAG contains six hybrid attention basic modules HAB, an overlapping cross attention mechanism, and a convolutional layer.

[0013] Furthermore, the structure of the hybrid attention basic module HAB is specifically as follows:

[0014] In the Swin Transformer module after the first LayerNorm layer, a parallel attention module PAB consisting of a Channel attention module and a W-MSA module is applied, as shown in the following formula:

[0015] f LN =LN(f in )

[0016] f PA =(S)W-MSA(f LN )+γCA(CONV(GELU(CONV(f LN ))))

[0017] Among them, f in 、f LN They represent the input features and the intermediate features output by the LayerNorm layer respectively; f PA represents the output features of the parallel attention module PAB, γ is a constant used to modulate the output features of the Channel attention module to avoid conflict with the output of the W-MSA module;

[0018] The calculation process of the hybrid attention basic module is:

[0019] f=f PA +f in

[0020] F=MLP(LN(f))+f.

[0021] Furthermore, an overlapping cross attention mechanism is introduced in each of the residual mixed attention group modules.

[0022] Furthermore, after obtaining the encoder features, a binary matching mechanism is used to match the features with the discrete codebook prior, and the matching mechanism is given greater variability in different defogging scenarios by adjusting the parameters of the modulation curve with quadratic characteristics.

[0023] Furthermore, the matching mechanism is obtained based on the extension of the nearest neighbor matching in VQGAN; for the nearest neighbor matching mechanism, its calculation formula is as follows:

[0024]

[0025] in, It is a potential feature A pixel in yes With codebook z k The discretized representation obtained after matching the most recent code;

[0026] Take a certain number of high-quality clean images as the input of the pre-trained VQGAN and calculate the activation frequency f of each code c ; Then take the same number of real fog images as input and calculate the activation frequency f of each code h .

[0027] Furthermore, the binary matching mechanism is a matching mechanism that uses a quadratic function curve for modulation. Let the curve be The arc transformation within the value range of is as follows:

[0028]

[0029] Furthermore, after feature extraction and binary matching, the encoder feature f is obtained H And the corresponding feature F' after matching with the codebook H ; Afterwards, f H , F' H Input decoder D and VQGAN decoder D respectively c In the above example, we get the corresponding decoder features F 1 and F 2 ; The decoder feature F is embedded using a feature embedding module FEM with deformable convolution 2 Embed F 1 , to restore a haze-free, clear result.

[0030] Since the method and model of the present invention are generally implemented in the form of a computer program through a computer system, the present invention also provides an image defogging system based on a hybrid attention encoder: comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method as described above when executing the program.

[0031] Compared with the prior art, the present invention and its preferred solution are mainly reflected in the following two aspects:

[0032] Considering the lack of local fitting ability of Transformer, directly applying it to the image dehazing task will still result in rough details in image reconstruction.

[0033] This paper introduces a hybrid attention encoder HAE, integrates the channel attention-based convolution block into the Swin-Transformer architecture, enhances the local features of each position through channel attention calculation, and uses the global information processing advantages of Transformer to enhance the representation ability of the network.

[0034] Considering that the existing discrete codebook matching adopts the nearest neighbor matching method, there is still room for improvement in matching with the discrete codebook prior.

[0035] Based on the original nearest neighbor matching method, the present invention adjusts the parameters of the modulation curve with quadratic characteristics to give the matching mechanism greater variability in different defogging scenarios, so that it can better adapt to the codebook prior.

[0036] in:

[0037] 1. A series of residual layers are used in the hybrid attention encoder for shallow feature extraction, and a hybrid attention module is used for deep feature extraction. A parallel attention block consisting of a channel attention module and a window multi-head self-attention module is introduced on top of the basic structure of the swin-transformer, replacing the original window multi-head self-attention module. By introducing the parallel channel attention module, the network can enhance the local features of each position through channel-wise attention calculations. This improves the retention of detailed features in the image, such as texture, edges, and colors. As a result, the model shows stronger robustness in dealing with different scene changes, such as lighting and color changes. This means that the model can handle various uncertainties more effectively and reduce performance fluctuations in different scenes. In the dehazing task, this translates into maintaining consistent dehazing performance under different types of haze conditions and reducing situations where the effect drops significantly.

[0038] 2. A binary matching mechanism is adopted to adjust the parameters of the modulation curve with quadratic characteristics, giving the matching mechanism greater variability in different dehazing scenarios, so that it can better adapt to the codebook prior and achieve flexible matching between encoder features and discrete codebook priors. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0040] Figure 1 This is a flow chart of a method according to an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a multi-scale encoder according to an embodiment of the present invention;

[0042] Figure 3 This is a structural diagram of a hybrid attention module according to an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of an overlapping cross attention mechanism according to an embodiment of the present invention;

[0044] Figure 5 This is a FEM structure diagram of the feature embedding module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the features and advantages of this patent more obvious and easy to understand, the following embodiments are specifically described in detail as follows:

[0046] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0047] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0048] Since the current Transformer-based neural network defogging method lacks detailed expression of the image, the present invention specifically provides an image defogging method based on a hybrid attention encoder, which introduces channel attention on the basic structure of swin-transformer, and can combine their complementary advantages of processing global information and local fitting capabilities. Its core design is to adopt a parallel attention block (PAB) composed of a channel attention module and a window multi-head self-attention module to replace the original window multi-head self-attention module. By introducing a parallel channel attention module, the network can enhance the local features of each position through channel-direction attention calculations. This improves the retention of detailed features in the image, such as texture, edges, and colors. Therefore, the model shows stronger robustness in dealing with different scene changes (such as lighting and color changes). This means that the model can handle various uncertainties more effectively and reduce performance fluctuations in different scenes. In the defogging task, this translates into maintaining consistent defogging performance under different types of haze conditions and reducing the situation where the effect is greatly reduced. In addition, the present invention also introduces an overlapping block spatial attention mechanism to promote better interaction between adjacent windows.

[0049] The encoder provided by the present invention effectively solves the inherent limitations of Transformer in the representation of local details and improves the defogging performance of the network. At the same time, a discrete codebook prior is introduced to further enhance the detail texture of the defogging result. By further proposing a novel binary matching mechanism, by adjusting the parameters of the modulation curve with quadratic characteristics, the matching mechanism is given greater variability in different defogging scenarios, so that it can better adapt to the codebook prior. The method of the present invention has demonstrated excellent performance in the processing of complex and diverse real haze images.

[0050] like Figure 1 As shown, the main process of the method of the present invention includes:

[0051] 1) Use a hybrid attention encoder to extract features from the input fog image;

[0052] 2) Match the extracted fog image features with the discrete codebook prior;

[0053] 3) Decode the matched features;

[0054] 4) Feature embedding is performed on the decoder features, and the decoded features are gradually decoded to obtain a haze-free result.

[0055] The specific implementation process is:

[0056] After acquiring the image, the present invention first uses a hybrid attention encoder to perform multi-scale feature extraction to obtain the final encoder features, such as Figure 2 As shown: For a given image input H, W, C h Respectively represent the length, width, and number of channels of the input. First, a convolutional layer and several residual layers are used to extract shallow features, and we get C S Indicates the number of channels of the intermediate features. Then, the encoder features are obtained through a deep feature extraction module consisting of four residual hybrid attention group modules RHAG. Each residual hybrid attention group module RHAG contains six hybrid attention basic modules HAB, an overlapping cross attention mechanism, and a convolutional layer. The structure of the hybrid attention basic module HAB is as follows: Figure 3 shown.

[0057] It applies the parallel attention module PAB consisting of the Channelattention module and the W-MSA module in the Swin Transformer module after the first LayerNorm layer, as shown in the following formula:

[0058] f LN =LN(f in )

[0059] f PA =(S)W-MSA(f LN )+γCA(CONV(GELU(CONV(f LN ))))

[0060] Among them, f in 、f LN They represent the input features and the intermediate features output by the LayerNorm layer. PArepresents the output feature of the parallel attention module PAB, and γ is a small constant used to modulate the output feature of the Channel attention module to avoid conflict with the output of the W-MSA module. The calculation process of the entire hybrid attention basic module is:

[0061] f=f PA +f in

[0062] F=MLP(LN(f))+f

[0063] The above formula expresses the two residual connection structures in the basic module, which can effectively avoid the "gradient explosion" problem and improve the training efficiency of the model. In addition, in order to directly establish cross-window connections and enhance the expression of window self-attention, the present invention introduces an overlapping cross-attention mechanism in each residual mixed attention group module, such as Figure 4 shown.

[0064] After obtaining the encoder features, a flexible binary matching mechanism is used to match them with the discrete codebook prior. Users can modify the parameters to adapt to a variety of haze scenes and explore the best defogging performance. This matching mechanism is based on the nearest neighbor matching extension in VQGAN. For the nearest neighbor matching mechanism, the calculation formula is as follows:

[0065]

[0066] in, It is a potential feature A pixel in yes With codebook z k The discretized representation obtained after matching the nearest code.

[0067] Take a certain number of high-quality clean images as the input of the pre-trained VQGAN and calculate the activation frequency f of each code c Then take the same number of real fog images as input and calculate the activation frequency f of each code h From this, we can observe that the activation frequencies of each code under the two input conditions are significantly different. Yes c and f h When When the value is large, the effect of the code will be weakened under the modulation curve, and vice versa. When the value of is 0, it means that no modulation is required, and the value of the curve is always 1. By adding the modulation curve, the potential of codebook prior in the dehazing task is stimulated to a certain extent.

[0068] The binary matching mechanism provided in this embodiment is a matching mechanism that uses a quadratic function curve to modulate it. This curve can be made The arc within the value range of can be flexibly transformed. It can be a convex arc, a concave arc, or even a straight line. As follows:

[0069]

[0070] So far, with the help of HAE and BMM provided by the present invention, the encoder feature f is obtained H And the corresponding feature F' after matching with the codebook H After that, f H , F' H Input decoder D and VQGAN decoder D respectively c In the above example, we obtain the corresponding decoder features F 1 and F 2 Next, F 2 Embed F 1 , helping it to restore a clear result without fog. However, the present invention has found that 1 and F 2 Simple concat does not seem to be feasible. Therefore, the present invention adopts a feature embedding module FEM with deformable convolution to transform F 2 Better embedding into F 1 Among them, its structure is as follows Figure 5 shown.

[0071] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0072] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0073] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0075] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0076] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

[0077] This patent is not limited to the above-mentioned optimal implementation mode. Anyone can derive various other forms of an image defogging method based on a hybrid attention encoder under the inspiration of this patent. All equal changes and modifications made according to the scope of the patent application of the present invention should be covered by this patent.

Claims

1. An image defogging method based on a hybrid attention encoder, characterized in that: A hybrid attention encoder is used to extract features from the input fog image. The extracted fog image features are then matched with the discrete codebook prior. After that, the matched features are decoded. Finally, the decoder features are feature embedded and gradually decoded to obtain a fog-free result.

2. The image defogging method based on a hybrid attention encoder according to claim 1, characterized in that: The hybrid attention encoder is based on Transformer and channel attention mechanism, introduces channel attention on the basic structure of swin-transformer, and adopts a parallel attention module PAB composed of a channel attention module and a window multi-head self-attention module to replace the window multi-head self-attention module.

3. The image defogging method based on hybrid attention encoder according to claim 1, characterized in that: After acquiring the image, the hybrid attention encoder is used to perform multi-scale feature extraction to obtain the final encoder features: For a given image input H, W, C h Respectively represent the length, width, and number of channels of the input; first, a convolutional layer and several residual layers are used to extract shallow features, and we get C S represents the number of channels of the intermediate features; then, the encoder features are obtained through a deep feature extraction module consisting of four residual hybrid attention group modules RHAG Among them, each residual hybrid attention group module RHAG contains six hybrid attention basic modules HAB, an overlapping cross attention mechanism, and a convolutional layer.

4. The image defogging method based on hybrid attention encoder according to claim 3, characterized in that: The structure of the hybrid attention basic module HAB is as follows: In the Swin Transformer module after the first LayerNorm layer, a parallel attention module PAB consisting of a Channel attention module and a W-MSA module is applied, as shown in the following formula: f LN =LN(f in ) f PA =(S)W-MSA(f LN )+γCA(CONV(GELU(CONV(f LN )))) Among them, f in 、f LN They represent the input features and the intermediate features output by the LayerNorm layer respectively; f PA represents the output features of the parallel attention module PAB, γ is a constant used to modulate the output features of the Channel attention module to avoid conflict with the output of the W-MSA module; The calculation process of the hybrid attention basic module is: f=f PA +f in F = MLP (LN (f)) + f.

5. The image defogging method based on hybrid attention encoder according to claim 4, characterized in that: An overlapping cross attention mechanism is introduced in each of the residual mixed attention group modules.

6. The image defogging method based on hybrid attention encoder according to claim 1, characterized in that: After obtaining the encoder features, a binary matching mechanism is used to match the features with the discrete codebook prior. By adjusting the parameters of the modulation curve with quadratic characteristics, the matching mechanism is given greater variability in different dehazing scenarios.

7. The image defogging method based on hybrid attention encoder according to claim 6, characterized in that: The matching mechanism is obtained based on the nearest neighbor matching extension in VQGAN; for the nearest neighbor matching mechanism, its calculation formula is as follows: in, It is a potential feature A pixel in yes With codebook z k The discretized representation obtained after matching the most recent code; Take a certain number of high-quality clean images as the input of the pre-trained VQGAN and calculate the activation frequency f of each code c ; Then take the same number of real fog images as input and calculate the activation frequency f of each code h .

8. The image defogging method based on hybrid attention encoder according to claim 7, characterized in that: The binary matching mechanism is a matching mechanism that uses a quadratic function curve for modulation. Let the curve be The arc transformation within the value range of is as follows:

9. The image defogging method based on hybrid attention encoder according to claim 1, characterized in that: After feature extraction and binary matching, the encoder features f are obtained H And the corresponding feature F' after matching with the codebook H ; Afterwards, f H 、F' H Input decoder D and VQGAN decoder D respectively c In the above figure, the corresponding decoder features F1 and F2 are obtained respectively; the decoder feature F2 is embedded into F1 using the feature embedding module FEM with deformable convolution to restore the clear result without fog.

10. An image defogging system based on a hybrid attention encoder, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 9 when executing the program.

Images