Image defogging method based on hybrid attention encoder

Abstract

The application provides an image defogging method based on a hybrid attention encoder, adopts a hybrid attention encoder to perform feature extraction on an input fog image; then performs matching on the extracted fog image features and a discrete codebook prior; then decodes the matched features; and finally performs feature embedding on the decoder features to obtain a defogging result step by step. The hybrid attention encoder based on the Transformer and the channel attention mechanism is proposed to enhance the expression ability of the network for local information; meanwhile, the discrete codebook prior is introduced to further enhance the texture details of the defogging result, and the binary modulation mechanism BMM is proposed to guide the matching process of the prior.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, deep learning, and specifically relates to an image defogging method based on a hybrid attention encoder. BACKGROUND

[0002] Images taken in foggy environments are prone to color and information distortion. In the image restoration problem, how to uncover the fog in the image and obtain the information hidden behind it is an important task. Through defogging, the clarity and visibility of the image are improved, which has positive practical significance in dealing with fog problems.

[0003] Traditional prior-based defogging methods and deep learning-based defogging methods are two basic methods in the field of image defogging. Prior-based defogging methods estimate the parameters in the physical scattering model for defogging, among which the dark channel defogging algorithm is well-known and has achieved good results in image defogging. Deep learning-based methods either estimate physical parameters through neural networks or directly learn the mapping relationship from foggy images to clear images without estimating physical parameters.

[0004] Deep learning-based defogging methods have gradually taken a leading position in the field of image defogging. Among them, Transformers have performed very well in image defogging. For example, Dehamer introduces prior related to fog density into the Transformer for defogging through a 3D position embedding module with transmission perception. TransRA proposes a dual-branch neural network that combines Transformer and residual attention for single remote sensing image defogging. However, although Transformers excel at handling global information, they still have some inherent problems, the most notable of which is the lack of local fitting capability. Directly applying them to the image defogging task still faces challenges. In image defogging, local fitting capability plays a crucial role in handling non-uniform fog in certain regions. SUMMARY

[0005] Considering that Transformers can provide long-range feature dependencies through cascading self-attention to achieve the image defogging task, but due to their lack of local fitting capability, Transformer-based defogging methods lack the ability to express local details in images.

[0006] To address the above problems, the application proposes a hybrid attention encoder based on Transformer and channel attention mechanism to enhance the network's ability to express local information. At the same time, a discrete codebook prior is introduced to further enhance the texture details of the defogging results, and a binary modulation mechanism (BMM) is proposed to guide the matching process of the prior.

[0007] Among them, a hybrid attention encoder (HAE) based on the standard Swin Transformer structure is proposed, which combines self-attention and channel attention mechanisms, the former can process global information, and the latter has local fitting capability.

[0008] Thanks to the above improvements, the dehazing framework provided by the application can further improve the performance of the Transformer structure in the dehazing task and show high robustness in complex real fog scenes.

[0009] The technical scheme specifically adopted by the application to solve its technical problems is:

[0010] An image dehazing method based on a hybrid attention encoder: a hybrid attention encoder is used to extract features from the input fog image; then the extracted fog image features are matched with the discrete codebook prior; then the matched features are decoded; finally, the decoder features are embedded and gradually decoded to obtain the dehazing result.

[0011] Further, the hybrid attention encoder is based on the Transformer and channel attention mechanism, and the channel attention is introduced on the basis of the swin-transformer structure, and the parallel attention module PAB composed of the channel attention module and the window multi-head self-attention module is used to replace the window multi-head self-attention module.

[0012] Further, after obtaining the image, the hybrid attention encoder is used for multi-scale feature extraction to obtain the final encoder features: for a given image input H, W, C h respectively represent the length, width and channel number of the input; first, a convolutional layer and several residual layers are used for shallow feature extraction to obtain C S represent the channel number of the intermediate feature; then, through the deep feature extraction module composed of four residual hybrid attention group modules RHAG, the encoder feature wherein each residual hybrid attention group module RHAG includes six hybrid attention basic modules HAB, an overlapping cross attention mechanism and a convolutional layer.

[0013] Further, the structure of the hybrid attention basic module HAB is specifically as follows:

[0014] In the Swin Transformer module after the first LayerNorm layer, a parallel attention module PAB composed of a Channel attention module and a W-MSA module is applied, as shown in the following formula:

[0015] f LN =LN(f in )

[0016] f PA =(S)W-MSA(f LN )+γCA(CONV(GELU(CONV(f LN ))))

[0017] wherein f in , f LN represent input features and intermediate features output by the LayerNorm layer respectively; f PA represents output features of the parallel attention module PAB, and γ is a constant used to modulate the output features of the Channel attention module to avoid conflict with the output of the W-MSA module;

[0018] The calculation process of the hybrid attention basic module is as follows:

[0019] f=f PA +f in

[0020] F=MLP(LN(f))+f。

[0021] Further, an overlapping cross-attention mechanism is introduced in each residual hybrid attention group module.

[0022] Further, after obtaining the encoder features, a binary matching mechanism is used to match the features with a discrete codebook prior, and by adjusting the parameters of the modulation curve with quadratic characteristics, the matching mechanism is given greater variability in different defogging scenarios.

[0023] Further, the matching mechanism is obtained based on the nearest neighbor matching expansion in VQGAN; for the nearest neighbor matching mechanism, the calculation formula is as follows:

[0024]

[0025] wherein, is a pixel in the latent feature , is the discrete representation obtained after the nearest code matching with the codebook z k ;

[0026] A certain number of high-quality, clean images are used as input to the pre-trained VQGAN, and the activation frequency f of each code is calculated. c Then, using the same number of real fog maps as input, the activation frequency f for each code is calculated. h .

[0027] Furthermore, the binary matching mechanism is a matching mechanism that uses a quadratic function curve for modulation, by adjusting... Let the curve be in The arc transformations within the range of values ​​are as follows:

[0028]

[0029] Furthermore, after feature extraction and binary matching, the encoder features f are obtained. H and the corresponding feature F' after matching with the codebook H After that, f H 、F' H Input decoder D and VQGAN decoder D respectively c In the process, the corresponding decoder features F1 and F2 are obtained respectively; the decoder feature F2 is embedded into F1 using the feature embedding module FEM with deformable convolution to restore the clear result without fog.

[0030] Since the methods and models of this invention are generally implemented in the form of computer programs through computer systems, this invention also provides an image dehazing system based on a hybrid attention encoder: including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the method described above.

[0031] Compared with the prior art, the present invention and its preferred embodiments are mainly reflected in the following two aspects:

[0032] Considering the Transformer's lack of local fitting ability, directly applying it to image dehazing tasks will still result in coarse details in image reconstruction.

[0033] This invention introduces a Hybrid Attention Encoder (HAE), which integrates channel attention-based convolutional blocks into the Swin-Transformer architecture. It enhances the local features at each location through channel attention computation, while leveraging the global information processing advantages of the Transformer to enhance the network's representational capabilities.

[0034] Considering that the existing discrete codebook matching uses the nearest neighbor matching method, there is still room for improvement in matching with the prior discrete codebook.

[0035] The application adjusts the parameters of the modulation curve with quadratic characteristics on the basis of the original nearest neighbor matching method, gives the matching mechanism greater variability under different defogging scenes, and enables it to better adapt to the codebook prior.

[0036] wherein:

[0037] 1. A series of residual layers are used in the hybrid attention encoder for shallow feature extraction, and a hybrid attention module is used for deep feature extraction. A parallel attention block composed of a channel attention module and a window multi-head self-attention module is introduced on the basis of the swin-transformer infrastructure, replacing the original window multi-head self-attention module. By introducing a parallel channel attention module, the network can enhance the local features of each position through channel direction attention calculation. This improves the preservation of detailed features in images, such as texture, edges, and colors. Therefore, the model exhibits stronger robustness when dealing with different scene changes, such as lighting and color changes. This means that the model can more effectively handle various uncertainties, reducing performance fluctuations under different scenes. In the defogging task, this translates to consistent defogging performance under different types of fog conditions, reducing the situation where the effect drops significantly.

[0038] 2. A binary matching mechanism is used to adjust the parameters of the modulation curve with quadratic characteristics, giving the matching mechanism greater variability under different defogging scenes, enabling it to better adapt to the codebook prior, and achieving flexible matching of encoder features and discrete codebook prior. BRIEF DESCRIPTION OF DRAWINGS

[0039] The application will be further described in detail below in combination with the drawings and specific embodiments:

[0040] Figure 1 The method flowchart of the embodiment of the application is shown in the figure:

[0041] Figure 2 The schematic diagram of the multi-scale encoder of the embodiment of the application is shown in the figure:

[0042] Figure 3 The structure diagram of the hybrid attention module of the embodiment of the application is shown in the figure:

[0043] Figure 4 The schematic diagram of the overlapping cross-attention mechanism of the embodiment of the application is shown in the figure:

[0044] Figure 5 The structure diagram of the feature embedding module (FEM) of the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0045] To make the features and advantages of the patent more obvious and easy to understand, the following specific examples are described in detail as follows:

[0046] It should be noted that the following detailed description is illustrative only and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0047] It should be noted that the terms used herein are only intended to describe specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a reference to the presence of a feature, step, operation, device, component and / or combinations thereof.

[0048] Since the current Transformer-based neural network dehazing method lacks the expression of image details, the application provides an image dehazing method based on a hybrid attention encoder. Channel attention is introduced on the basis of the swin-transformer infrastructure, which can combine the complementary advantages of their global information processing capabilities and local fitting capabilities. The core design is to replace the original window multi-head self-attention module with a parallel attention block (PAB) composed of a channel attention module and a window multi-head self-attention module. By introducing a parallel channel attention module, the network can enhance the local features of each position through channel direction attention calculation. This improves the preservation of detailed features in the image, such as texture, edges, and color. Therefore, the model shows stronger robustness when dealing with different scene changes (such as light and color changes). This means that the model can more effectively handle various uncertainties and reduce performance fluctuations under different scenes. In the dehazing task, this translates to consistent dehazing performance under different types of fog conditions, reducing the situation where the effect drops significantly. In addition, the application also introduces an overlapping block spatial attention mechanism to promote better interaction between adjacent windows.

[0049] The encoder provided by the application effectively solves the inherent limitations of Transformer in local detail representation, improving the dehazing performance of the network. At the same time, a discrete codebook prior is introduced to further enhance the details and textures of the dehazing results. By further proposing a novel binary matching mechanism, the parameters of the modulation curve with quadratic characteristics are adjusted, giving the matching mechanism greater variability under different dehazing scenarios, allowing it to better adapt to the codebook prior. The method of the application has shown excellent performance in the processing of complex and diverse real fog pictures.

[0050] As shown in Figure 1 The main process of the method of the application includes:

[0051] 1) A hybrid attention encoder is used to extract features from the input fog map;

[0052] 2) Match the extracted fog map features with the discrete codebook priors;

[0053] 3) Decode the matched features;

[0054] 4) Embed the features of the decoder and decode step by step to obtain a fog-free result.

[0055] The specific implementation process is as follows:

[0056] After acquiring the image, this invention first uses a hybrid attention encoder to perform multi-scale feature extraction to obtain the final encoder features, such as... Figure 2 As shown: For a given image input H, W, C h These represent the length, width, and number of channels of the input, respectively. First, a shallow feature extraction is performed using a convolutional layer and several residual layers to obtain... C S This represents the number of channels in the intermediate features. Then, the encoder features are obtained through a deep feature extraction module consisting of four residual hybrid attention group modules (RHAG). Each residual hybrid attention group module (RHAG) contains six hybrid attention basic modules (HAB), an overlapping cross-attention mechanism, and a convolutional layer. The structure of the hybrid attention basic module (HAB) is as follows: Figure 3 As shown.

[0057] In the Swin Transformer module following the first LayerNorm layer, it applies a parallel attention module PAB composed of the Channalattention module and the W-MSA module, as shown in the following equation:

[0058] f LN =LN(f in )

[0059] f PA =(S)W-MSA(f LN )+γCA(CONV(GELU(CONV(f LN ))))

[0060] Among them, f in f LN These represent the input features and the intermediate features output by the LayerNorm layer, respectively. PAThe output feature of the parallel attention module PAB is represented, and gamma is a small constant used to modulate the output feature of the Channel attention module to avoid conflict with the output of the W-MSA module. The calculation process of the whole mixed attention basic module is as follows:

[0061] f=f PA +f in

[0062] F=MLP(LN(f))+f

[0063] The above formula expresses two residual connection structures in the basic module, which can effectively avoid the problem of "gradient explosion" and improve the training efficiency of the model. In addition, in order to directly establish a cross-window connection and enhance the expression of window self-attention, the present application introduces an overlapping cross-attention mechanism in each residual mixed attention group module, as shown in Figure 4 .

[0064] After obtaining the encoder feature, a flexible binary matching mechanism is adopted to match it with the discrete codebook prior. Users can modify the parameters in it to adapt to rich haze scenes and explore the best haze removal performance. This matching mechanism is based on the extension of the nearest neighbor matching in VQGAN. For the nearest neighbor matching mechanism, the calculation formula is as follows:

[0065]

[0066] Wherein, is a pixel in the latent feature is the is the closest code matching with the codebook z k .

[0067] A certain number of high-quality clean images are input into the pre-trained VQGAN to calculate the activation frequency f c of each code. Then the same number of real fog images are input to calculate the activation frequency f h of each code. It can be observed that the activation frequencies of each code under the two input conditions are significantly different. is the difference between f c and f h . When the value of is large, the effect of the code will be weakened under the action of the modulation curve, and vice versa. When the value of is 0, it means that no modulation is needed, and the value of the curve is always 1. By adding the modulation curve, the potential of the codebook prior in the haze removal task is stimulated to a certain extent.

[0068] The binary matching mechanism provided in this embodiment is a matching mechanism that uses a quadratic function curve to modulate the matching mechanism, and can be flexibly adjusted. This curve can be made in The arc can be flexibly varied within its range of values. It can be a convex arc, a concave arc, or even a straight line. For example:

[0069]

[0070] Thus, with the help of HAE and BMM provided by this invention, encoder features f have been obtained. H and the corresponding feature F' after matching with the codebook H After that, f H 、F' H Input decoder D and VQGAN decoder D respectively c In the process, the corresponding decoder features F1 and F2 were obtained. The next step is to embed F2 into F1 to help recover a clear, fog-free result. However, this invention found that simply concatenating F1 and F2 is not feasible. Therefore, this invention employs a feature embedding module (FEM) with deformable convolution to better embed F2 into F1, the structure of which is as follows: Figure 5 As shown.

[0071] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0073] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow Figure 1 The functions of a flow or multiple flows and / or a block or multiple blocks in the flow Figure 1 The functions of a flow or multiple flows and / or a block or multiple blocks in the flow

[0074] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the flow Figure 1 The functions of a flow or multiple flows and / or a block or multiple blocks in the flow Figure 1 The functions of a flow or multiple flows and / or a block or multiple blocks in the flow

[0075] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the present application shall be understood as having the common meaning to those skilled in the art to which the present application pertains. The terms "first", "second" and similar terms used in the present application do not denote any order, quantity or importance, but are used to distinguish different components. The terms "include" or "contain" and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "upper", "lower", "left", "right" and the like are used only to indicate relative positional relationships, and when the absolute positions of the described objects are changed, the relative positional relationships can also be changed accordingly.

[0076] The above description is only the preferred embodiment of the present application, and is not intended to limit the other forms of the present application. Any skilled person in the art can modify or change the above-mentioned disclosed technical content into equivalent embodiments with equivalent changes. However, any simple modification, equivalent change and modification of the above-mentioned embodiments made without departing from the technical solution of the present application, and in accordance with the technical essence of the present application, shall still fall within the protection scope of the present application.

[0077] The present patent is not limited to the above-mentioned preferred embodiments, and anyone can derive other various forms of an image defogging method based on a mixed attention encoder from the present patent. Any equivalent changes and modifications made in accordance with the scope of the present patent application shall fall within the scope of the present patent.

Claims

1. An image defogging method based on a hybrid attention encoder, characterized in that: The input fog image is extracted by a hybrid attention encoder; then the extracted fog image features are matched with a discrete codebook prior; then the matched features are decoded; Finally, the decoder features are embedded and gradually decoded to obtain the haze-free result. The hybrid attention encoder is based on Transformer and channel attention mechanism, and introduces channel attention into the basic structure of swin-transformer, and uses parallel attention module PAB composed of channel attention module and window multi-head self-attention module to replace window multi-head self-attention module. After the image is acquired, multi-scale feature extraction is performed using the mixed attention encoder to obtain final encoder features: for a given image input H, W, C h respectively represent the length, width, and channel number of the input; first, a convolutional layer and several residual layers are used for shallow feature extraction to obtain C S represent the channel number of the intermediate features; then, the deep feature extraction module composed of four residual mixed attention group modules RHAG is passed to obtain the encoder features After obtaining the encoder features, a binary matching mechanism is used to match the features with the discrete codebook prior, and by adjusting the parameters of the modulation curve with quadratic characteristics, the matching mechanism is given greater variability in different dehazing scenarios.

2. The image dehazing method based on hybrid attention encoder according to claim 1, characterized in that: Each residual hybrid attention group module RHAG contains six hybrid attention basic modules HAB, an overlapping cross attention mechanism and a convolution layer; The structure of the hybrid attention basic module HAB is specifically as follows: In the Swin Transformer module after the first LayerNorm layer, the parallel attention module PAB composed of the Channel attention module and the W-MSA module is applied, as shown in the following formula: f LN = LN(f in ) f PA = W - MSA(f LN )+ γCA(CONV(GELU(CONV(f LN )))) wherein f in , f LN respectively represent the input feature and the intermediate feature output by the LayerNorm layer; f PA represents the output feature of the parallel attention module PAB, and γ is a constant used to modulate the output feature of the Channel attention module to avoid conflict with the output of the W-MSA module. The calculation process of the hybrid attention basic module is as follows: f = f PA + f in F=MLP(LN(f))+f.

3. The image defogging method based on the hybrid attention encoder of claim 2, characterized in that: An overlapping cross attention mechanism is introduced in each residual hybrid attention group module.

4. The image dehazing method based on hybrid attention encoder according to claim 1, characterized in that: The matching mechanism is obtained by extending the nearest neighbor matching in VQGAN; for the nearest neighbor matching mechanism, the calculation formula is as follows: wherein, is a latent feature of one pixel in the image, is a discrete representation resulting from the last code match with the codebook z k ; Take a number of high-quality clean images as input of pre-trained VQGAN, calculate the activation frequency f of each code c ; Then take the same number of real fog images as input, calculate the activation frequency f of each code h .

5. The image dehazing method based on hybrid attention encoder according to claim 4, characterized in that: The binary matching mechanism is a matching mechanism modulated by a quadratic function curve, which is transformed by adjusting the value range of the curve to the arc line within the value range of , specifically as follows:

6. The image defogging method based on a hybrid attention encoder according to claim 1, characterized in that: After feature extraction and binary matching, the encoder features f are obtained. H and the corresponding feature F after matching with the codebook ′ H After that, f H F ′ H Input decoder D and VQGAN decoder D respectively c In the process, the corresponding decoder features F1 and F2 are obtained respectively; the decoder feature F2 is embedded into F1 using the feature embedding module FEM with deformable convolution to restore the clear result without fog.

7. An image defogging system based on a hybrid attention encoder, characterized in that: A computer program product, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any one of claims 1-6 when executing the program.

Images