Image defogging method and device based on multiple attention

Abstract

The application provides a kind of image defogging method and device based on multiple attention, the method comprises the following steps: S1, obtains original fog image;S2, extract fog features from fog image;S3, encode the fog features to obtain the encoded features;S4, the encoded features are decoded to obtain the decoded features;S5, the fog residual is obtained by decoding the fog residual of the decoded features;S6, the clear fog-free image is obtained by the residual of the original fog image and the fog residual.It can solve the problem of resolution loss, inaccurate fog feature representation and uneven defogging faced by the existing method in the defogging process of optical sensor in multi-scene shooting.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to an image dehazing method and apparatus based on multiple attention. Background Technology

[0002] In recent years, the rapid development of deep learning technology and its widespread application in computer vision have made computer vision-related applications a research hotspot. Examples include autonomous driving, border security, and urban surveillance. In these open environments, applications are frequently affected by adverse weather conditions such as fog, haze, and rain, severely impacting the imaging quality of optical sensors and consequently affecting the effectiveness of these applications. Therefore, removing fog from images and obtaining clear images is of paramount importance.

[0003] To address the problem of image dehazing, researchers have proposed image enhancement-based methods, image restoration-based algorithms, and deep learning-based methods. Image enhancement-based methods struggle to adapt flexibly to contrast variations in images and are prone to distortion of image features such as color and contours. Image restoration-based algorithms often fail to respond effectively to large areas of sky and white space and can produce halos. Deep learning-based methods effectively solve these problems and have therefore attracted widespread attention.

[0004] Traditional deep learning algorithms, which use simple stacking of convolutional layers, struggle to accurately remove fog from images with flexibly varying scenes. Specifically, they have the following shortcomings:

[0005] First, the original resolution is lost during the convolution stacking process;

[0006] Secondly, the fog feature extraction network is simple and difficult to accurately represent fog features;

[0007] Third, the correlation between different regions of the image itself was not considered, resulting in uneven dehazing in different regions. Summary of the Invention

[0008] The present invention aims to provide an image dehazing method and apparatus based on multiple attention, so as to solve the problems of resolution loss, inaccurate fog feature representation and uneven dehazing faced by existing methods in the process of dehazing fog images captured by optical sensors in multiple scenes.

[0009] This invention provides an image dehazing method based on multiple attention, comprising the following steps:

[0010] S1, Obtain the original fogged image;

[0011] S2, extract haze features from the hazy image;

[0012] S3, Encode the haze features to obtain the encoded features;

[0013] S4, decode the encoded features to obtain the decoded features;

[0014] S5, perform haze residue decoding on the decoded features to obtain haze residue;

[0015] S6, the original foggy image is compared with the residual fog to obtain a clear, fog-free image.

[0016] Furthermore, in step S1, the original foggy image includes:

[0017] The image to be processed is a foggy image acquired by an image sensor.

[0018] The training dataset, namely the RESIDE dataset used in the training phase, is used, and the original RESIDE dataset is resized to the size of the foggy image to be processed.

[0019] Furthermore, in step S2, the method for extracting haze features from the hazy image includes:

[0020] Haze features are extracted from hazy images using a focus feature extraction network based on smooth dilated convolution.

[0021] Furthermore, step S2 includes the following sub-steps:

[0022] S21, the focus module is used to sample the foggy image by spatial pixels at intervals of 1, resulting in four downsampled image blocks;

[0023] S22, stitch the four downsampled image blocks together along the channel direction to obtain a downsampled feature map without loss of resolution;

[0024] S23, the downsampled feature map is fed into a smooth dilated convolutional network to obtain smooth convolutional features;

[0025] S24, the smooth convolutional features and the downsampled feature map are concatenated along the channel direction to obtain the haze features.

[0026] Furthermore, in step S3, the method for encoding the haze features to obtain the encoded features includes:

[0027] A multi-attention module is used to calculate attention weights at three levels: channel, space, and itself. Haze features are weighted according to their respective attention weights at different levels, thereby obtaining encoded features in the encoding space.

[0028] Furthermore, step S3 includes the following sub-steps:

[0029] S31, the haze features are passed through a channel attention weighting layer to obtain channel attention weighted features;

[0030] S32, the channel attention-weighted features are passed through the spatial attention-weighted layer to obtain the spatial channel attention-weighted features;

[0031] S33, the spatial channel attention-weighted features are divided into multiple image blocks through a self-attention layer;

[0032] S34, multiple image patches are processed through a multi-head attention layer to compute multiple self-attention;

[0033] S35 concatenates multiple self-attention signals and passes them through an MLP residual network to obtain the encoded features corresponding to multiple image patches.

[0034] Furthermore, in step S4, the method for decoding the encoded features to obtain the decoded features includes:

[0035] A self-attention decoder is used to sequentially decode the coded features corresponding to multiple image blocks to obtain the decoded features.

[0036] Furthermore, step S4 includes the following sub-steps:

[0037] S41, the encoded features corresponding to the first image block are decoded by the self-attention decoder to obtain the decoded features corresponding to the first image block;

[0038] S42, the decoded features corresponding to the first image block and the encoded features corresponding to the second image block are decoded together by the self-attention decoder to obtain the decoded features corresponding to the second image block;

[0039] S43, decode the encoded features corresponding to all image blocks in sequence according to the method in step S42, and finally obtain the decoded features corresponding to all image blocks.

[0040] Furthermore, in step S5, the method for obtaining haze residue by performing haze residue decoding on the decoded features includes:

[0041] By using a multi-layer hybrid convolutional network, the decoded features corresponding to all image blocks in the original feature space are mapped to the target haze residue space, thus extracting the haze residue.

[0042] Furthermore, step S5 includes the following sub-steps:

[0043] S51, convert the decoding features corresponding to all image blocks to their original size, and then stitch all the decoding features together according to the original position of each image block to form a decoding feature;

[0044] S52 decodes the assembled features into the original feature space through a dimension adaptation module composed of a deconvolution layer;

[0045] S53 uses a haze residual mapping module composed of convolutional layers to map the feature mapping of the original feature space to the target haze space, thus obtaining haze residual.

[0046] The present invention also provides an image dehazing apparatus based on multiple attention, for implementing the above-mentioned image dehazing method based on multiple attention, comprising:

[0047] The data acquisition module is used to acquire raw, foggy images;

[0048] The feature extraction module is used to extract haze features from hazy images;

[0049] The feature encoding module is used to encode the haze features to obtain encoded features;

[0050] The feature decoding module is used to decode the encoded features to obtain the decoded features;

[0051] The haze persistence decoding module is used to perform haze persistence decoding on the decoding features to obtain haze persistence;

[0052] The residual module is used to obtain a clear, fog-free image by performing a residual operation between the original foggy image and the remaining fog.

[0053] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0054] This invention addresses the resolution loss problem through a focus feature extraction network based on smooth dilated convolution, the inaccurate fog feature representation problem through a multiple attention module, and the uneven defogging problem through decoding based on a self-attention mechanism. Furthermore, it addresses the poor restoration of fog details through dimensional adaptation and spatial mapping. Therefore, by fully utilizing information such as the distribution of fog in the image pixel space and channel space, as well as the inherent correlation of fog, this invention can solve the problems of resolution loss, inaccurate fog feature representation, and uneven defogging encountered by existing methods in defogging images captured by optical sensors in multiple scenes. Attached Figure Description

[0055] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings in the embodiments will be briefly described below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart of an image dehazing method based on multiple attention in an embodiment of the present invention.

[0057] Figure 2This is a schematic diagram of the focus haze feature extraction network based on smooth dilated convolution in an embodiment of the present invention.

[0058] Figure 3a This is a schematic diagram of the channel attention weighting layer in an embodiment of the present invention.

[0059] Figure 3b This is a schematic diagram of the spatial attention weighting layer in an embodiment of the present invention.

[0060] Figure 3c This is a schematic diagram of the self-attention layer in an embodiment of the present invention.

[0061] Figure 4 This is a schematic diagram of a self-attention decoder in an embodiment of the present invention.

[0062] Figure 5 This is a flowchart illustrating how a multi-layer deconvolutional network is used to decode the features to obtain haze residue in an embodiment of the present invention.

[0063] Figure 6 This is a schematic diagram of an image dehazing device based on multiple attention in an embodiment of the present invention. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0065] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0066] Example

[0067] like Figure 1 As shown, this embodiment proposes an image dehazing method based on multiple attention, including the following steps:

[0068] S1, Obtain the original fogged image;

[0069] S2, extract haze features from the hazy image;

[0070] S3, Encode the haze features to obtain the encoded features;

[0071] S4, decode the encoded features to obtain the decoded features;

[0072] S5, perform haze residue decoding on the decoded features to obtain haze residue;

[0073] S6, the original foggy image is compared with the residual fog to obtain a clear, fog-free image.

[0074] The aforementioned image dehazing method based on multiple attention fully utilizes information such as the distribution of haze in the image pixel space and channel space, as well as the correlation of haze itself, to solve the problems of resolution loss, inaccurate haze feature representation, and uneven dehazing faced by existing methods in the process of dehazing haze images captured by optical sensors in multiple scenes. Specifically:

[0075] S1, Obtain the original fogged image;

[0076] For different stages, the original fogged image includes:

[0077] The image to be processed is a foggy image acquired by an image sensor.

[0078] And the training dataset, namely the RESIDE dataset used in the training phase, including the original fog-free images I clear Its corresponding foggy image I t .

[0079] Since the image sizes in the RESIDE dataset may not be consistent with the actual image sizes of the images to be processed, the RESIDE dataset needs to be preprocessed before extracting haze features. The preprocessing methods include:

[0080] The image sizes in the RESIDE dataset are resized to the size of the foggy image to be processed using bilinear encoding; wherein the image size of the foggy image to be processed is obtained from the image sensor.

[0081] S2, extract haze features from the hazy image;

[0082] In this embodiment, a visual feature tensor of the target is extracted from the target image. The extracted visual feature tensor initially reflects the overall distribution characteristics of fog in the image. Therefore, in this embodiment, a focus feature extraction network based on smooth dilated convolution is used to extract fog features from the foggy image. Assume the foggy image is I. t The haze feature is f, and the haze feature extraction operation is as follows: Then we have:

[0083]

[0084] like Figure 2As shown, step S2 specifically includes the following steps:

[0085] S21, the focus module is used to sample the foggy image at spatial pixel intervals of 1, resulting in four downsampled image blocks. This means that every other point in the input signal is sampled, and four such points constitute four downsampled image blocks, represented as follows:

[0086] f i ={I(4n-4+i, 4k-4+i), i=1,2,3,4 n=1,2,...,N / 4 k=1,2,...,K} (2)

[0087] S22, the four downsampled image blocks are stitched together along the channel direction to obtain a downsampled feature map without loss of resolution, represented as:

[0088] f focus =Concatenate(f1,f2,f3,f4) (3)

[0089] S23, the downsampled feature map is fed into a smooth dilated convolutional network, and the feature channels are expanded using the smooth dilated convolutional network to enrich the information of the downsampled feature map and obtain smooth convolutional features;

[0090] S24, concatenate the smoothed convolutional features with the downsampled feature map along the channel direction to obtain the haze feature f. haze1 Therefore, the above describes a focus feature extraction network based on smooth dilated convolution. Introducing a focus network can obtain downsampled feature maps without losing original image information. Furthermore, by introducing smooth dilated convolution into the focus structure, contextual information can be stacked without losing spatial resolution, enabling effective extraction of the original feature tensor of haze features and solving the resolution loss problem during feature extraction.

[0091] S3, Encode the haze features to obtain encoded features; in this embodiment, a multi-attention module is used to calculate attention weights at three levels: channel, space, and itself. The haze features are weighted according to their respective attention weights at different levels, thereby obtaining encoded features in the encoding space. Specifically, the steps are as follows:

[0092] S31, the haze features are passed through a channel attention weighting layer to obtain channel attention weighted features:

[0093] like Figure 3a As shown, first make the haze feature f haze1 The weighted feature f is obtained by passing the data through two 3×3 convolution kernels, with the number of kernels equal to the number of feature channels. w Then, for the weighted feature f wGlobal average pooling (GMP) is performed to obtain pre-weights, which are then fed into a multilayer perceptron (MLP) to obtain the final channel attention feature map W. c , is represented as:

[0094] W c =MLP(GMP(Conv(Conv(f)) haze1 (4)

[0095] Then the haze feature f haze1 With channel attention feature map W c Multiplying by elements, we obtain the channel attention-weighted feature f. c .

[0096] S32, the channel attention-weighted features are passed through a spatial attention-weighted layer to obtain spatial channel attention-weighted features:

[0097] like Figure 3b As shown, the channel attention-weighted features f are first... c The spatial attention feature map W is obtained by feeding in two 3×3 convolutional kernels with a kernel size of 1. s Then, the channel attention-weighted features f c Spatial attention feature map W s Multiplying by elements, we obtain the channel-space attention-weighted feature f. cs .

[0098] S33 divides the spatial channel attention-weighted features into multiple image patches through a self-attention layer:

[0099] like Figure 3c As shown, the self-attention layer first applies channel-space attention-weighted features f cs Divide the image into 16×16 blocks p i,j (i,j=1,...,16), and then project each image patch as a fixed-length vector x. i,j , is represented as:

[0100] x i,j = Norm(MLP(flattened(p i,j ))) (5)

[0101] S34, passing multiple image patches through a multi-head attention layer to compute multiple self-attention mechanisms:

[0102] The vector set X of all image patches has a dimension of 256×L. p L p This represents the total number of pixels in a single image patch. The vector set X is then mapped to the Q, K, V spaces through multiple MLP networks, as follows:

[0103]

[0104] Then, self-attention is calculated for the Q, K, and V vectors:

[0105]

[0106] Where, d k Let V be the vector dimension.

[0107] S35, multiple self-attention Z i After concatenation, the images are processed through an MLP residual network to obtain the encoded features corresponding to multiple image patches:

[0108] Z self =Norm(MLP(concat(Z1,Z2,...,Z n ))+concat(Z1,Z2,...,Z n (8)

[0109] As can be seen, the above-mentioned design of multiple attention modules enables the network to focus on channels and spatial distribution areas with higher importance, while reducing the model's dependence on external information, effectively capturing the internal correlation of features, and solving the problem of inaccurate representation of haze features.

[0110] S4, decode the encoded features to obtain the decoded features; that is, use a self-attention decoder to sequentially decode the encoded features corresponding to multiple image blocks to obtain the decoded features. Specifically, step S4 includes the following sub-steps:

[0111] S41, the encoded features corresponding to the first image block are decoded by the self-attention decoder to obtain the decoded features corresponding to the first image block;

[0112] S42, the decoded features corresponding to the first image block and the encoded features corresponding to the second image block are decoded together by the self-attention decoder to obtain the decoded features corresponding to the second image block;

[0113] S43, decode the encoded features corresponding to all image blocks in sequence according to the method in step S42, and finally obtain the decoded features corresponding to all image blocks.

[0114] like Figure 4As shown, the self-attention decoder consists of multiple feature fast decoding modules and a residual feedforward network layer. Each feature fast decoding module includes a residual masked attention layer and an encoder-decoder attention layer. For the encoded features corresponding to the first image patch, the vector set X is first used as the input to the self-attention decoder. It first passes through the masked attention layer, and the output of the masked attention layer is residually calculated with the vector set X. The output of the masked attention layer is then used as the input to the Q matrix of the residual multi-head attention layer. The other K and V matrices are output Z by the self-attention decoder. self The same method is used to obtain the decoded features corresponding to the first image patch. Then, the decoded features corresponding to the first image patch are used to replace the first column of the vector set X, serving as the input X to the decoder of the second feature patch. mask1 The encoded features corresponding to the second image patch are then decoded. The decoding process for the encoded features of subsequent image patches is repeated in this manner until the decoded features for all image patches are obtained. Finally, the decoded features for all image patches are passed through an MLP layer to obtain the final feature vector f of the decoded features for all image patches. haze2 Its dimensionality and haze characteristics f haze1 Consistent.

[0115] Therefore, the above-designed decoding method based on the self-attention mechanism can make full use of the information learned by self-attention, eliminate redundant information of haze features, realize global decoding of haze features, and solve the problem of uneven defogging.

[0116] S5, perform haze persistence decoding on the decoded features to obtain haze persistence. In this embodiment, a multi-layer hybrid convolutional network maps the decoded features corresponding to all image blocks in the original feature space to the target haze persistence space, thus extracting the haze persistence. Figure 5 As shown, step S5 includes the following sub-steps:

[0117] S51, reshape the decoded features corresponding to all image blocks to their original size, and then concatenate all decoded features according to the original position of each image block to form a decoded feature f. d ;

[0118] S52, the decoded features f d The original feature space is decoded through a dimension adaptation module consisting of a deconvolutional layer.

[0119] S53, through a haze persistence mapping module composed of convolutional layers, maps the feature maps of the original feature space to the target haze space, obtaining the haze persistence.

[0120]

[0121] Therefore, by studying a haze retention recovery method based on hybrid convolution, we can use dimensionality adaptation to map the haze decoding features to the original feature space, and then use the haze retention mapping module to further obtain the haze retention, thus solving the problem of poor haze detail restoration.

[0122] S6, the original foggy image I t With haze residue Obtain a clear, haze-free image by performing residual analysis. dehaze , is represented as:

[0123]

[0124] The loss function used in this example is:

[0125]

[0126] Among them, I haze =I t -I clear This is a residual of the original haze.

[0127] Repeat the above steps until the parameters of each network or module in the entire model converge. Pass the foggy image to be processed through the trained model (network or module) to obtain the final clear, fog-free image.

[0128] In addition, such as Figure 6 As shown, this embodiment is still a multi-attention-based image dehazing device, used to perform the above-described multi-attention-based image dehazing method, specifically including:

[0129] The data acquisition module is used to acquire raw, foggy images;

[0130] The feature extraction module is used to extract haze features from hazy images;

[0131] The feature encoding module is used to encode the haze features to obtain encoded features;

[0132] The feature decoding module is used to decode the encoded features to obtain the decoded features;

[0133] The haze persistence decoding module is used to perform haze persistence decoding on the decoding features to obtain haze persistence.

[0134] The residual module is used to obtain a clear, fog-free image by performing a residual operation between the original foggy image and the remaining fog.

[0135] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image dehazing method based on multiple attention, characterized in that, Includes the following steps: S1, Obtain the original fogged image; S2, extract haze features from the hazy image; S3 employs a multi-attention module to calculate attention weights at three levels: channel, space, and itself. Haze features are weighted according to their respective attention weights at different levels, thereby obtaining the encoded features corresponding to multiple image blocks in the encoding space. S4, a self-attention decoder is used to sequentially decode the encoded features corresponding to multiple image blocks to obtain the decoded features; S5, perform haze residue decoding on the decoded features to obtain haze residue; S6, the original foggy image is compared with the residual fog to obtain a clear, fog-free image; In step S2, the method for extracting haze features from the hazy image includes: extracting haze features from the hazy image using a focus feature extraction network based on smooth dilated convolution; specifically, it includes the following sub-steps: S21, the focus module is used to sample the foggy image by spatial pixels at intervals of 1, resulting in four downsampled image blocks; S22, stitch the four downsampled image blocks together along the channel direction to obtain a downsampled feature map without loss of resolution; S23, the downsampled feature map is fed into a smooth dilated convolutional network to obtain smooth convolutional features; S24, the smoothed convolutional features and the downsampled feature map are concatenated along the channel direction to obtain the haze features; Step S3 includes the following sub-steps: S31, the haze features are passed through a channel attention weighting layer to obtain channel attention weighted features; S32, the channel attention-weighted features are passed through the spatial attention-weighted layer to obtain the spatial channel attention-weighted features; S33, the spatial channel attention-weighted features are divided into multiple image blocks through a self-attention layer; S34, multiple image patches are processed through a multi-head attention layer to compute multiple self-attention; S35: After concatenating multiple self-attention signals, the signals are passed through an MLP residual network to obtain the encoded features corresponding to multiple image patches. Step S4 includes the following sub-steps: S41, the encoded features corresponding to the first image block are decoded by the self-attention decoder to obtain the decoded features corresponding to the first image block; S42, the decoded features corresponding to the first image block and the encoded features corresponding to the second image block are decoded together by the self-attention decoder to obtain the decoded features corresponding to the second image block; S43, decode the encoded features corresponding to all image blocks in sequence according to the method in step S42, and finally obtain the decoded features corresponding to all image blocks; In step S5, the method for obtaining haze residue by performing haze residue decoding on the decoded features includes: The decoded features corresponding to all image blocks in the original feature space are mapped to the target haze residue space through a multi-layer hybrid convolutional network, thus extracting the haze residue; step S5 includes the following sub-steps: S51, convert the decoding features corresponding to all image blocks to their original size, and then stitch all the decoding features together according to the original position of each image block to form a decoding feature; S52 decodes the assembled features into the original feature space through a dimension adaptation module composed of a deconvolution layer; S53 uses a haze residual mapping module composed of convolutional layers to map the feature mapping of the original feature space to the target haze space, thus obtaining haze residual.

2. The image dehazing method based on multiple attention according to claim 1, characterized in that, In step S1, the original foggy image includes: The image to be processed is a foggy image acquired by an image sensor. The training dataset, namely the RESIDE dataset used in the training phase, is used, and the original RESIDE dataset is resized to the size of the foggy image to be processed.

3. An image dehazing device based on multiple attention, characterized in that, The apparatus is used to implement the multi-attention-based image dehazing method as described in any one of claims 1-2; The device includes: The data acquisition module is used to acquire raw, foggy images; The feature extraction module is used to extract haze features from hazy images; The feature encoding module is used to encode the haze features to obtain encoded features; The feature decoding module is used to decode the encoded features to obtain the decoded features; The haze persistence decoding module is used to perform haze persistence decoding on the decoding features to obtain haze persistence; The residual module is used to obtain a clear, fog-free image by performing a residual operation between the original foggy image and the remaining fog.

Images