Image rain and fog removing method based on U-shaped network with asymmetric double-branch decoder

Abstract

The invention discloses an image rain and fog removing method based on a U-shaped network with an asymmetric double-branch decoder, and belongs to the field of computer vision. The method is implemented through a U-shaped network, wherein the U-shaped network comprises a single-branch encoder and an asymmetric double-branch decoder. The asymmetric double-branch decoder is composed of a pollutant residual error branch and a scene residual error branch, the pollutant residual error branch is used for extracting residual error characteristics of sheltering objects such as rain, fog and snow, and the scene residual error branch is used for recovering scene information influenced by atmospheric light in severe weather. The method comprises the following steps: firstly, inputting an original image with rain and fog into a single-branch encoder to obtain encoding features of each convolution block; and then decoding is carried out in sequence through an asymmetric double-branch decoder, and pollutant and scene residual features are recovered to the same size as the original image. And finally, subtracting the restored feature image from the original image to obtain a clear image after rain and fog removal. According to the method, pollutants in the image can be efficiently removed, and the detail information of the scene can be recovered at the same time.

Description

Technical Field

[0001] The invention relates to the field of computer vision, and in particular to an image rain and fog removal method based on a U-shaped network with an asymmetric double-branch decoder. Background Art

[0002] Most existing image de-raining and defogging methods can only handle a single type of pollutant, and it is difficult to remove multiple pollutants such as rain, fog, and snow at the same time, and there are deficiencies in restoring scene information in bad weather. Therefore, there is an urgent need for a technical solution that can effectively remove multiple pollutants and restore scene information. Summary of the invention

[0003] The purpose of the present invention is to address the defect of the prior art that only focuses on removing pollutants but does not consider restoring scene information, and proposes an image deraining and fogging method based on a U-shaped network with an asymmetric double-branch decoder.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] An image deraining and fogging method based on a U-shaped network with an asymmetric dual-branch decoder comprises the following steps:

[0006] Step 1: Input the original scene image with rain and fog into a single-branch encoder, use several convolution blocks connected in series in the encoder to encode the scene image with rain and fog, and output the encoding features of each convolution block;

[0007] Step 2: Input the encoding features output by the last two convolution blocks of the encoder into the 0th asymmetric dual-branch decoding block in the asymmetric dual-branch decoder, and output the dual-branch decoding features; sequentially complete the decoding processing of the N asymmetric dual-branch decoding blocks in the decoder, where the input of the i-th asymmetric dual-branch decoding block of the decoder is the dual-branch decoding features output by the i1-th asymmetric dual-branch decoding block and the encoding features output by the i+1-th convolution block from the end of the encoder, 1≤i≤N-1, and N is the number of asymmetric dual-branch decoding blocks in the decoder;

[0008] Step 3: restore the dual-branch decoding features output by the last asymmetric dual-branch decoding block to the same size as the original scene image with rain and fog; the dual-branch decoding features include pollutant residual features and scene residual features;

[0009] Step 4: Subtract the restored size dual-branch decoding features from the original rainy and foggy scene image to obtain a clean image after removing rain and fog.

[0010] Furthermore, the asymmetric dual-branch decoding block includes a pollutant residual branch and a scene residual branch, and an input convolution block is respectively provided before the pollutant residual branch and the scene residual branch of the i-th (1≤i≤N-1) asymmetric dual-branch decoding block; the pollutant residual branch and the scene residual branch are respectively used to calculate the pollutant residual features and the scene residual features.

[0011] Furthermore, the pollutant residual branch in the asymmetric dual-branch decoding block includes a channel feature fusion block, a window multi-head self-attention mechanism and an output convolution block. The channel feature fusion block is first used to fuse the two input features to generate a fusion feature of the pollutant residual branch; then the window multi-head self-attention mechanism and the output convolution block are used in turn to obtain the pollutant residual feature based on the fused feature.

[0012] The scene residual branch includes a global channel feature fusion block, a shifted window multi-head self-attention mechanism and an output convolution block. First, the global channel feature fusion block is used to fuse two input features to generate a fused feature of the scene residual branch; then the shifted window multi-head self-attention mechanism and the output convolution block are used in turn to obtain a scene residual feature based on the fused feature.

[0013] Furthermore, the calculation of the channel feature fusion block is expressed as:

[0014]

[0015] in, represents element-by-element addition, σ, BN, and ReLU represent the sigmoid function, batch normalization, and rectified linear unit activation functions, respectively. Indicates element-by-element multiplication, I indicates multiplication with W c The same size unit matrix, Conv represents the convolutional layer, represents the first and second input features of the pollutant residual branch, W c represents the attention weight of the pollutant residual branch, G c Represents the fused features of the pollutant residual branch.

[0016] Furthermore, the calculation of the global channel feature fusion block is expressed as:

[0017]

[0018]

[0019] in, represents element-by-element addition, σ, BN and ReLU represent the sigmoid function, batch normalization and rectified linear unit activation functions respectively, GAP represents the global average pooling operation, Indicates element-by-element multiplication, i indicates multiplication with w s Unit vectors of the same size, Conv represents the convolutional layer, represents the first and second input features of the scene residual branch, w s represents the attention weight of the scene residual branch, G s Represents the fused features of the scene residual branch.

[0020] Furthermore, in the i-th (1≤i≤N-1) asymmetric dual-branch decoding block, the pollutant residual features and the scene residual features output by the i-1-th asymmetric dual-branch decoding block are first fused through the input convolution blocks set before the pollutant residual branch and the scene residual branch, and the fusion results are used as the first input features of the two branches, respectively, and the encoded features output by the penultimate i+1-th convolution block in the encoder are used as the second input features of the two branches;

[0021] In the 0th asymmetric dual-branch decoding block, the encoded features output by the last two convolution blocks of the encoder are directly used as the first input features and the second input features of the two branches.

[0022] The beneficial effect of the present invention is that it solves the problem that the existing image deraining and fogging network does not consider the importance of restoring scene information, resulting in the lack of authenticity of the restored image, and while removing pollutants in the image, it also restores the scene detail information as much as possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is the basic structure diagram of the U-shaped network framework with an asymmetric dual-branch decoder.

[0024] Figure 2 This is the basic structure diagram of the asymmetric dual-branch decoding block framework.

[0025] Figure 3 This is the basic structure diagram of the channel feature fusion block and the global channel feature fusion block framework. DETAILED DESCRIPTION

[0026] The present invention is described in detail below with reference to the accompanying drawings. The accompanying drawings are only schematic illustrations of the present invention, and some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities, and these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0027] The present invention proposes an image de-raining and fogging method based on a U-shaped network with an asymmetric dual-branch decoder. The U-shaped network with an asymmetric dual-branch decoder is composed of a single-branch encoder and a dual-branch decoder. Figure 1 As shown, the single-branch encoder consists of five convolutional blocks, and the dual-branch decoder consists of four asymmetric dual-branch decoding blocks and two convolutional blocks.

[0028] like Figure 1 The framework structure shown in the figure, the implementation of the present invention mainly includes the following steps:

[0029] Step 1: Input the original scene image with rain and fog into the single-branch encoder, use the five convolution blocks in the encoder to encode the image with rain and fog in turn, extract shallow features, and output the features of each convolution block;

[0030] like Figure 1 As shown, in the encoder, this embodiment has five convolution blocks, each block uses Conv i (0≤i≤4). The output of each convolutional block is represented by F i =Conv i (F i-1 ) indicates that the output of the previous convolution block is the input of the next convolution block, where That is, the original image is input. In this embodiment, each convolution block includes two convolution layers and one maximum pooling layer. Each time a convolution block in the encoder is passed, the feature length and width will be reduced by half, and the channel will be doubled.

[0031] Step 2: Input the features output by the last convolution block of the encoder and the features output by the penultimate convolution block into the first asymmetric dual-branch decoder block in the dual-branch decoder for decoding, and the two asymmetric branches output the pollutant residual features and the scene residual features respectively, which are then input into the next asymmetric dual-branch decoder together with the features output by the penultimate convolution block, and so on until the calculation of the fourth asymmetric dual-branch decoder block is completed;

[0032] In this step, if Figure 1 As shown, the asymmetric dual-branch decoder is composed of four asymmetric dual-branch decoding blocks (ADB j , j=0,1,2,3) and two convolutional layers. The asymmetric dual-branch decoding block ADB 0 ( Figure 2 (a)) and ADB j (j=1,2,3)( Figure 2 (b)) has some differences in structure, ADB j (j=1,2,3) by ADB 0 After some modification, first use ADB 0 For example, the following steps are implemented:

[0033] (1) Pollutant residual branch

[0034] First, the features F output by the last two convolutional blocks in the encoder 3 and F 4 As the input channel feature fusion block (CFF(·)), that is:

[0035]

[0036] In a specific implementation of the present invention, the channel feature fusion block framework is as follows Figure 3 As shown in (a), the two inputs are first fused by element-wise addition, and then the fused features are passed to the 2-layer convolutional block to obtain the attention weights. It is expressed as:

[0037]

[0038] in, represents element-by-element addition, σ, BN, and ReLU represent the sigmoid function, batch normalization, and rectified linear unit activation functions, respectively. Here, Conv represents a convolutional layer with a convolution kernel size of 1×1, which can be understood as applying a fully connected layer to channel features.

[0039] Next, the attention weights are applied to the input feature map and the fused features are obtained As shown below:

[0040]

[0041] in, represents element-by-element multiplication, and I represents The identity matrix of the same size, Represents the fused features output by the input channel feature fusion block.

[0042] The channel feature fusion block is used to fuse the input features, and the fusion weights are generated by the module. The self-attention mechanism is then further used to build the fusion feature The spatial long-range dependence of , we get:

[0043]

[0044] Among them, SA is a window multi-head self-attention mechanism. After the input features are fused and processed by the attention mechanism, the contaminated residual features can be obtained through the output convolution block. As shown below:

[0045]

[0046] Among them, Conv out,1Represents a convolutional block with two layers of convolution in the pollutant residual branch. The first layer has a 3×3 kernel size, followed by a leaky ReLU layer, followed by a 1×1 kernel size, and also followed by a leaky ReLU layer.

[0047] (2) Scene residual branch

[0048] First, the features F output by the last two convolutional blocks in the encoder 3 and F 4 As the input global channel feature fusion block (GCFF(·)), that is:

[0049]

[0050] In a specific implementation of the present invention, the global channel feature fusion block framework is as follows Figure 3 As shown in (b), firstly, F 3 and F 4 Add them element by element, and then use the global average pooling operation (GAP) to get its global feature vector Right now:

[0051]

[0052] Then, a two-layer convolutional block is used to adjust the global feature vector Each element of The specific expression is:

[0053]

[0054] Then use the fusion weights To fuse the input feature map, the specific expression is:

[0055]

[0056] Among them, i represents Unit vectors of the same magnitude.

[0057] Subsequently, the shifted window multi-head self-attention SSA is used to enhance the spatial interaction of features and obtain the scene residual features, which are described as follows:

[0058]

[0059] Among them, Conv out,2Represents a convolutional block with two layers of convolution in the scene residual branch. The first layer has a 3×3 kernel size, followed by a convolutional layer with a leaky rectified linear unit, followed by a convolutional layer with a 1×1 kernel size, and also followed by a convolutional layer with a leaky rectified linear unit.

[0060] Next, we introduce the asymmetric dual branch decoding block (ADB j (j=1,2,3)) structure:

[0061] like Figure 2 As shown, the first asymmetric dual branch decoder ADB 0 The difference is that in ADB j (j=1,2,3), before entering the two branches, the pollutant residual characteristics are first and scene residual features Perform preliminary fusion, the calculation process includes:

[0062] In any j-th (1≤j≤3) asymmetric dual-branch decoding block, its input includes the pollutant residual features from the output of the j-1-th asymmetric dual-branch decoding block and scene residual features and the output F from the 3-jth convolutional block in the encoder 3-j First, and Concatenate by channel and reduce its dimension from 2d×h×w to d×h×w as follows:

[0063]

[0064] In two branches, Perform convolution operation to obtain and

[0065]

[0066] Among them, Conv in,1 、Conv in,2 Both represent 2-layer convolution blocks with batch normalization and rectified linear unit activation function, where the convolution kernel size of the convolution layer is 3×3.

[0067] In the pollutant residual branch, and F 3-j As input, it passes through the channel feature fusion block CFF, the window multi-head self-attention mechanism, and the output convolution block to obtain the contaminated residual feature The specific implementation process parameters are as described in (1) above.

[0068] In the scene residual branch, and F 3-j As input, it passes through the global channel feature fusion block GCFF, the shift window multi-head self-attention, and the output convolution block to obtain the contaminated residual feature The specific implementation process parameters are as described in (2) above.

[0069] Step 3: After the last asymmetric dual-branch decoding block, the output pollutant residual features and scene residual features Decoding is performed through the convolution block to restore the channel dimension to the image space (i.e., channel C = 3), that is,

[0070]

[0071]

[0072] Among them, Conv 5 The convolution layer has a convolution kernel size of 1×1, and the output Y c That is the final pollutant residual, Y s This is the final scene residual.

[0073] Step 4: Subtract the pollutant residual Y from the original rainy and foggy image I c and scene residual Y s Get the clean image Y after removing rain and fog, that is:

[0074] Y=IY c -Y s

[0075] The above U-network consisting of a single-branch encoder and a two-branch decoder is trained with a negative structural similarity loss (SSIM), i.e.:

[0076]

[0077] Among them, I gt represents the real clean image, and Y represents the clean image after removing rain and fog.

[0078] Finally, the image with rain and fog is input into the trained network to obtain a clean image after removing the rain and fog.

[0079] The above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many variations are possible. All variations that can be directly derived or associated with the contents disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

Claims

1. An image defogging method based on a U-shaped network with an asymmetric dual-branch decoder, characterized in that: The following steps are involved: Step 1: Input the original scene image with rain and fog into a single-branch encoder, use several convolution blocks connected in series in the encoder to encode the scene image with rain and fog, and output the encoding features of each convolution block; Step 2: Input the encoded features output by the last two convolution blocks of the encoder into the 0th asymmetric dual-branch decoding block in the asymmetric dual-branch decoder, and output the dual-branch decoding features; Complete the decoding processing of N asymmetric dual-branch decoding blocks in the decoder in sequence, where the input of the i-th asymmetric dual-branch decoding block of the decoder is the dual-branch decoding feature output by the i-1-th asymmetric dual-branch decoding block and the encoding feature output by the penultimate i+1-th convolution block in the encoder, 1≤i≤N-1, N is the number of asymmetric dual-branch decoding blocks in the decoder; Step 3: restore the dual-branch decoding features output by the last asymmetric dual-branch decoding block to the same size as the original scene image with rain and fog; the dual-branch decoding features include pollutant residual features and scene residual features; Step 4: Subtract the restored size dual-branch decoding features from the original rainy and foggy scene image to obtain a clean image after removing rain and fog.

2. The image deraining and fogging method based on a U-shaped network with an asymmetric dual-branch decoder according to claim 1 is characterized in that: The U-shaped network is composed of the single-branch encoder and the asymmetric dual-branch decoder, and the U-shaped network is trained using a structural similarity loss function.

3. The image de-raining and fogging method based on a U-shaped network with an asymmetric dual-branch decoder according to claim 2 is characterized in that: Each convolution block in the single-branch encoder contains two convolution layers and one maximum pooling layer. After each convolution block, the length and width of the input feature are reduced by half, and the number of channels is doubled.

4. The image de-raining and fogging method based on a U-shaped network with an asymmetric dual-branch decoder according to claim 1 or 3, characterized in that: The number of convolution blocks in the single-branch encoder is greater than the number of asymmetric dual-branch decoding blocks in the asymmetric dual-branch decoder.

5. The image defogging method based on a U-shaped network with an asymmetric dual-branch decoder according to claim 1, characterized in that: The asymmetric dual-branch decoder includes several asymmetric dual-branch decoding blocks and two convolutional layers. The two convolutional layers are used to restore the pollutant residual features and scene residual features output by the last asymmetric dual-branch decoding block to the original scene image size with rain and fog.

6. The image de-raining and fogging method based on a U-shaped network with an asymmetric dual-branch decoder according to claim 5, characterized in that: The asymmetric dual-branch decoding block includes a pollutant residual branch and a scene residual branch, and an input convolution block is respectively provided before the pollutant residual branch and the scene residual branch of the i-th, 1≤i≤N-1-th asymmetric dual-branch decoding block; the pollutant residual branch and the scene residual branch are respectively used to calculate the pollutant residual features and the scene residual features.

7. The image de-raining and fogging method based on a U-shaped network with an asymmetric dual-branch decoder according to claim 6, characterized in that: The pollutant residual branch in the asymmetric dual-branch decoding block includes a channel feature fusion block, a window multi-head self-attention mechanism and an output convolution block. First, the channel feature fusion block is used to fuse the two input features to generate the fusion feature of the pollutant residual branch; Then, the window multi-head self-attention mechanism and the output convolution block are used in sequence to obtain the pollutant residual features based on the fusion features; The scene residual branch includes a global channel feature fusion block, a shifted window multi-head self-attention mechanism and an output convolution block. First, the global channel feature fusion block is used to fuse two input features to generate a fused feature of the scene residual branch; then the shifted window multi-head self-attention mechanism and the output convolution block are used in turn to obtain a scene residual feature based on the fused feature.

8. The image de-raining and fogging method based on a U-shaped network with an asymmetric dual-branch decoder according to claim 7, characterized in that: The calculation of the channel feature fusion block is expressed as: in, represents element-by-element addition, σ, BN, and ReLU represent the sigmoid function, batch normalization, and rectified linear unit activation functions, respectively. Indicates element-by-element multiplication, I indicates multiplication with W c The same size unit matrix, Conv represents the convolutional layer, represents the first and second input features of the pollutant residual branch, W c represents the attention weight of the pollutant residual branch, G c Represents the fused features of the pollutant residual branch.

9. The image de-raining and fogging method based on a U-shaped network with an asymmetric dual-branch decoder according to claim 7, characterized in that: The calculation of the global channel feature fusion block is expressed as: in, represents element-by-element addition, σ, BN and ReLU represent the sigmoid function, batch normalization and rectified linear unit activation functions respectively, GAP represents the global average pooling operation, Indicates element-by-element multiplication, i indicates multiplication with w s Unit vectors of the same size, Conv represents the convolutional layer, represents the first and second input features of the scene residual branch, w s represents the attention weight of the scene residual branch, G s Represents the fused features of the scene residual branch.

10. The image defogging method based on a U-shaped network with an asymmetric dual-branch decoder according to claim 1, characterized in that: In the i-th, 1≤i≤N-1th asymmetric dual-branch decoding block, the pollutant residual features and the scene residual features output by the i-1th asymmetric dual-branch decoding block are first fused through the input convolution blocks set before the pollutant residual branch and the scene residual branch, and the fusion results are used as the first input features of the two branches, respectively, and the encoding features output by the penultimate i+1th convolution block in the encoder are used as the second input features of the two branches; In the 0th asymmetric dual-branch decoding block, the encoded features output by the last two convolution blocks of the encoder are directly used as the first input features and the second input features of the two branches.

Images