A Wavelet-Based Dual-Branch Network Image Deblurring Method, System, and Medium

Abstract

A wavelet-based dual-branch network image de-raining method, system, and medium disclosed by the present invention; the method includes: obtaining a training image set and a test image set of rain-free images and their corresponding synthetic rain images as a sample pair; performing a two-level wavelet transform on the rain images, and performing de-raining operations on the images to be de-rained by transferring them from the image domain to the wavelet domain; extracting the channel and spatial information of the rain images based on a preset adaptive attention branch, and adaptively calculating the weights of the attention and non-attention modules; based on a preset extended branch, expanding the receptive field through different dilated convolution operations; obtaining the de-rained images through inverse wavelet transform; and constraining the de-rained images based on a preset loss function of wavelet and structural similarity to obtain standard de-rained images. The present invention improves the accuracy of image de-raining, and combines the characteristics of rain images to improve the generalization ability of de-raining and the protection ability of image background information.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to a method, system and storage medium for removing rain from images based on a dual-branch network of wavelets. Background Art

[0002] In reality, bad weather outdoors often causes various degrees of degradation of the images obtained by camera devices, which will greatly limit the normal use of computer vision systems. Rainy days are one of the most common bad weather conditions in nature. The intersecting rain streaks cause the background information to become blurred, and the areas affected by the rain streaks cause sudden changes in pixel values, covering the original background information and seriously affecting the subsequent advanced computer vision tasks. Therefore, how to effectively remove the rain streaks on the rain image while retaining the original background information and clarifying the rain image has important significance and application value.

[0003] With the development of deep learning, it has become relatively easy to construct a neural network model to achieve single-image rain removal. However, most of the existing methods build a neural network model based on a convolutional neural network according to experience, which requires a lot of time and effort to debug the network structure, and the locality of its receptive field and the redundancy of the model make the model unable to learn the global information of the rain image well, resulting in the inability to remove the rain streaks well.

[0004] Therefore, there are deficiencies in the prior art and urgent improvements are needed. Summary of the Invention

[0005] In view of the above problems, the purpose of the present invention is to provide a method, system and storage medium for removing rain from images based on a dual-branch network of wavelets, which can more effectively improve the accuracy of rain removal and the generalization ability of the model.

[0006] The first aspect of the present invention provides a method for removing rain from images based on a dual-branch network of wavelets, including:

[0007] Obtaining a training image set and a test image set of rain-free images and their corresponding synthetic rain images as a sample pair;

[0008] Performing a two-level wavelet transform on the synthetic rain image to transfer the image to be de-rained from the image domain to the wavelet domain for rain removal processing;

[0009] Based on a preset dual-branch adaptive extended attention network, obtaining a de-rained image through inverse wavelet transform;

[0010] Constraining the de-rained image based on a preset loss function of wavelets and structural similarity to obtain a standard de-rained image.

[0011] In this solution, the secondary wavelet transform is specifically: the secondary Haar wavelet transform.

[0012] In this solution, the preset dual-branch adaptive extended attention network specifically includes an adaptive attention branch and an extended branch. The adaptive attention branch includes a dynamic weight allocation module, an attention module, and a non-attention module. The attention module and the non-attention module contain multiple residual network modules. The residual network module Res(·) is a ResNet module without the BN layer. The non-attention module consists of two residual network modules and can be expressed as:

[0013] F non-attention =Res(Res(R i ))

[0014] where R i represents the input of the i-th adaptive extended attention module, Res(·) represents the residual network module, and F non-attention represents the output of the non-attention module;

[0015] In addition, the attention module contains multiple residual network modules and up / down sampling layers and can be expressed as:

[0016] F attention =Res(Res(R i ))*σ(Up(Res(Down(R i )))+R i )

[0017] where Up and Down represent the up / down sampling layers respectively, σ(·) represents the Sigmoid activation function, * represents matrix multiplication, and F attention represents the output of the attention module.

[0018] In this solution, the dynamic weight allocation module of the adaptive attention branch specifically generates dynamic weights through a spatial channel attention module and can be expressed as:

[0019] α, β=FC(AvgPool(Concate(MaxPool(R i ),AvgPool(R i ))*R i ))

[0020] where α and β represent two weight vectors generated by the adaptive attention branch, FC represents the fully connected layer, AvgPool represents the average pooling layer, and MaxPool represents the maximum pooling layer; the two generated weight vectors α and β will be assigned to the attention module and the non-attention module, that is, the outputs of the two modules will be multiplied by the corresponding weight vectors and can be expressed as: F1=αF non-attention+βF attention 。

[0021] In this solution, the extended branch is specifically composed of dilated convolutions with four different dilation rates to extract rain streak information at different scales, and then the information at different scales extracted is merged. It can be expressed as:

[0022] F2 = Concatenate(Conv1(R i ), Conv2(R i ), Conv3(R i ), Conv4(R i ))

[0023] where Conv i represents the i-th dilated convolution kernel, F2 is the output of the extended branch, and finally it is fused with the output of the adaptive attention branch to obtain the output feature F out which can be expressed as: F out = F1 + F2.

[0024] In this solution, the loss function based on the preset wavelet and structural similarity is specifically: including the wavelet domain loss function and the structural similarity loss function. The wavelet domain loss function assigns a higher weight to the constraint of the low-frequency part, enabling the network to better restore the low-frequency background information. The loss function is:

[0025]

[0026] where: is the frequency domain information of the i-th channel output by the network, c i is the frequency domain information of the i-th channel after the corresponding rain-free image is transformed into the wavelet domain, N is the total number of channels, and ω1 and ω2 are the weight coefficients for adjusting the two parts of the loss function; the structural similarity loss constrains the loss in the image domain, and its mathematical expression is:

[0027]

[0028] where μ B represents the pixel mean of the rain-free image, μ D represents the pixel mean of the de-rained image, * represents multiplication, σ represents the covariance of the image, and e1 and e2 are constants, e1 = (K1 * L) 2 , e2 = (K2 * L) 2 , and K1, K2, and L are coefficients;

[0029] Therefore, the loss function based on the preset wavelet and structural similarity constrains the de-rained image in the wavelet domain and the image domain respectively. Thus, the total loss function can be expressed as:

[0030] Loss total = λ1Loss1 + λ2Loss2

[0031] where λ1 and λ2 are weight coefficients for adjusting the two parts of the loss function.

[0032] In a second aspect of the present invention, a wavelet-based dual-branch network image de-raining system is provided, including a memory and a processor. A wavelet-based dual-branch network image de-raining method program is stored in the memory. When the wavelet-based dual-branch network image de-raining method program is executed by the processor, the following steps are implemented:

[0033] Obtain a training image set and a test image set of a rain-free image and its corresponding synthesized rain image as a sample pair;

[0034] Perform a two-level wavelet transform on the synthesized rain image to transfer the image to be de-rained from the image domain to the wavelet domain for de-raining processing;

[0035] Based on a preset dual-branch adaptive extended attention network, obtain a de-rained image through inverse wavelet transform;

[0036] Constrain the de-rained image based on a preset loss function of wavelet and structural similarity to obtain a standard de-rained image.

[0037] In this solution, the two-level wavelet transform is specifically: a two-level Haar wavelet transform.

[0038] In this solution, the preset dual-branch adaptive extended attention network specifically includes an adaptive attention branch and an extended branch. The adaptive attention branch includes a dynamic weight allocation module, an attention module, and a non-attention module. The attention module and the non-attention module contain multiple residual network modules. The residual network module Res(·) is a ResNet module without the BN layer. The non-attention module is composed of two residual network modules, which is expressed as:

[0039] F non-attention = Res(Res(R i ))

[0040] where R i represents the input of the i-th adaptive extended attention module, Res(·) represents the residual network module, and F non-attention represents the output of the non-attention module;

[0041] In addition, the attention module contains multiple residual network modules and up / down sampling layers, and can be expressed as:

[0042] F attention = Res(Res(R i)) * σ(Up(Res(Down(R i )) + R i )

[0043] where Up and Down represent the up / down sampling layers respectively, σ(·) represents the Sigmoid activation function, * represents matrix multiplication, and F attention represents the output of the attention module.

[0044] In this solution, the dynamic weight allocation module of the adaptive attention branch is specifically: generating dynamic weights through the spatial channel attention module, which can be expressed as:

[0045] α, β = FC(AvgPool(Concate(MaxPool(R i ) AvgPool(R i )) * R i ))

[0046] where α and β represent two weight vectors generated by the adaptive attention branch, FC represents the fully connected layer, AvgPool represents the average pooling layer, and MaxPool represents the max pooling layer; the two generated weight vectors α and β will be assigned to the attention module and the non-attention module, that is, the outputs of the two modules will be multiplied by the corresponding weight vectors, which can be expressed as: F1 = αF non-attention + βF attention .

[0047] In this solution, the extended branch is specifically: composed of dilated convolutions with four different dilation rates to extract rain streak information of different scales, and then the information of different scales extracted is merged, which can be expressed as:

[0048] F2 = Concatenate(Conv1(R i ), Conv2(R i ), Conv3(R i ), Conv4(R i ))

[0049] where Conv i represents the i-th dilated convolution kernel, F2 is the output of the extended branch, and finally it is fused with the output of the adaptive attention branch to obtain the output feature F out which can be expressed as: F out = F1 + F2.

[0050] In this solution, the loss function based on the preset wavelet and structural similarity is specifically as follows: It includes a wavelet domain loss function and a structural similarity loss function. The wavelet domain loss function assigns a higher weight to the constraint of the low-frequency part, enabling the network to better recover the low-frequency background information. The loss function is:

[0051]

[0052] Where: is the frequency domain information of the i-th channel output by the network, and c i is the frequency domain information of the i-th channel after the corresponding rain-free image is transformed into the wavelet domain. N is the total number of channels, and ω1 and ω2 are weight coefficients for adjusting the two parts of the loss function. The structural similarity loss constrains the loss in the image domain, and its mathematical expression is:

[0053]

[0054] Where μ B represents the pixel mean of the rain-free image, μ D represents the pixel mean of the rain-removed image, * represents multiplication, σ represents the covariance of the image, and e1 and e2 are constants, e1 = (K1 * L) 2 and e2 = (K2 * L) 2 , where K1, K2, and L are coefficients;

[0055] Therefore, the loss function based on the preset wavelet and structural similarity constrains the rain-removed image in the wavelet domain and the image domain respectively. Thus, the total loss function can be expressed as:

[0056] Loss total = λ1Loss1 + λ2Loss2

[0057] Where λ1 and λ2 are weight coefficients for adjusting the two parts of the loss function.

[0058] The third aspect of the present invention provides a computer-readable storage medium. A program for a wavelet-based dual-branch network image rain removal method is stored in the computer-readable storage medium. When the program for the wavelet-based dual-branch network image rain removal method is executed by a processor, the steps of a wavelet-based dual-branch network image rain removal method as described in any one of the above are implemented.

[0059] A method, system, and storage medium for removing rain from images based on a wavelet-based dual-branch network, wherein the method includes: obtaining a training image set and a test image set of a sample pair including a rain-free image and its corresponding synthesized rain image; performing a two-level wavelet transform on the rain image, and performing rain removal operations on the image to be de-rained in the wavelet domain; extracting channel and spatial information of the rain image based on a preset adaptive attention branch, and adaptively calculating the weights of the attention and non-attention modules; based on a preset extended branch, expanding the receptive field through different dilated convolution operations; obtaining the de-rained image through an inverse wavelet transform; and constraining the de-rained image based on a loss function of preset wavelet and structural similarity to obtain a standard de-rained image. The present invention improves the accuracy of image rain removal while combining the characteristics of the rain image, improving the generalization ability of rain removal and the ability to protect the background information of the image. Description of the Drawings

[0060] Figure 1 Shows a flowchart of a method for removing rain from images based on a wavelet-based dual-branch network according to the present invention;

[0061] Figure 2 Shows a diagram of a dual-branch adaptive extended attention network model according to the present invention;

[0062] Figure 3 Shows a model diagram of the adaptive attention branch according to the present invention;

[0063] Figure 4 Shows a model diagram of the extended branch according to the present invention;

[0064] Figure 5 Shows a block diagram of a system for removing rain from images based on a wavelet-based dual-branch network according to the present invention. Detailed Embodiments

[0065] In order to more clearly understand the above objects, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present application and the features in the embodiments may be combined with each other.

[0066] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention may be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0067] Figure 1 Shows a flowchart of a method for removing rain from images based on a wavelet-based dual-branch network according to the present invention.

[0068] As Figure 1As shown in the figure, a wavelet-based dual-branch network image de-raining method disclosed by the present invention includes:

[0069] Step 1: Obtain a training image set and a test image set of a sample pair including a rain-free image and its corresponding synthesized rain image. Since it is difficult to obtain paired rainy and rain-free images from the real world, the dataset Rain800 of synthesized rain images is used when training the network of the present invention. This dataset includes 700 pairs of samples in the training set and 100 pairs of samples in the test set.

[0070] Step 2: Perform a two-level wavelet transform on the synthesized rain image to transfer the image to be de-rained from the image domain to the wavelet domain for de-raining processing. The two-level Haar wavelet transform is used in the present invention.

[0071] Step 3: Based on a preset dual-branch adaptive extended attention network, obtain a de-rained image through inverse wavelet transform. The preset dual-branch adaptive extended attention network includes an adaptive attention branch and an extended branch.

[0072] Step 4: Constrain the de-rained image based on a preset loss function of wavelet and structural similarity to obtain a standard de-rained image.

[0073] In this method, the network structure of the dual-branch adaptive extended attention network mentioned in Step 3 is as Figure 2 shown. After the input of the network model undergoes wavelet transform, it enters multiple dual-branch modules, and finally, the negative rain streak information is extracted and residually added to the original input rain image to obtain a rain-free image.

[0074] The dual-branch module mainly includes an adaptive attention branch and an extended branch module. The adaptive attention branch includes a dynamic weight allocation module, an attention module, and a non-attention module. Its structure is as Figure 3 shown. Among them, the attention module and the non-attention module contain multiple residual network modules. The residual network module Res(·) is a ResNet module without the BN layer. The non-attention module is composed of two residual network modules, which is expressed as:

[0075] F non-attention =Res(Res(R i ))

[0076] where R i represents the input of the i-th adaptive extended attention module, Res(·) represents the residual network module, and F non-attention represents the output of the non-attention module.

[0077] In addition, the attention module contains multiple residual network modules and up / down sampling layers, and can be expressed as:

[0078] F attention= Res(Res(R i )) * σ(Up(Res(Down(R i )))+R i )

[0079] where Up and Down represent the up / down sampling layers respectively, σ(·) represents the Sigmoid activation function, * represents the multiplication of matrices, and F attention represents the output of the attention module.

[0080] The dynamic weight allocation module of the adaptive attention branch generates dynamic weights through the spatial channel attention module, which can be expressed as:

[0081] α, β = FC(AvgPool(Concate(MaxPool(R i ), AvgPool(R i )) * R i )

[0082] where α and β represent two weight vectors generated by the adaptive attention branch, FC represents the fully connected layer, AvgPool represents the average pooling layer, and MaxPool represents the max pooling layer.

[0083] The two generated weight vectors α and β will be assigned to the attention module and the non-attention module, that is, the outputs of the two modules will be multiplied by the corresponding weight vectors, which can be expressed as:

[0084] F1 = αF non-attention + βF attention

[0085] Thus, the model will adaptively combine the information of the attention module and the non-attention module by learning the weight vectors.

[0086] In addition to the adaptive attention branch, in order to increase the receptive field of the model, the model additionally adds an extended branch, and its structure is as Figure 4 shown. This extended branch extracts rain streak information at different scales by dilated convolutions with four different dilation rates, and then merges the information at different scales extracted, which can be expressed as:

[0087] F2 = Concatenate(Conv1(R i ), Conv2(R i ), Conv3(R i ), Conv4(R i ))

[0088] where Conv iIt represents the i-th dilated convolution, and F2 is the output of the expansion branch. Finally, a fusion operation is performed with the output of the adaptive attention branch to obtain the output feature F out It can be expressed as:

[0089] F out = F1 + F2

[0090] In this method, step 4 mentions that the loss function designed based on wavelet and structural similarity includes a wavelet domain loss function and a structural similarity loss function. The wavelet domain loss function assigns higher weights to the constraints on the low-frequency part, enabling the network to better recover the low-frequency background information. The loss function is expressed as:

[0091]

[0092] Among them, is the frequency domain information of the i-th channel output by the network, c i is the frequency domain information of the i-th channel after the corresponding rain-free image is transformed into the wavelet domain. N is the total number of channels, and ω1 and ω2 are the weight coefficients for adjusting the two parts of the loss function, which are set to 1 and 0.1 respectively.

[0093] The structural similarity loss constrains the loss in the image domain, and its mathematical expression is:

[0094]

[0095] Among them, μ B represents the pixel mean of the rain-free image, μ D represents the pixel mean of the de-rained image, * represents multiplication, σ represents the covariance of the image, and e1 and e2 are constants, e1 = (K1 * L) 2 , e2 = (K2 * L) 2 , where K1, K2, and L are coefficients. Generally, K1 = 0.01, K2 = 0.03, and L = 255.

[0096] Therefore, the loss function based on the preset wavelet and structural similarity constrains the de-rained image in the wavelet domain and the image domain respectively. Thus, the total loss function can be expressed as:

[0097] Loss total = λ1Loss1 + λ2Loss2

[0098] Among them, λ1 and λ2 are the weight coefficients for adjusting the two parts of the loss function.

[0099] The above network training process can be implemented based on the deep learning framework Pytorch. When training the model, first preprocess the training dataset, including randomly cropping and rotating 700 sample pairs in the training set to complete data augmentation. Finally, 2,800 training sample pairs can be obtained as the data for model training. When testing the book, 100 test sample pairs are used, and the PSNR and SSIM metrics are used to measure the performance of the model for rain removal. The specific definitions of the PSNR and SSIM metrics are as follows:

[0100] PSNR = 10 * log10(255 2 / mean(mean((X - Y) 2 )))

[0101] SSIM = [L(X, Y) a * [C(X, Y) b * [S(X, Y) c

[0102] Among them, μ X and μ Y represent the means of X and Y respectively, and σ X , σ Y and σ XY represent the variances of X and Y and their covariance respectively. The higher the PSNR and SSIM values, the better the reconstruction effect.

[0103] Figure 5 shows a block diagram of an image rain removal system based on a wavelet-based dual-branch network according to the present invention.

[0104] As Figure 5 shown, a second aspect of the present invention provides an image rain removal system 5 based on a wavelet-based dual-branch network, including a memory 51 and a processor 52. A program of an image rain removal method based on a wavelet-based dual-branch network is stored in the memory. When the program of the image rain removal method based on a wavelet-based dual-branch network is executed by the processor, the following steps are implemented:

[0105] Step 1, obtain a training image set and a test image set including a rain-free image and its corresponding synthetic rain image as a sample pair. Since it is difficult to obtain paired rain and rain-free images from the real world, the dataset Rain800 of synthetic rain images is used when training the network in the present invention. This dataset includes 700 pairs of samples in the training set and 100 pairs of samples in the test set;

[0106] Step 2, perform a two-level wavelet transform on the synthetic rain image to transfer the image to be de-rained from the image domain to the wavelet domain for rain removal processing. The two-level Haar wavelet transform is used in the present invention;

[0107] ​Step 3: Based on the preset dual-branch adaptive extended attention network, through inverse wavelet transform, a rain-removed image is obtained. The preset dual-branch adaptive extended attention network includes an adaptive attention branch and an extended branch;

[0108] Step 4: Based on the loss function of the preset wavelet and structural similarity, the rain-removed image is constrained to obtain a standard rain-removed image.

[0109] In this method, the network structure of the dual-branch adaptive extended attention network mentioned in Step 3 is as Figure 2 shown. After the input of the network model passes through wavelet transform, it enters multiple dual-branch modules, and finally the negative rain streak information and the original input rain image are extracted and added residually to obtain a rain-free image.

[0110] The dual-branch module mainly includes an adaptive attention branch and an extended branch module. The adaptive attention branch includes a dynamic weight allocation module, an attention module, and a non-attention module. Its structure is as Figure 3 shown. Among them, the attention module and the non-attention module contain multiple residual network modules. The residual network module Res(·) is a ResNet module without the BN layer. The non-attention module is composed of two residual network modules, which is expressed as:

[0111] F non-attention =Res(Res(R i ))

[0112] where R i represents the input of the i-th adaptive extended attention module, Res(·) represents the residual network module, and F non-attention represents the output of the non-attention module.

[0113] In addition, the attention module contains multiple residual network modules and up / down sampling layers, which can be expressed as:

[0114] F attention =Res(Res(R i ))*σ(Up(Res(Down(R i )))+R i )

[0115] where Up and Down represent the up / down sampling layers respectively, σ(·) represents the Sigmoid activation function, * represents matrix multiplication, and F attention represents the output of the attention module.

[0116] The dynamic weight allocation module of the adaptive attention branch generates dynamic weights through a spatial channel attention module, which can be expressed as:

[0117] α, β = FC(AvgPool(Concate(MaxPool(R i ), AvgPool(R i )) * R i ))

[0118] Among them, α and β represent two weight vectors generated by the adaptive attention branch, FC represents the fully connected layer, AvgPool represents the average pooling layer, and MaxPool represents the max pooling layer.

[0119] The two generated weight vectors α and β will be assigned to the attention module and the non-attention module, that is, the outputs of the two modules will be multiplied by the corresponding weight vectors, which can be expressed as:

[0120] F1 = αF non-attention + βF attention

[0121] Thus, the model will adaptively combine the information of the attention module and the non-attention module by learning the weight vectors.

[0122] In addition to the adaptive attention branch, in order to increase the receptive field of the model, the model additionally adds an extended branch, and its structure is as Figure 4 shown. This extended branch extracts raindrop pattern information of different scales by dilated convolutions with four different dilation rates, and then merges the extracted information of different scales, which can be expressed as:

[0123] F2 = Concatenate(Conv1(R i ), Conv2(R i ), Conv3(R i ), Conv4(R i ))

[0124] Among them, Conv i represents the i-th dilated convolution, and F2 is the output of the extended branch. Finally, it is fused with the output of the adaptive attention branch to obtain the output feature F out which can be expressed as:

[0125] F out = F1 + F2

[0126] In this method, the loss function based on wavelet and structural similarity mentioned in step 4 includes the wavelet domain loss function and the structural similarity loss function. The wavelet domain loss function assigns higher weights to the constraints on the low-frequency part, enabling the network to better recover the low-frequency background information. The loss function is expressed as:

[0127]

[0128] Among them, is the frequency domain information of the i-th channel of the network output, c i is the frequency domain information of the i-th channel after the corresponding rain-free image is transformed into the wavelet domain. N is the total number of channels, and ω1 and ω2 are the weight coefficients for adjusting the two parts of the loss function, which are set to 1 and 0.1 respectively.

[0129] The structural similarity loss constrains the loss in the image domain, and its mathematical expression is:

[0130]

[0131] Among them, μ B represents the pixel mean of the rain-free image, μ D represents the pixel mean of the rain-removed image, * represents multiplication, σ represents the covariance of the image, and e1 and e2 are constants, e1 = (K1 * L) 2 , e2 = (K2 * L) 2 , K1, K2, and L are coefficients. Generally, K1 = 0.01, K2 = 0.03, and L = 255.

[0132] Therefore, the loss function based on the preset wavelet and structural similarity constrains the rain-removed image in the wavelet domain and the image domain respectively. Therefore, the total loss function can be expressed as:

[0133] Loss total = λ1Loss1 + λ2Loss2

[0134] Among them, λ1 and λ2 are the weight coefficients for adjusting the two parts of the loss function.

[0135] The above network training process can be implemented based on the deep learning framework Pytorch. When training the model, first preprocess the training dataset, including randomly cropping and rotating 700 sample pairs in the training set to complete data augmentation. Finally, 2800 training sample pairs can be obtained as the data for model training; when testing the book, 100 test sample pairs are used, and the PSNR and SSIM metrics are used to measure the rain-removing performance of the model. The specific definitions of the PSNR and SSIM metrics are as follows:

[0136] PSNR = 10 * log10(255 2 / mean(mean((X - Y) 2 )))

[0137] SSIM = [L(X, Y) a * [C(X, Y) b * [S(X, Y) c

[0138] Among them, μ​X and μ Y represent the means of X and Y respectively, and σ X , σ Y and σ XY represent the variances of X and Y and their covariance respectively. The higher the PSNR and SSIM values, the better the reconstruction effect.

[0139] In a third aspect of the present invention, there is provided a computer-readable storage medium storing a program for an image de-raining method based on a wavelet dual-branch network. When the program for the image de-raining method based on the wavelet dual-branch network is executed by a processor, the steps of an image de-raining method based on the wavelet dual-branch network as described in any one of the above are implemented.

[0140] An image de-raining method, system and storage medium based on a wavelet dual-branch network disclosed in the present invention, wherein the method includes: obtaining a training image set and a test image set of a rain-free image and its corresponding synthesized rain image as a sample pair; performing a two-level wavelet transform on the rain image, and performing a de-raining operation on the image to be de-rained by transferring it from the image domain to the wavelet domain; extracting the channel and spatial information of the rain image based on a preset adaptive attention branch, and adaptively calculating the weights of the attention and non-attention modules; based on a preset expansion branch, expanding the receptive field through different dilated convolution operations; obtaining the de-rained image through an inverse wavelet transform; and constraining the de-rained image based on a preset loss function of wavelet and structural similarity to obtain a standard de-rained image. The present invention improves the accuracy of image de-raining, and combines the characteristics of the rain image to improve the generalization ability of de-raining and the protection ability of the image background information.

[0141] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed with each other can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical, or other forms.

[0142] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units; they can be located in one place or distributed to multiple network units; some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, each functional unit in the embodiments of the present invention may all be integrated into one processing unit, or each unit may be separately regarded as one unit, or two or more units may be integrated into one unit; the above-mentioned integrated unit may be implemented in the form of hardware, or in the form of a combination of hardware and software functional units.

[0144] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the aforementioned storage medium includes: removable storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks or optical discs and other various media that can store program codes.

[0145] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present invention, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention. And the aforementioned storage medium includes: removable storage devices, ROM, RAM, magnetic disks or optical discs and other various media that can store program codes.

Claims

1. A wavelet-based dual-branch network image de-raining method, characterized in that Including: Obtain a training image set and a test image set that include a rain-free image and its corresponding synthesized rain image as a sample pair; Perform a two-level wavelet transform on the synthesized rain image, and transfer the image to be de-rained from the image domain to the wavelet domain for de-raining processing; Based on a preset dual-branch adaptive extended attention network, obtain a de-rained image through inverse wavelet transform; Constrain the de-rained image based on a preset loss function of wavelet and structural similarity to obtain a standard de-rained image; The preset dual-branch adaptive extended attention network is specifically as follows: It includes an adaptive attention branch and an extended branch. The adaptive attention branch includes a dynamic weight allocation module, an attention module, and a non-attention module. The attention module and the non-attention module contain multiple residual network modules, and the residual network module is a ResNet module without the BN layer. The non-attention module is composed of two residual network modules, which is expressed as follows: ； Among them, represents the input of the $i$-th adaptive extended attention module, represents the residual network module, represents the output of the non-attention module; In addition, the attention module includes multiple residual network modules and up / down sampling layers, which can be expressed as follows: ； Among them, Up and Down respectively represent the up / down sampling layer, represents the activation function, represents the multiplication of matrices, represents the output of the attention module; The preset loss function of wavelet and structural similarity is specifically: including a wavelet domain loss function and a structural similarity loss function. The wavelet domain loss function assigns higher weights to the constraints of the low-frequency part, enabling the network to better recover the low-frequency background information. The loss function is: ； Wherein: is the frequency domain information of the th channel output by the network, is the frequency domain information of the th channel after the corresponding rain-free image is transformed into the wavelet domain. N is the total number of channels, and are the weight coefficients for adjusting the loss functions of the two parts; the structural similarity loss constrains the loss in the image domain, and its mathematical expression is: ； Among them, represents the pixel mean of the rain-free image, represents the pixel mean of the de-rained image, * represents multiplication, represents the covariance of the image, and are constants, , , , and L are coefficients.

2. The method for removing rain from images using a dual-branch network based on wavelets according to claim 1, wherein The two-level wavelet transform is specifically: a two-level Haar wavelet transform.

3. A wavelet-based dual-branch network image de-raining method according to claim 1, wherein The dynamic weight allocation module of the adaptive attention branch is specifically: generating dynamic weights through a spatial channel attention module, which can be expressed as: ； Among them, and represent two weight vectors generated by the adaptive attention branch, represents a fully connected layer, represents an average pooling layer, represents a max pooling layer; the two generated weight vectors and will be assigned to the attention module and the non-attention module, that is, the outputs of the two modules will be multiplied by the corresponding weight vectors, which can be expressed as: .

4. A wavelet-based dual-branch network image de-raining method according to claim 1, characterized in that The extended branch is specifically: composed of dilated convolutions with four different dilation rates to extract rain streak information at different scales, and then merging the information extracted at different scales, which can be expressed as: ； Among them, represents the i-th dilated convolutional kernel, is the output of the extended branch, and finally, a fusion operation is performed with the output of the adaptive attention branch to obtain the output feature which can be expressed as: .

5. A wavelet-based dual-branch network image de-raining system, characterized in that, Including a memory and a processor. A program for a wavelet-based dual-branch network image de-raining method is stored in the memory. When the program for the wavelet-based dual-branch network image de-raining method is executed by the processor, the following steps are implemented: Obtain a training image set and a test image set that include a rain-free image and its corresponding synthesized rain image as a sample pair; Perform a two-level wavelet transform on the synthesized rain image, and transfer the image to be de-rained from the image domain to the wavelet domain for de-raining processing; Based on a preset dual-branch adaptive extended attention network, obtain a de-rained image through inverse wavelet transform; Constrain the de-rained image based on a preset loss function of wavelet and structural similarity to obtain a standard de-rained image; The preset dual-branch adaptive extended attention network is specifically as follows: It includes an adaptive attention branch and an extended branch. The adaptive attention branch includes a dynamic weight allocation module, an attention module, and a non-attention module. The attention module and the non-attention module contain multiple residual network modules, and the residual network module is a ResNet module without the BN layer. The non-attention module is composed of two residual network modules, and its representation is as follows: ； Among them, represents the input of the i-th adaptive extended attention module, represents the residual network module, represents the output of the non-attention module; In addition, the attention module includes multiple residual network modules and up / down sampling layers, which can be expressed as follows: ； Among them, Up and Down represent the up / down sampling layers, respectively, represents the activation function, represents the multiplication of matrices, represents the output of the attention module; The preset loss function of wavelet and structural similarity is specifically: including a wavelet domain loss function and a structural similarity loss function. The wavelet domain loss function assigns higher weights to the constraints of the low-frequency part, enabling the network to better recover the low-frequency background information. The loss function is: ； Wherein: is the frequency domain information of the th channel of the network output, is the frequency domain information of the th channel after the corresponding rain-free image is transformed into the wavelet domain, N is the total number of channels, and are the weight coefficients for adjusting the two parts of the loss function; the structural similarity loss constrains the loss in the image domain, and its mathematical expression is: ； Among them, represents the pixel mean of the rain-free image, represents the pixel mean of the rain-removed image, * represents multiplication, represents the covariance of the image, and are constants, , , , and L are coefficients.

6. The image de-raining system based on a wavelet-based dual-branch network according to claim 5, wherein, The two-level wavelet transform is specifically: a two-level Haar wavelet transform.

7. A computer-readable storage medium, characterized in that, A program for a wavelet-based dual-branch network image de-raining method is stored in the computer-readable storage medium. When the program for the wavelet-based dual-branch network image de-raining method is executed by the processor, the steps of a wavelet-based dual-branch network image de-raining method as described in any one of claims 1 to 4 are implemented.

Images