Direction-aware attention image deraining method, system and device based on wavelet

Abstract

The present invention discloses a method, system and device for removing rain from an image based on wavelet-based direction-aware attention. The method includes wavelet decomposition and embedding expression of a rain image, vector decomposition and feature expression based on direction-awareness, inverse wavelet background reconstruction, and joint optimization of gradient weights of image space and color channels. In the process of wavelet decomposition and embedding expression of a rain image, Haar wavelet decomposition is used to obtain wavelet components of four different frequencies, namely low frequency, horizontal, vertical and diagonal. In the process of vector decomposition and feature expression based on direction-awareness, horizontal and vertical components are separated and reconstructed from low frequency and diagonal rain components through a direction-aware attention learning strategy. In the process of inverse wavelet background reconstruction, the wavelet components corresponding to a clean background image are obtained by removing rain interference of different components at different stages of the network, and the background image is obtained through the corresponding Haar wavelet inverse transform. The present invention achieves higher quality background texture restoration.

Description

Technical Field

[0001] The present invention belongs to the field of digital image technology, and relates to an image deraining method, system and device, and specifically to a wavelet-based direction-aware attention image deraining method, system and device. Background Art

[0002] Under low visibility conditions on rainy days, image quality will be disturbed to varying degrees, including image content occlusion, brightness changes, contrast distortion, etc., which greatly affects the visibility of image information and restricts the processing accuracy of downstream high-level visual tasks. Therefore, image deraining is an important preprocessing step for many computer vision tasks (such as segmentation, detection, tracking, recognition, classification, etc.).

[0003] In recent years, deep learning-based image deraining methods have been proposed one after another. In 2017, Fu et al. first constructed a three-layer CNN to simultaneously evaluate rain streaks and perform image deraining. In 2018, Zhang et al. proposed a multi-task deep neural network to evaluate and remove rain streak density by considering the density information of rain streaks. In 2019, Yang et al. proposed a multi-stage recurrent neural network to restore the effect of rainy images through staged deraining.

[0004] Existing deep learning-based image deraining models use convolutional neural networks to directly model the rain distribution in rainy images, and obtain the derained image by subtracting the rain distribution from the original input. Although this method can remove most rain streaks, it does not consider the specific direction of rain streaks, and has a significant impact on the vertical background texture or causes color distortion. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a wavelet-based direction-aware attention image deraining method, system and device.

[0006] The technical solution adopted by the method of the present invention is: a wavelet-based direction-awareness attention image deraining method, which uses a wavelet-based direction-awareness attention image deraining network to derain rainy images;

[0007] The wavelet-based direction-aware attention image deraining network includes an initial embedding expression layer, a plurality of direction-aware attention modules and a wavelet reconstruction module;

[0008] The initial embedding expression layer uses three parallel 3×3 convolutions to respectively Rain , vertical component V Rain The combined features of low-frequency and diagonal components are embedded and expressed, and then the embedded expression features are input into the direction-aware attention module to express the features of a specific direction;

[0009] The direction-aware attention module includes a residual channel attention module, a horizontal direction-aware attention unit and a vertical direction-aware attention unit; firstly, the embedded features of the low-frequency and diagonal components are deeply extracted through a residual channel attention module to obtain the corresponding rain feature expression; then, the horizontal rain component is separated and extracted from the low-frequency and diagonal rain feature expression through a horizontal direction-aware attention unit, and the vertical rain component is separated and extracted from the low-frequency and diagonal rain feature expression through a vertical direction-aware attention unit, and the corresponding horizontal and vertical embedded expressions are updated to obtain the rain removal background feature; then, the corresponding horizontal rain removal background feature, the vertical rain removal background feature, and the rain feature of the low-frequency and diagonal components are input into the next direction-aware attention module for further feature expression; by stacking a number of direction-aware attention modules, clean horizontal and vertical background features and low-frequency and diagonal rain expressions are finally obtained;

[0010] The wavelet reconstruction module projects the wavelet features in the specific directions into the image space through four parallel 3×3 convolutions to obtain corresponding wavelet components;

[0011] The method comprises the following steps:

[0012] Step 1: Take the rain image I Rain Decomposed into low frequency components A Rain , horizontal component H Rain , vertical component V Rain and the diagonal component D Rain ;

[0013] Step 2: Input the four-component data into the initial embedding expression layer and extract the vertical component V through convolution Rain and the horizontal component H Rain The initial feature f V,R and f H,R , and at the same time the low frequency component A Rain and the diagonal component D Rain Perform cascade combination and extract the mixed initial feature f through convolution AD, ;

[0014] Step 3: The initial feature f in step 2 AD,R 、f V,R and f H,R Input to the direction-aware attention module, through a channel attention module from f AD,R The mixed rain information is extracted from the horizontal direction perception attention unit and the vertical direction perception attention unit are used to respectively distill the horizontal rain information in a specific direction from the extracted mixed rain information.H,AD,R and vertical rain information f V,AD,R ;

[0015] Step 4: From the initial feature f H,R and f V,R Subtract the corresponding horizontal rain information f obtained in step 3 from H,AD,R and vertical rain information f V,AD,R , obtain the rain removal background features f corresponding to the horizontal and vertical components H,B and f V,B ;

[0016] Step 5: Loop through steps 3 and 4 N times to progressively extract the high-precision low-frequency component A. Rain and the diagonal component D Rain The mixed rainwater information gradually obtains the clean background features of the corresponding horizontal and vertical components; where N is a preset value;

[0017] Step 6: Convert the low frequency component A in step 5 Rain and the diagonal component D Rain The mixed rainwater information is separated to obtain the corresponding low-frequency component A Rain and the diagonal component D Rain The rain information is mapped back to the image space at the same time to obtain the rain residual image A corresponding to the low-frequency component and the diagonal component R and D R , and subtract the corresponding rain residual image from the original low-frequency component and diagonal component to obtain the rain-removed background A corresponding to the low-frequency component and diagonal component B and D B ; At the same time, the de-raining background features of the horizontal component and the vertical component in step 5 are mapped back to the image space to obtain the corresponding de-raining background image H of the horizontal component and the vertical component B and V B ;

[0018] Step 7: Take the A obtained in step 6 B , D B , H B and V B Perform inverse wavelet transform to obtain a clean background image without rain.

[0019] The technical solution adopted by the system of the present invention is: a wavelet-based direction-awareness attention image deraining system, which uses a wavelet-based direction-awareness attention image deraining network to derain rainy images;

[0020] The wavelet-based direction-aware attention image deraining network includes an initial embedding expression layer, a plurality of direction-aware attention modules and a wavelet reconstruction module;

[0021] The initial embedding expression layer uses three parallel 3×3 convolutions to respectively Rain , vertical component V Rain The combined features of low-frequency and diagonal components are embedded and expressed, and then the embedded expression features are input into the direction-aware attention module to express the features of a specific direction;

[0022] The direction-aware attention module includes a residual channel attention module, a horizontal direction-aware attention unit and a vertical direction-aware attention unit; firstly, the embedded features of the low-frequency and diagonal components are deeply extracted through a residual channel attention module to obtain the corresponding rain feature expression; then, the horizontal rain component is separated and extracted from the low-frequency and diagonal rain feature expression through a horizontal direction-aware attention unit, and the vertical rain component is separated and extracted from the low-frequency and diagonal rain feature expression through a vertical direction-aware attention unit, and the corresponding horizontal and vertical embedded expressions are updated to obtain the rain removal background feature; then, the corresponding horizontal rain removal background feature, the vertical rain removal background feature, and the rain feature of the low-frequency and diagonal components are input into the next direction-aware attention module for further feature expression; by stacking a number of direction-aware attention modules, clean horizontal and vertical background features and low-frequency and diagonal rain expressions are finally obtained;

[0023] The wavelet reconstruction module projects the wavelet features in the specific directions into the image space through four parallel 3×3 convolutions to obtain corresponding wavelet components;

[0024] The system includes the following modules:

[0025] Module 1 is used to process the rain image I Rain Decomposed into low frequency components A Rain , horizontal component H Rain , vertical component V Rain and the diagonal component D Rain ;

[0026] Module 2 is used to input the four-component data into the initial embedding expression layer and extract the vertical component V through convolution Rain and the horizontal component H Rain The initial feature f V,R and f H,R , and at the same time the low frequency component A Rain and the diagonal component D Rain Perform cascade combination and extract the mixed initial feature f through convolution AD, R;

[0027] Module 3 is used to convert the initial feature f in module 2 AD,R 、fV,R and f H,R Input to the direction-aware attention module, through a channel attention module from f AD,R The mixed rain information is extracted from the horizontal direction perception attention unit and the vertical direction perception attention unit are used to respectively distill the horizontal rain information in a specific direction from the extracted mixed rain information. H,AD,R and vertical rain information f V,AD,R ;

[0028] Module 4 is used to obtain the initial feature f H,R and f V,R Subtract the corresponding horizontal rain information f obtained in module 3 H,AD,R and vertical rain information f V,AD,R , obtain the rain removal background features f corresponding to the horizontal and vertical components H,B and f V,B ;

[0029] Module 5 is used to loop through module 3 and module 4 N times to progressively extract the high-precision low-frequency component A. Rain and the diagonal component D Rain The mixed rainwater information gradually obtains the clean background features of the corresponding horizontal and vertical components; where N is a preset value;

[0030] Module 6 is used to convert the low frequency component A in module 5 Rain and the diagonal component D Rain The mixed rainwater information is separated to obtain the corresponding low-frequency component A Rain and the diagonal component D Rain The rain information is mapped back to the image space at the same time to obtain the rain residual image A corresponding to the low-frequency component and the diagonal component R and D R , and subtract the corresponding rain residual image from the original low-frequency component and diagonal component to obtain the rain-removed background A corresponding to the low-frequency component and diagonal component B and D B At the same time, the rain-removed background features of the horizontal component and the vertical component in module 5 are mapped back to the image space to obtain the corresponding rain-removed background image H of the horizontal component and the vertical component B and V B ;

[0031] Module 7 is used to convert the A obtained in module 6 into B , D B , H B and V B Perform inverse wavelet transform to obtain a clean background image without rain.

[0032] The technical solution adopted by the device of the present invention is: a direction-aware attention image deraining device based on wavelet, comprising:

[0033] one or more processors;

[0034] A storage device is used to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the wavelet-based direction-aware attention image deraining method.

[0035] The present invention introduces a direction-aware attention learning mechanism by combining wavelet transform and vector decomposition, and learns the feature expression of a specific direction in the frequency domain space, thereby achieving background texture restoration with higher accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A method framework diagram of an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of the structure of a wavelet-based direction-aware attention image deraining network according to an embodiment of the present invention;

[0038] Figure 3 A structural diagram of a direction-aware attention unit according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0040] Please see Figure 1 , the present invention provides a wavelet-based direction-aware attention image deraining method, which uses a wavelet-based direction-aware attention image deraining network to derain rainy images;

[0041] Please see Figure 2 ,The direction-aware attention image deraining network based on wavelet in this embodiment includes an initial embedding expression layer, a number of direction-aware attention modules and a wavelet reconstruction module;

[0042] The initial embedding expression layer of this embodiment uses three parallel 3×3 convolutions to respectively perform the horizontal component H Rain , vertical component V Rain The combined features of low-frequency and diagonal components are embedded and expressed, and then the embedded expression features are input into the direction-aware attention module to express the features of a specific direction;

[0043] Please see Figure 2The direction-aware attention module of this embodiment includes a residual channel attention module, a horizontal direction-aware attention unit, and a vertical direction-aware attention unit. First, the embedded features of the low-frequency and diagonal components are deeply extracted through a residual channel attention module to obtain the corresponding rain feature expression. Then, the horizontal rain component is separated and extracted from the low-frequency and diagonal rain feature expression through a horizontal direction-aware attention unit, and the vertical rain component is separated and extracted from the low-frequency and diagonal rain feature expression through a vertical direction-aware attention unit. The corresponding horizontal and vertical embedded expressions are updated to obtain the rain removal background feature. Subsequently, the corresponding horizontal rain removal background feature, vertical rain removal background feature, and low-frequency and diagonal component rain feature are input into the next direction-aware attention module for further feature expression. By stacking a number of direction-aware attention modules, clean horizontal and vertical background features and low-frequency and diagonal rain expressions are finally obtained.

[0044] The wavelet reconstruction module of this embodiment projects the wavelet features in the above-mentioned specific directions into the image space through four parallel 3×3 convolutions to obtain corresponding wavelet components;

[0045] Please see Figure 3 The residual channel attention module of this embodiment is composed of two 3×3 convolutional layers with 48 channels, a global average pooling layer, two 1×1 convolutional layers with 12 channels, and a Sigmoid function layer connected in series; the output of the second 3×3 convolutional layer with 48 channels is multiplied pixel by pixel with the output of the Sigmoid function layer, and then summed pixel by pixel with the original input and output.

[0046] The horizontal direction perception attention unit and the vertical direction perception attention unit of this embodiment are composed of a directional average pooling layer, a 1×1 convolution layer with 8 channels, a nonlinear mapping layer, a 1×1 convolution layer with 64 channels, a Sigmoid function layer, and a depth-separated residual channel attention unit. The output of the Sigmoid function layer is multiplied pixel by pixel with the original input, and then sent to the depth-separated residual channel attention unit. The output of the depth-separated residual channel attention unit is multiplied pixel by pixel with the original input. Among them, the depth-separated residual channel attention unit is a unit obtained by replacing the 3x3 convolution in the residual channel attention module with a depth-separable convolution.

[0047] The direction-aware attention unit constructed in this embodiment has three main functions: position weight learning based on a specific direction, rain information extraction and fusion in a specific direction; the first part is to input the mixed low-frequency and diagonal rain information into the direction-aware attention unit to learn the rain weight in a specific direction. Specifically, the descriptor of a specific direction is obtained through an average pooling operation in one direction (horizontal or vertical direction), and then the number of channels is adjusted through two 1×1 convolutions, and the attention weight in a specific direction is learned through the Sigmoid function. The second part uses the attention weight in a specific direction to act on the mixed low-frequency and diagonal rain information to realize the extraction of rain stripes in a specific direction. The third part uses the channel attention unit to further realize the refined expression of rain information, specifically including two 3×3 convolutions to enhance the feature expression, in which the traditional convolution is replaced by the deep separation convolution to reduce the amount of calculation and complexity. In addition, two 1×1 convolutions are applied to adjust the number of channels, and the channel attention weight is learned through the global pooling layer and the Sigmoid function, which acts on the enhanced features to realize the refined expression of rain information.

[0048] The method comprises the following steps:

[0049] Step 1: Take the rain image I Rain Decomposed into low frequency components A Rain , horizontal component H Rain , vertical component V Rain and the diagonal component D Rain ;

[0050] In this embodiment, Haar wavelet transform is used to transform the rain image I to be processed. Rain Decomposed into low frequency components A Rain , horizontal component H Rain , vertical component V Rain and the diagonal component D Rain ;

[0051] A Rain ,H Rain ,V Rain ,D Rain =Haar(I Rain );

[0052] Here, Haar(·) represents Haar wavelet transform.

[0053] Step 2: Input the four-component data into the initial embedding expression layer and extract the vertical component V through convolution Rain and the horizontal component H Rain The initial feature f V,R and f H,R , and at the same time the low frequency component A Rain and the diagonal component D RainPerform cascade combination and extract the mixed initial feature f through convolution AD,R ;

[0054] In this embodiment, the four-component data is input into the initial embedding expression layer, and the wavelet component of the rainy image is mapped from the image space to the feature space to obtain the initial feature f of the corresponding component. AD,R , f H,R , f V,R ;

[0055] f AD,R =F AD,R (A Rain ,D Rain );f H,R =F H,R (H Rain );f V,R =F V,R (V Rain ).

[0056] Step 3: The initial feature f in step 2 AD,R 、f V,R and f H,R Input to the direction-aware attention module, through a channel attention module from f AD,R The mixed rain information is extracted from the image. At the same time, based on the vector decomposition theory, the horizontal direction perception attention unit and the vertical direction perception attention unit are used to respectively distill the horizontal rain information in a specific direction from the extracted mixed rain information. H,AD,R and vertical rain information f V,AD,R ;

[0057] In this embodiment, a channel attention unit is first used to extract mixed rain information from the mixed initial features of low-frequency and diagonal components, specifically using the formula:

[0058] f RCAB,AD,R =F RCAB (f AD,R );

[0059] Among them, f RCAB,AD,R Represents the channel attention unit F RCAB (·) Mixed rain information extracted from the mixed initial features of low-frequency and diagonal components.

[0060] At the same time, the direction perception unit is used to learn the feature expressions of specific horizontal and vertical directions from the above mixed rain information, and the rain component features f in the corresponding directions are obtained. H,AD,R and f V,AD,R , the specific formula used is:

[0061] f H,AD,R =F DAU,H (fRCAB,AD,R );f V,AD,R =F DAU,V (f RCAB,AD,R );

[0062] Among them, F DAU,H (·) and F DAU,V (·) denote the directional attention units for extracting horizontal and vertical rain information, respectively.

[0063] Step 4: From the initial feature f H,R and f V,R Subtract the corresponding horizontal rain information f obtained in step 3 from H,AD,R and vertical rain information f V,AD,R , obtain the rain removal background features f corresponding to the horizontal and vertical components H,B and f V,B ;

[0064] In this embodiment, from the initial expressions of the horizontal and vertical components f H,R and f V,R Subtract the learned rain information f corresponding to the horizontal and vertical directions from H,AD,R and f V,AD,R , obtain the rain removal background feature f in the corresponding direction H,B and f V,B , the specific formula used is:

[0065] f H,B =f H,R -f H,AD,R ;f V,B =f V,R -f V,AD,R .

[0066] Step 5: Loop through steps 3 and 4 N times to progressively extract the high-precision low-frequency component A. Rain and the diagonal component D Rain The mixed rainwater information gradually obtains the clean background features of the corresponding horizontal and vertical components; where N is a preset value;

[0067] Step 6: Convert the low frequency component A in step 5 Rain and the diagonal component D Rain The mixed rainwater information is separated to obtain the corresponding low-frequency component A Rain and the diagonal component D Rain The rain information is mapped back to the image space at the same time to obtain the rain residual image A corresponding to the low-frequency component and the diagonal component R and D R , and subtract the corresponding rain residual image from the original low-frequency component and diagonal component to obtain the rain-removed background A corresponding to the low-frequency component and diagonal component B and DB ; At the same time, the de-raining background features of the horizontal component and the vertical component in step 5 are mapped back to the image space to obtain the corresponding de-raining background image H of the horizontal component and the vertical component B and V B ;

[0068] In this embodiment, two 3×3 convolutions are performed from f H,AD,R,n The rain information of low-frequency and diagonal components is extracted respectively, and remapped back to the image space to obtain the rain residual image A corresponding to the wavelet component. R and D R , the specific formula used is:

[0069] A R =F A,3 (f H,AD,R,n );D R =F D,3 (f H,AD,R,n );

[0070] Among them, F A,3 (·) and F D,3 (·) represent two 3×3 convolutions. Then, the corresponding rain information is subtracted from the original low-frequency and diagonal initialization features to obtain the rain-removed background features of the corresponding low-frequency and diagonal components. The specific formula is:

[0071]

[0072] At the same time, the background features f of the horizontal and numerical components are transformed through two 3×3 convolutions. H,B,n and f V,B,n Remap back to the image space to obtain the rain-free background image H corresponding to the horizontal and vertical components B and V B , the specific formula used is:

[0073] H B =F H,3 (f H,B,n );V B =F V,3 (f V,B,n ).

[0074] Step 7: Take the A obtained in step 6 B , D B , H B and V B Perform inverse wavelet transform to obtain a clean background image without rain.

[0075] In this embodiment, the wavelet background images of four corresponding frequencies are inversely transformed to output a clean rain-free image. Specific formula used:

[0076]

[0077] The wavelet-based direction-aware attention image deraining network used in this embodiment is a trained wavelet-based direction-aware attention image deraining network; its training process includes the following steps:

[0078] (1) Collect and construct a paired rain image dataset, including training data and test data; select part of the rain image data Y and crop it to obtain an M×M image block as a training sample; where M is a preset value;

[0079] (2) The training sample I Rain Decomposed into low frequency components A Rain , horizontal component H Rain , vertical component V Rain and the diagonal component D Rain ;

[0080] (3) Input the four-component data into the initial embedding expression layer and extract the vertical component V through convolution Rain and the horizontal component H Rain The initial feature f V,R and f H,R , and at the same time the low frequency component A Rain and the diagonal component D Rain Perform cascade combination and extract the mixed initial feature f through convolution AD,R ;

[0081] (4) The initial feature f in (3) AD,R 、f V,R and f H,R Input to the direction-aware attention module, through a channel attention module from f AD,R The mixed rain information is extracted from the image, and based on the vector decomposition theory, the direction-aware attention unit is used to distill the horizontal rain information in a specific direction from the extracted mixed rain information. H,AD,R and vertical rain information f V,AD,R ;

[0082] (5) From the initial feature f H,R and f V,R Subtract the corresponding horizontal rain information f obtained in (4) from H,AD,R and vertical rain information f V,AD,R , obtain the rain removal background features f corresponding to the horizontal and vertical components H,B and f V,B ;

[0083] (6) Loop (4) and (5) N times to gradually extract the high-precision low-frequency component A Rain and the diagonal component DRain The mixed rainwater information is obtained, and the clean background features of the corresponding horizontal and vertical components are gradually obtained; wherein N is a preset value;

[0084] (7) Substitute the low-frequency component A in (6) Rain and the diagonal component D Rain The mixed rainwater information is separated to obtain the corresponding low-frequency component A Rain and the diagonal component D Rain The rain information is mapped back to the image space at the same time to obtain the rain residual image A corresponding to the low-frequency component and the diagonal component R and D R , and subtract the corresponding rain residual image from the original low-frequency component and diagonal component to obtain the rain-removed background A corresponding to the low-frequency component and diagonal component B and D B ; At the same time, the de-raining background features of the horizontal and vertical components in (6) are mapped back to the image space to obtain the corresponding de-raining background image H of the horizontal and vertical components B and V B ;

[0085] (8) Substitute the A obtained in (7) B , D B , H B and V B Perform inverse wavelet transform to obtain a clean background image without rain;

[0086] (9) By minimizing the consistency between the rain-free image output in (8) and the original clean image in the image space and the brightness and color space, the wavelet-based direction-aware attention image deraining network is optimized to obtain a trained wavelet-based direction-aware attention image deraining network;

[0087] Among them, the reconstructed rain-free image is constrained by the Charbonnier loss function. and the original clean image I B Maintain consistent statistical distribution, and introduce weight gradient constraints in color brightness space to ensure the naturalness of generated texture colors;

[0088] Specific use function:

[0089]

[0090] Among them, L(θ) and θ represent the optimization objective equation and parameter respectively. and C B Reconstructed rain-free images and the original clean image I B The color channel, Δ gw =(1+wΔ x)(1+wΔ y ), where Δ x and Δ y express and C B The absolute value difference of the gradient in the horizontal and vertical directions, compensation factor ε = 10 -3 .

[0091] Compared with the early rain removal methods based on deep learning and additive models, this paper combines wavelet transform and vector decomposition to realize rain removal and background texture reconstruction in the frequency domain. At the same time, through vector decomposition, a direction-aware attention learning mechanism is introduced to focus on feature learning in specific directions, achieving high-precision and high-quality background restoration effects.

[0092] This method can remove rain streaks while protecting image quality to the greatest extent.

[0093] The present invention outperforms the current best algorithm MPRNet[1] by an average of 0.23 dB on four benchmark rainy day data sets. Please refer to Appendix 1.

[0094] Table 1

[0095] method Test100 Test1200 R100H R100L average value MPRNet[1] 30.27 32.91 30.41 36.40 32.49 This method 30.58 33.64 30.17 36.49 32.72

[0096] [1]Zamir SW, Arora A, Khan S, et al.Multi-stage progressive imagerestoration[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition.2021:14821-14831.

[0097] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A wavelet-based direction-aware attention image deraining method, characterized by: A wavelet-based direction-aware attention image deraining network is used to remove rainy images. The wavelet-based direction-aware attention image deraining network includes an initial embedding expression layer, a plurality of direction-aware attention modules and a wavelet reconstruction module; The initial embedding expression layer uses three parallel 3×3 convolutions to respectively Rain , vertical component V Rain The combined features of low-frequency and diagonal components are embedded and expressed, and then the embedded expression features are input into the direction-aware attention module to express the features of a specific direction; The direction perception attention module includes a residual channel attention module, a horizontal direction perception attention unit and a vertical direction perception attention unit; firstly, the embedded features of the low-frequency and diagonal components are deeply extracted through a residual channel attention module to obtain the corresponding rain feature expression; Then, a horizontal direction perception attention unit is used to separate and extract the horizontal rain component from the low-frequency and diagonal rain feature expressions, and a vertical direction perception attention unit is used to separate and extract the vertical rain component from the low-frequency and diagonal rain feature expressions, and the corresponding horizontal and vertical embedding expressions are updated to obtain the rain removal background feature; Then, the corresponding horizontal de-raining background features, vertical de-raining background features, and low-frequency and diagonal rain features are input into the next direction-aware attention module for further feature expression. By stacking several direction-aware attention modules, clean horizontal and vertical background features, as well as low-frequency and diagonal rain expressions are finally obtained. The wavelet reconstruction module projects the wavelet features in the specific directions into the image space through four parallel 3×3 convolutions to obtain corresponding wavelet components; The method comprises the following steps: Step 1: Take the rain image I Rain Decomposed into low frequency components A Rain , horizontal component H Rain , vertical component V Rain and the diagonal component D Rain ; Step 2: Input the four-component data into the initial embedding expression layer and extract the vertical component V through convolution Rain and the horizontal component H Rain The initial feature f V，R and f H，R , and at the same time the low frequency component A Rain and the diagonal component D Rain Perform cascade combination and extract the mixed initial feature f through convolution AD，R ; Step 3: The initial feature f in step 2 AD，R 、f V，R and f H，R Input to the direction-aware attention module, through a channel attention module from f AD，R The mixed rain information is extracted from the horizontal direction perception attention unit and the vertical direction perception attention unit are used to respectively distill the horizontal rain information in a specific direction from the extracted mixed rain information. H，AD，R and vertical rain information f V，AD，R ; Step 4: From the initial feature f H，R and f V，R Subtract the corresponding horizontal rain information f obtained in step 3 from H，AD，R and vertical rain information f V，AD，R , obtain the rain removal background features f corresponding to the horizontal and vertical components H，B and f V，B ; Step 5: Loop through steps 3 and 4 N times to progressively extract the high-precision low-frequency component A. Rain and the diagonal component D Rain The mixed rainwater information gradually obtains the clean background features of the corresponding horizontal and vertical components; where N is a preset value; Step 6: Convert the low frequency component A in step 5 Rain and the diagonal component D Rain The mixed rainwater information is separated to obtain the corresponding low-frequency component A Rain and the diagonal component D Rain The rain information is mapped back to the image space at the same time to obtain the rain residual image A corresponding to the low-frequency component and the diagonal component R and D R , and subtract the corresponding rain residual image from the original low-frequency component and diagonal component to obtain the rain-removed background A corresponding to the low-frequency component and diagonal component B and D B ; At the same time, the de-raining background features of the horizontal component and the vertical component in step 5 are mapped back to the image space to obtain the corresponding de-raining background image H of the horizontal component and the vertical component B and V B ; Step 7: Take the A obtained in step 6 B , D B , H B and V B Perform inverse wavelet transform to obtain a clean background image without rain.

2. The wavelet-based direction-aware attention image deraining method according to claim 1, characterized in that: The residual channel attention module described in step 1 is composed of two 3×3 convolutional layers with a channel of 48, a global average pooling layer, two 1×1 convolutional layers with a channel of 12, and a Sigmoid function layer connected in series; the output of the second 3×3 convolutional layer with a channel of 48 is multiplied pixel by pixel with the output of the Sigmoid function layer, and then summed pixel by pixel with the original input and output.

3. The wavelet-based direction-aware attention image deraining method according to claim 1, characterized in that: The horizontal direction perception attention unit and the vertical direction perception attention unit in step 1 are both composed of a directional average pooling layer, a 1×1 convolution layer with 8 channels, a nonlinear mapping layer, a 1×1 convolution layer with 64 channels, a Sigmoid function layer, and a depth-separated residual channel attention unit connected in series; the output of the Sigmoid function layer is dot-multiplied with the original input pixel by pixel and then sent to the depth-separated residual channel attention unit; the output of the depth-separated residual channel attention unit is dot-multiplied with the original input pixel by pixel and then output; wherein, the degree-separated residual channel attention unit is a unit obtained by replacing the 3x3 convolution in the residual channel attention module with a depth-separable convolution.

4. The wavelet-based direction-aware attention image deraining method according to claim 1, characterized in that: In step 1, the rain image I to be processed is transformed by using Haar wavelet transform. Rain Decomposed into low frequency components A Rain , horizontal component H Rain , vertical component V Rain and the diagonal component D Rain ; A Rain ,H Rain ,V Rain ,D Rain =Hair(I Rain ); Here, Haar(·) represents Haar wavelet transform.

5. The wavelet-based direction-aware attention image deraining method according to claim 1, characterized in that: In step 2, the four-component data is input into the initial embedding expression layer, and the wavelet component of the rainy image is mapped from the image space to the feature space to obtain the initial feature f of the corresponding component. AD，R , f H，R , f V，R ; f AD，R =F AD，R (A Rain ,D Rain );f H，R =F H，R (H Rain );f V，R =f V，R (V Rain )。 6. The wavelet-based direction-aware attention image deraining method according to claim 1, characterized in that: In step 6, the low-frequency component A in step 5 is transformed into Rain and the diagonal component D Rain Mixed rainwater information H，AD，R，n Separate and obtain the corresponding low-frequency component A Rain and the diagonal component D Rain Rainwater information.

7. The wavelet-based direction-aware attention image deraining method according to claim 1, characterized in that: In step 6, the de-raining background features of the horizontal and vertical components in step 5 are mapped back to the image space through two 3×3 convolutions.

8. The wavelet-based direction-aware attention image deraining method according to any one of claims 1 to 7, characterized in that: The wavelet-based direction-aware attention image deraining network is a trained wavelet-based direction-aware attention image deraining network; The training process includes the following steps: (1) Collect and construct a paired rain image dataset, including training data and test data; select part of the rain image data Y and crop it to obtain an M×M image block as a training sample; where M is a preset value; (2) The training sample I Rain Decomposed into low frequency components A Rain , horizontal component H Rain , vertical component V Rain and the diagonal component D Rain ; (3) Input the four-component data into the initial embedding expression layer and extract the vertical component V through convolution Rain and the horizontal component H Rain The initial feature f V，R and f H，R , and at the same time the low frequency component A Rain and the diagonal component D Rain Perform cascade combination and extract the mixed initial feature f through convolution AD，R ; (4) The initial feature f in (3) AD，R 、f V，R and f H，R Input to the direction-aware attention module, through a channel attention module from f AD，R The mixed rain information is extracted from the image, and the direction-aware attention unit is used to distill the horizontal rain information in a specific direction from the extracted mixed rain information. H，AD，R and vertical rain information f V，AD，R ; (5) From the initial feature f H，R and f V，R Subtract the corresponding horizontal rain information f obtained in (4) from H，AD，R and vertical rain information f V，AD，R , obtain the rain removal background features f corresponding to the horizontal and vertical components H，B and f V，B ; (6) Loop (4) and (5) N times to gradually extract the high-precision low-frequency component A Rain and the diagonal component D Rain The mixed rainwater information gradually obtains the clean background features of the corresponding horizontal and vertical components; where N is a preset value; (7) Substitute the low-frequency component A in (6) Rain and the diagonal component D Rain The mixed rainwater information is separated to obtain the corresponding low-frequency component A Rain and the diagonal component D Rain The rain information is mapped back to the image space at the same time to obtain the rain residual image A corresponding to the low-frequency component and the diagonal component R and D R , and subtract the corresponding rain residual image from the original low-frequency component and diagonal component to obtain the rain-removed background A corresponding to the low-frequency component and diagonal component B and D B ; At the same time, the de-raining background features of the horizontal and vertical components in (6) are mapped back to the image space to obtain the corresponding de-raining background image H of the horizontal and vertical components B and V B ; (8) Substitute the A obtained in (7) B , D B , H B and V B Perform inverse wavelet transform to obtain a clean background image without rain; (9) optimizing the wavelet-based direction-aware attention image deraining network by minimizing the consistency between the rain-free image output in (8) and the original clean image in the image space and the brightness and color space, thereby obtaining a trained wavelet-based direction-aware attention image deraining network; Among them, the reconstructed rain-free image is constrained by the Charbonnier loss function. and the original clean image I B Maintain consistent statistical distribution, and introduce weight gradient constraints in color brightness space to ensure the naturalness of generated texture colors; Specific use function: Among them, L(θ) and θ represent the optimization objective equation and parameter respectively. and C B Reconstructed rain-free images and the original clean image I B The color channel, Δ gw =(1+wΔ x )(1+wΔ y ), where Δ x and Δ y express and C B The absolute value difference of the gradient in the horizontal and vertical directions, compensation factor ε = 10 -3 .

9. A wavelet-based direction-aware attention image deraining system, characterized by: A wavelet-based direction-aware attention image deraining network is used to remove rainy images. The wavelet-based direction-aware attention image deraining network includes an initial embedding expression layer, a plurality of direction-aware attention modules and a wavelet reconstruction module; The initial embedding expression layer uses three parallel 3×3 convolutions to respectively Rain , vertical component V Rain The combined features of low-frequency and diagonal components are embedded and expressed, and then the embedded expression features are input into the direction-aware attention module to express the features of a specific direction; The direction perception attention module includes a residual channel attention module, a horizontal direction perception attention unit and a vertical direction perception attention unit; firstly, the embedded features of the low-frequency and diagonal components are deeply extracted through a residual channel attention module to obtain the corresponding rain feature expression; Then, a horizontal direction perception attention unit is used to separate and extract the horizontal rain component from the low-frequency and diagonal rain feature expressions, and a vertical direction perception attention unit is used to separate and extract the vertical rain component from the low-frequency and diagonal rain feature expressions, and the corresponding horizontal and vertical embedding expressions are updated to obtain the rain removal background feature; Then, the corresponding horizontal de-raining background features, vertical de-raining background features, and low-frequency and diagonal rain features are input into the next direction-aware attention module for further feature expression. By stacking several direction-aware attention modules, clean horizontal and vertical background features, as well as low-frequency and diagonal rain expressions are finally obtained. The wavelet reconstruction module projects the wavelet features in the specific directions into the image space through four parallel 3×3 convolutions to obtain corresponding wavelet components; The system includes the following modules: Module 1 is used to process the rain image I Rain Decomposed into low frequency components A Rain , horizontal component H Rain , vertical component V Rain and the diagonal component D Rain ; Module 2 is used to input the four-component data into the initial embedding expression layer and extract the vertical component V through convolution Rain and the horizontal component H Rain The initial feature f V，R and f H，R , and at the same time the low frequency component A Rain and the diagonal component D Rain Perform cascade combination and extract the mixed initial feature f through convolution AD，R ; Module 3 is used to convert the initial feature f in module 2 AD，R 、f V，R and f H，R Input to the direction-aware attention module, through a channel attention module from f AD，R The mixed rain information is extracted from the horizontal direction perception attention unit and the vertical direction perception attention unit are used to respectively distill the horizontal rain information in a specific direction from the extracted mixed rain information. H，AD，R and vertical rain information f V，AD，R ; Module 4 is used to obtain the initial feature f H，R and f V，R Subtract the corresponding horizontal rain information f obtained in module 3 H，AD，R and vertical rain information f V，AD，R , obtain the rain removal background features f corresponding to the horizontal and vertical components H，B and f V，B ; Module 5 is used to execute module 3 and step module 4 N times in a loop, and gradually extract the high-precision low-frequency component A. Rain and the diagonal component D Rain The mixed rainwater information gradually obtains the clean background features of the corresponding horizontal and vertical components; where N is a preset value; Module 6 is used to convert the low frequency component A in module 5 Rain and the diagonal component D Rain The mixed rainwater information is separated to obtain the corresponding low-frequency component A Rain and the diagonal component D Rain The rain information is mapped back to the image space at the same time to obtain the rain residual image A corresponding to the low-frequency component and the diagonal component R and D R , and subtract the corresponding rain residual image from the original low-frequency component and diagonal component to obtain the rain-removed background A corresponding to the low-frequency component and diagonal component B and D B At the same time, the rain-removed background features of the horizontal component and the vertical component in module 5 are mapped back to the image space to obtain the corresponding rain-removed background image H of the horizontal component and the vertical component B and V B ; Module 7 is used to convert the A obtained in module 6 into B , D B , H B and V B Perform inverse wavelet transform to obtain a clean background image without rain.

10. A wavelet-based direction-aware attention image deraining device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the wavelet-based direction-aware attention image deraining method as described in any one of claims 1 to 8.

Images