Image rain removal method based on space-frequency double-domain perception

Abstract

The invention relates to the technical field of image processing, and provides an image rain removal method based on space-frequency double-domain perception, which comprises the following steps of: 1, constructing an image rain removal network based on space-frequency double-domain perception; 2, training the image rain removal network model in the step 1 until the model converges; and step 3, for any acquired rainy day scene image, reasoning by using the rain removal network trained in the step 2 to obtain a rain-free background image. Compared with the prior art, the rain-removed image generated by the method has fewer rain stripe residues and clearer texture features, and the problems of rain stripe residues, blurred background and the like in an existing image rain-removing method can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image rain removal method based on space-frequency dual-domain perception. Background Art

[0002] Image deraining technology aims to remove rain streaks, raindrops and other noise from rainy images through algorithms to restore the clarity and authenticity of the image. This is of great significance in the fields of autonomous driving, video surveillance, satellite image analysis, etc. In these applications, images acquired under rainy conditions are often disturbed by rain, which affects the data quality of the sensor and reduces the effectiveness of subsequent analysis. Therefore, image deraining technology can not only improve the quality of the image itself, but also effectively improve the performance of the image-based intelligent decision-making system.

[0003] In the process of studying the image deraining problem, early methods mainly relied on artificially designed prior knowledge. These methods removed rain streaks by integrating a variety of prior information reflecting rain streaks and clear image features. For example, some methods try to predict and remove rain streaks based on physical models by analyzing the optical properties of raindrops or rain streaks in images and combining reflection models; while other methods rely on image processing techniques, such as edge detection and texture analysis, to extract features related to rain streaks in images and remove them through filtering, image decomposition, etc. Although these methods have made progress to a certain extent, they rely on artificially designed prior knowledge, and for complex rainy scenes, they are often difficult to adapt to the changing environment and different types of rain interference. Therefore, these traditional methods often show limitations when facing complex rainy conditions, and cannot effectively deal with strong background changes and multiple rain streak patterns, resulting in unsatisfactory rain removal effects.

[0004] With the continuous development of technologies such as convolutional neural networks (CNN), image deraining technology has been significantly improved. In the early deep learning methods, convolutional neural networks were widely used in image deraining tasks. CNN can effectively extract local features in images and gradually capture deeper rain streaks and background information through a multi-layer network structure. This enables CNN-based deraining methods to better handle relatively simple rainy scenes. However, with the deepening of research, the single CNN structure began to expose limitations, especially when dealing with complex scenes, the deraining effect is still insufficient, and the Transformer can simultaneously process the global information and local details of the image through the self-attention mechanism, especially when dealing with scenes with complex backgrounds and strong rain interference, showing its advantages. The Transformer-based deraining method can learn the deeper relationship between rain streaks and background through global context information modeling, avoiding the blurring or artifacts commonly seen in traditional methods. Although Transformer can capture global context information well, it may not be as flexible as CNN in extracting local detail features, especially in removing the subtle structure, texture information and local rain streaks of the image. Transformer often relies on global context, which may lead to insufficient processing of local features. Due to the respective advantages and limitations of CNN and Transformer, existing image deraining methods usually face a trade-off between global information and local feature modeling. Therefore, how to remove interference information such as rain streaks in the image while retaining important details of the image has become a key issue in the field of image deraining. Summary of the invention

[0005] In view of the above problems existing in the field of image deraining, the present invention provides an image deraining method based on space-frequency dual-domain perception. The method performs information fusion in the deraining process by simultaneously utilizing feature information in the spatial domain and the frequency domain. Specifically, the spatial domain is used to capture the global semantic structure of the image, while the frequency domain can effectively suppress the frequency noise caused by raindrops and rain streaks. Through the deep fusion of the global semantic information and local structural information of the image, the present invention can maintain the clear texture and details of the image while removing raindrops and rain streaks, thereby significantly improving the deraining effect.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] An image deraining method based on space-frequency dual-domain perception, the method comprising the following steps:

[0008] Step 1: Construct an image deraining network based on space-frequency dual-domain perception;

[0009] Step 2: Train the image deraining network model described in step 1 until the model converges;

[0010] Step 3: For any rainy scene image collected, use the deraining network trained in step 2 to perform inference to obtain a rain-free background image.

[0011] Furthermore, the image deraining network based on space-frequency dual-domain perception in step 1 adopts a symmetric encoder-decoder architecture and performs Multiple downsampling is performed and added to the encoder input of each layer. This residual connection can retain the low-level information of the input image and enhance the ability to restore details.

[0012] Furthermore, the specific structures of the encoder and decoder are:

[0013] The encoder uses a space-frequency attention module and a frequency-modulated feedforward network to extract and compress image features and identify and remove rain interference;

[0014] The decoder also uses the space-frequency attention module and the frequency-modulated feedforward network to restore the resolution and details of the image. At the same time, it uses the information of the encoder to perform rain removal and output the final rain-free image I derain .

[0015] Furthermore, the space-frequency attention module includes layer normalization, convolutional layer, high-frequency guidance module, frequency enhancement module, channel attention module and fusion module. First, the output of the layer normalization is not only connected to the input of the high-frequency guidance module, but also to the input of the convolutional layer for generating key and value mapping. Secondly, the output of the high-frequency guidance module is connected to the input of the convolutional layer for generating query mapping. In addition, the outputs of all convolutional layers are connected to the input of the channel attention module. The output of the convolutional layer for generating value mapping is also connected to the input of the frequency enhancement module; the frequency enhancement module can capture and integrate information at different scales by adopting two frequency enhancement modules of different scales, thereby constructing a richer and more refined local feature representation. The fusion module aims to integrate the output features of the multi-scale frequency enhancement module and the channel attention module, including element-by-element addition and convolution application. Element-by-element addition is used to fuse features, and convolution realizes cross-channel information interaction. Specifically, the output of the frequency enhancement module is added element by element, and then deep interaction is performed through convolution. Then, it is added element by element with the output of the channel attention module to enrich the feature expression. Finally, the fusion is completed through convolution mapping comprehensive features.

[0016] Furthermore, the frequency enhancement module performs the following operations:

[0017] F low =AvgPool(F)

[0018] F high =FF low

[0019] F out =W1F low +W2F high

[0020] Among them, AvgPool is the average pooling operation, F represents the input feature, and F low and F high Represent the low-frequency and high-frequency components of the input features, W1 and W2 represent linear projections, and F out The above operations are performed in the horizontal and vertical directions in sequence. The results of the operations in the horizontal direction are used as the input of the operations in the vertical direction. The output in the vertical direction is added to the input feature F to obtain the final feature containing rich and compact context information.

[0021] Furthermore, the frequency-controlled feedforward network includes layer normalization, Fourier transform, convolution mapping, inverse transform and gated convolution from front to back.

[0022] Furthermore, the downsampling of the input image adopts a nearest neighbor downsampling mode, and the input image downsampling features are fused with the deep encoder input using convolution operations and element-by-element addition operations.

[0023] Furthermore, the loss function used in the training in step 2 is L1 loss, and its description formula is as follows:

[0024]

[0025] Among them, I derain represents the output result of the network described in step 1, I gt A reference image representing no rain.

[0026] Furthermore, the arbitrary rainy scene images in step 3 include images under natural conditions, and also include images that have been subjected to technical means such as scaling, cropping and transformation.

[0027] Compared with the prior art, the present invention has the following advantages:

[0028] (1) The present invention proposes an image deraining method based on space-frequency dual-domain perception. To address the background blur and structural inconsistency problems in existing image deraining methods, the present invention uses high-frequency channel attention combined with frequency domain information to fully extract global semantic information and local texture detail features, thereby achieving effective processing of rainy images and improving the quality of reconstructed images.

[0029] (2) The present invention proposes a spatial-frequency self-attention module (SFSA), which adds a high-frequency guidance module (HFGM) and a frequency enhancement module (FEM) on the basis of the channel attention module (CAM). This module enables the attention module to improve its modeling ability for high-frequency information and extract contextual information in local areas to compensate for the local blur caused by rain streaks, thereby further improving the structural integrity and detail fidelity of the derained image.

[0030] (3) The present invention uses a frequency control feed-forward network (FCFN) for feature refinement. Based on the frequency selection strategy, the influence of frequency information with weak correlation on the overall features is weakened, and the core frequency information is focused on, which can better capture the essential features and improve the generalization and anti-interference capabilities of the model. In addition, the present invention also adds a residual connection to the input image to supplement the details that may be lost in the encoder features and improve the reconstruction quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings involved in the embodiments. The drawings are only some embodiments of the present invention and should not be understood as limiting the present invention in any way. For those of ordinary skill in the art, other drawings can also be obtained based on the provided drawings without creative work.

[0032] Figure 1 is a schematic flow chart of an image rain removal method based on space-frequency dual-domain perception in an embodiment of the present invention;

[0033] Figure 2 is a schematic diagram of the structure of an image deraining network based on space-frequency dual-domain perception in an embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of the structure of a space-frequency attention module in an embodiment of the present invention;

[0035] Figure 4 is a schematic structural diagram of a high-frequency guiding module in an embodiment of the present invention;

[0036] Figure 5 is a structural schematic diagram of a frequency enhancement module in an embodiment of the present invention;

[0037] Figure 6is a schematic diagram of the structure of a frequency control feedforward network in an embodiment of the present invention;

[0038] Figure 7 is a rain removal effect diagram after processing a rainy day image in an embodiment of the present invention;

[0039] Figure 8 It is a comparison diagram of the effects of the image deraining method in the embodiment of the present invention and other image deraining methods. DETAILED DESCRIPTION

[0040] In order to more clearly show the technical details and advantages of the present invention, the following will be a comprehensive and complete description of its technical solution through the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the present invention, mainly used to explain the technical concept of the present invention, and do not limit other related contents of the present invention.

[0041] Figure 1 This is a flow chart of the image deraining method based on space-frequency dual-domain perception proposed by the present invention, and the method comprises the following steps:

[0042] Step 1: Construct an image deraining network based on space-frequency dual-domain perception.

[0043] Figure 2 This is a model structure diagram of the image deraining network based on space-frequency dual-domain perception proposed in the present invention. The designed image deraining network first receives an enhanced rainy day image of a specified size as input. After network processing, a rain-free image is directly generated. The network adopts a 4-layer symmetric encoder-decoder architecture with a sophisticated structural design. Among them, the encoder part uses the SFSA module, which is responsible for extracting and compressing important features of the image; the decoder part also uses the SFSA module, focusing on restoring the details and resolution of the image, thereby outputting a high-quality derained image.

[0044] like Figure 2 As shown, the present invention is specifically implemented as follows: first, the network receives a rainy day image I of a specified size rain as input. In order to further process the image information, the image is downsampled to generate two images with half and one-quarter the size of the original image respectively. These downsampled images are integrated into the deep network through the skip connection method to supplement and enrich the feature information of the network. In the image feature extraction stage, a convolution operation with a kernel size of 3×3 is selected to efficiently capture the effective information and feature details in the input image. The extracted initial features are then input into the encoder for deeper feature extraction. The encoder structure is designed in a hierarchical manner, and each layer contains a different number of SFSA modules, namely 4, 6, 6 and 8. Through this design combined with the downsampling operation, the multi-scale processing capability of image features is realized. Figure 3 It is a schematic diagram of the structure of the space-frequency attention module, such as Figure 3 As shown in Figure 1, the SFSA module uses a channel attention mechanism to efficiently capture the global semantic and structural features of the image. In order to further improve the sensitivity to high-frequency information, the query input will be processed by the HFGM module. This process significantly enhances the attention mechanism's ability to capture high-frequency components of the image. The HFGM module structure is shown in Figure 1. Figure 4 As shown in Figure 1. In order to better restore the local blur caused by rain streaks, the SFSA module cleverly utilizes the contextual information of the local area. In specific implementation, the value input is processed at multiple scales through FEM modules with kernel sizes of 3×3 and 5×5. The FEM module structure is as follows: Figure 5 As shown. This process ensures that the model can make full use of local spatial information to improve the restoration of image details. Finally, the output of the frequency enhancement module is added element by element, and then deeply interacted through convolution. Then, it is added element by element with the output of the channel attention module to enrich the feature expression. Finally, the output feature is generated through convolution mapping. The overall process of the SFSA module can be simply expressed as:

[0045] F s =CAM(HFGM(Q),K,V)

[0046] F f =Conv(FEM3(V)+FEM5(V))

[0047] F out =Conv(F s +F f )

[0048] Among them, CAM is the channel attention mechanism, Conv is the convolution module, FEM3 and FEM5 represent the frequency enhancement modules with internal kernel sizes of 3 and 5. Q, K and V represent the query, key and value mappings for the channel attention mechanism respectively. These mappings are generated by the input features through layer normalization and convolution layers in sequence. Before passing through the convolution layer, Q is also processed by the HFGM module. s Focuses on extracting the global semantic information of the image, F f Focuses on strengthening the local detail features of the image. The addition of the two can achieve feature complementarity. out Represents the characteristics of the final output.

[0049] Since the high-frequency components in the image usually contain rain streak interference, while the low-frequency components reflect the overall structure of the image, by weighting different frequency components, rain streak interference can be effectively removed while retaining the details and background information of the image. The present invention adopts the FCFN module, which can perform detailed feature representation for different frequency information by converting spatial features into the frequency domain. Figure 6 This is a schematic diagram of the structure of the FCFN module. Figure 6 As shown in the figure, FCFN converts spatial features into frequency domain features through two-dimensional discrete Fourier transform, and uses convolution mapping to give different weights to frequency components, so that the most important frequency information can be accurately selected and strengthened. Finally, it is reversely transformed through inverse transform and features are aggregated using gated convolution to obtain compact and rich feature representation. The decoder and encoder are designed to have a symmetrical structure, mainly composed of SFSA modules and upsampling operations. This symmetrical structure ensures the effective restoration and rich expression of features in the decoding stage. In addition, the jump connection directly transfers the low-level features extracted from the encoder to the corresponding level of the decoder. This design not only allows the decoder to make full use of the rich semantic information transmitted by the encoder, but also combines the deep features learned by the decoder itself to achieve a more accurate image reconstruction process. Finally, the features are mapped to the distribution space of the real image through convolution with a kernel size of 3×3, and added to the original input image to obtain the derained image I derain .

[0050] Step 2: Train the image deraining network model described in step 1 until the model converges.

[0051] The data used in the present invention are derived from four different data sets: Rain200H, Rain200L, DID-Data and DDN-Data. The resolution of all images from these data sets is uniformly set to 128×128 pixels. For images with a resolution lower than 128×128, the image is expanded using a mirror reflection filling mode; for images with a resolution higher than 128×128, the required image parts are extracted by random cropping. On this basis, data enhancement techniques are further applied to these images, including random rotation and flipping operations. The rotation angles are set to 90 degrees, 180 degrees and 270 degrees, and the images are flipped up and down. Such processing helps to improve the diversity of the data set while ensuring consistency and comparability between images.

[0052] The embodiments of the present invention are developed based on the PyTorch framework and trained on two NVIDIAGeForce RTX 3090 graphics cards. In order to optimize the model performance, the AdamW optimizer is used, in which the parameters β1 and β2 are set to 0.9 and 0.999, respectively. The entire training process includes 300,000 iterations. In the initial 92,000 iterations, the learning rate is fixed to 3e-4; and in the subsequent 208,000 iterations, the cosine annealing strategy is used to gradually reduce the learning rate to 1e-6. In order to speed up the training, the batch size is set to 6. The parameter weights of the model are optimized by the L1 loss function, and its description formula is as follows:

[0053]

[0054] Among them, I derain represents the output result of the network described in step 1, I gt A reference image representing no rain.

[0055] Step 3: For any rainy scene image collected, use the deraining network trained in step 2 to perform inference to obtain a rain-free background image.

[0056] Figure 7 This is a diagram showing the effect of rainy day image processing provided by the present invention.

[0057] like Figure 7 The figure shows the effect of image deraining network processing applied to the Rain200H dataset. There is obvious rain streak interference in the original image, and after being processed by the trained deraining network, the reconstructed image successfully removes the rain streaks while retaining and restoring the details and background in the image. Specifically, after processing, the tree trunks in the image appear clear and have rich texture, while the glass surface of the building retains the original stripe details. This result further verifies the effectiveness and superior performance of the image deraining method based on space-frequency dual-domain perception proposed in the present invention.

[0058] Figure 8 This is a comparison chart of the effects of the image deraining method proposed in the present invention and other image deraining methods.

[0059] Figure 8 (a) is a rainy day image in the Rain200H dataset. Figure 8 (b) is the deraining effect of the Dual GCN method. Figure 8 (c) is the deraining effect of the SPDNet method. Figure 8 (d) is the rain removal effect of the IDT method. Figure 8 (e) is the rain removal effect of the Restormer method. Figure 8 (f) is the deraining effect of the DRSformer method. Figure 8 (g) is the rain removal effect of the method proposed by the present invention, Figure 8 (h) Figure 8(a) The real rain-free image corresponding to the rainy day image. From the comparison of the visualization results, it can be seen that although the Dual GCN and SPDNet methods have achieved certain success in removing the rain marks on the boy's chin, they are still insufficient in restoring the natural appearance of the image. This method may cause the boy's facial features to appear less realistic and natural. In contrast, IDT, Restormer and DRSformer perform better in overall image reconstruction, but sometimes cause unnatural white spots to appear in certain areas of the skin. The present invention focuses on accurately restoring image details, especially in complex areas such as the chin, making the image look more natural and realistic. By utilizing spatial and frequency domain information, the appearance of white spots and unnatural phenomena is effectively avoided. This not only makes the boy's face more delicate and smooth, but also improves the overall visual effect, significantly improves the problems that previous methods could not solve, and ensures high-quality reconstruction of the final image.

[0060] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although these embodiments are described in detail, it should be understood by those skilled in the art that they can still modify the technical solutions or replace some technical features by equivalents. These modifications or replacements will not cause the essence of the technical solutions to deviate from the spirit and scope of the embodiments of the present invention.

Claims

1. An image deraining method based on space-frequency dual-domain perception, characterized in that: The following steps are involved: Step 1: Construct an image deraining network based on space-frequency dual-domain perception. The image deraining network uses a space-frequency attention module and a frequency-regulated feedforward network as the internal structure of the encoder and decoder. The space-frequency attention module effectively captures and enhances rain streak features while maintaining background details. The control feedforward network can accurately extract rain streak information and reduce background interference by separating and modeling rain streak features in the frequency domain; In addition, the downsampled features of the input image are fused with the deep encoder input to better preserve the original information; Step 2: Train the image deraining network model described in step 1 until the model converges; Step 3: For any rainy scene image collected, use the deraining network trained in step 2 to perform inference to obtain a rain-free background image.

2. The image deraining method based on space-frequency dual-domain perception according to claim 1 is characterized in that: The spatial-frequency self-attention module (SFSA) includes layer normalization, convolutional layer, high-frequency guidance module (HFGM), frequency enhancement module (FEM), channel attention module (CAM) and fusion module; The output of the layer normalization is connected not only to the input of the high-frequency guidance module, but also to the input of the convolutional layer for generating key and value mappings; The output of the high frequency guidance module is connected to the input of the convolutional layer for generating the query map; The output of the convolutional layer is connected to the input of the channel attention module, and the output of the convolutional layer for generating the value map is also connected to the input of the frequency enhancement module; The frequency enhancement module is designed to enrich the feature representation by introducing local context information. By adopting two frequency enhancement modules of different scales, it is possible to capture and integrate information at different scales, thereby constructing a richer and more refined local feature representation. The channel attention module performs weighting between different channels of the feature map, thereby improving the model's ability to focus on key information, enhancing the ability to express important features, and suppressing redundant information; The fusion module is designed to integrate the output features of the multi-scale frequency enhancement module and the channel attention module, including element-by-element addition and convolution application. Element-by-element addition is used to fuse features, and convolution realizes cross-channel information interaction. First, the output of the frequency enhancement module is added element by element, and then cross-channel interaction is performed through convolution. Then, it is added element by element with the output of the channel attention module to enrich the feature expression. Finally, the fusion is completed through convolution mapping. The overall process of the space-frequency attention module can be expressed by the following formula: F s =CAM(HFGM(Q),K,V) F f =Conv(FEM3(V)+FEM5(V)) F out =Conv(F s +F f ) Among them, CAM is the channel attention mechanism, Conv is the convolution module, FEM3 and FEM5 represent the frequency enhancement modules with internal kernel sizes of 3 and 5; Q, K and V represent the query, key and value mappings for the channel attention mechanism respectively, which are generated by the input features through layer normalization and convolution layers in sequence. Before passing through the convolution layer, Q is also processed by the HFGM module; F s Focuses on extracting the global semantic information of the image, F f Focuses on strengthening the local detail features of the image. The addition of the two can achieve feature complementarity. out Represents the characteristics of the final output.

3. The image deraining method based on space-frequency dual-domain perception according to claim 2 is characterized in that: The frequency enhancement module performs the following operations: F low =AvgPool(F) F high =F-F low F out =w1F low +W2F high Among them, AvgPool is the average pooling operation, F represents the input feature, and F low and F high Represent the low-frequency and high-frequency components of the input features, W1 and W2 represent linear projections, and F out The above operations are performed in the horizontal and vertical directions in sequence, and the operation results in the horizontal direction are used as the input of the vertical operation. The output in the vertical direction is added to the input feature F to obtain rich and compact context information.

4. The image deraining method based on space-frequency dual-domain perception according to claim 1 is characterized in that: The frequency-controlled feedforward network includes layer normalization, Fourier transform, convolution mapping, inverse transform and gated convolution from front to back.

5. The image deraining method based on space-frequency dual-domain perception according to claim 1 is characterized in that: The downsampling of the input image adopts the nearest neighbor downsampling mode, and the fusion of the input image downsampling features and the deep encoder input adopts convolution operation and element-by-element addition operation.

Images