A multi-scale efficient convolutional self-attention single image rain removal method

Abstract

The present invention discloses a multi-scale efficient convolutional self-attention single image rain removal method. First, image data is preprocessed to obtain corresponding rainy and rain-free images. Then, the rainy image is input into a network model that integrates an improved Transformer self-attention module and a multi-scale spatial feature fusion module for iterative training. After optimization through a hybrid loss function, a processed image close to the rain-free image is output. The trained network model is saved, and then the trained network model is used to predict and output the rain-free image for the image data to be tested. This method avoids the problems of insufficient generalization ability of model-driven rain removal methods and large computational overhead of data-driven rain removal methods, and solves the technical problems of large computational complexity and low efficiency of single image rain removal methods in the existing technology.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-scale efficient convolutional self-attention single image rain removal method. Background Art

[0002] Rain, a common natural weather condition, significantly impacts the quality of images and videos captured by outdoor vision systems, hindering the performance of subsequent advanced computer vision tasks such as object tracking, target detection, and image segmentation. Removing rain noise from rainy images and restoring a clear background is a crucial image preprocessing problem.

[0003] Due to the limited available image feature information in a single image, rain removal from a single image is challenging. Existing single-image rain removal methods can be divided into two categories: model-driven and data-driven methods. Model-driven methods first establish a physical model for rain streaks based on some prior knowledge, such as the physical properties of rainwater. Then, a series of sophisticated mathematical models are manually designed to remove rain noise from the rainy image, ultimately obtaining a clean, rain-free background image. However, these model-driven rain removal methods are only applicable to specific rainfall types and cannot cope with the irregular distribution of actual rainfall images. Moreover, the optimization algorithms used in such methods typically involve many computational iterations, resulting in low efficiency. Data-driven rain removal methods leverage the powerful feature extraction capabilities of deep learning network models. After training on large datasets, they learn the features of rain streaks and effective background information, thereby restoring rainy images to a rain-free image. These deep learning rain removal methods all treat the single-image rain removal problem as a multi-stage task and use recursive structures for rain streak feature extraction and removal. This leads to a deeper network layer and a sharp increase in the number of parameters, which increases the network training time and incurs a huge computational overhead. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-scale efficient convolutional self-attention single image rain removal method, aiming to solve the technical problems of large computational complexity and low efficiency of the single image rain removal method in the prior art.

[0005] To achieve the above objectives, the present invention adopts a multi-scale efficient convolutional self-attention single image rain removal method, which includes the following steps:

[0006] Data preprocessing;

[0007] Build a network model;

[0008] training the network model;

[0009] Network model optimization;

[0010] The predicted output is the image after rain removal.

[0011] In the process of data preprocessing, the image data is preprocessed to obtain a rainy image and a rainless image, wherein the rainy image and the rainless image are respectively scenes with rain and without rain in the same environment.

[0012] The rainy images are used as initial image data for training, and the rainless images are used as comparison data after processing.

[0013] Among them, the network model includes an encoding structure and a decoding structure. The encoding structure integrates the improved Transformer self-attention module, and the encoding structure also embeds a multi-scale spatial feature fusion module. The decoding structure contains conventional efficient convolution blocks and integrates the semantic features of the corresponding scale in the encoding structure.

[0014] The improved Transformer attention module, due to the addition of positional encoding, not only has the ability to model global features, but is also sensitive to local similar features, which is conducive to removing rain noise and preserving background detail texture to the maximum extent. The multi-scale spatial feature fusion block embedded in the encoding stage can alleviate the problem of partial image feature loss during the downsampling process in the encoding stage.

[0015] In the process of training the network model, the optimal parameters of the pre-trained model are first loaded into the network model, wherein the pre-trained model is the network model trained before the network is improved, and then the rainy image is passed into the network model for iterative training.

[0016] In the process of optimizing the network model, a hybrid loss function back-propagation optimization is used to iteratively update the network parameters of the network model so that the output result is close to the rain-free image, and the trained network model is saved.

[0017] The rainy image is iteratively processed in the network model, which is also a process of training the network model. Under the optimization of the hybrid loss function, the output processed image becomes closer and closer to the rain-free image. At this time, the network model is a trained network model and can be used to perform rain removal processing on other images.

[0018] In the process of predicting and outputting the image after rain removal, the prepared test image data is loaded into the trained network model for forward calculation to obtain the image after rain removal of the test image.

[0019] The present invention discloses a multi-scale efficient convolutional self-attention single-image rain removal method. First, image data is preprocessed to obtain corresponding rainy and rain-free images. The rainy images are then fed into a network model that incorporates an improved Transformer self-attention module and a multi-scale spatial feature fusion module for iterative training. A processed image close to the rain-free image is output through hybrid loss function optimization. The trained network model is saved and then used to predict and output the rain-free image for image data to be tested. This method avoids the problems of insufficient generalization capability of model-driven rain removal methods and large computational overhead of data-driven rain removal methods, and solves the technical problems of large computational complexity and low efficiency of single-image rain removal methods in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 This is a flow chart of a multi-scale efficient convolutional self-attention single image rain removal method of the present invention.

[0022] Figure 2 This is a network model structure diagram of the multi-scale efficient convolutional self-attention single image rain removal method of the present invention.

[0023] Figure 3 It is a structural diagram of the multi-scale spatial feature fusion module of the present invention.

[0024] Figure 4 This is a comparison chart of subjective experimental results of different algorithms in a specific embodiment of the present invention on the synthetic dataset Rain100H.

[0025] Figure 5 This is a comparison chart of average running time and evaluation indicators of different algorithms on Rain100H in a specific embodiment of the present invention.

[0026] Figure 6 1 is a diagram of subjective experimental results of different algorithms in a specific embodiment of the present invention on the simulation data set SPA.

[0027] Figure 7 This is a structural comparison diagram of two combination schemes of the cross-scale convolutional self-attention module of the present invention. DETAILED DESCRIPTION

[0028] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0029] See also Figure 1 The present invention proposes a multi-scale efficient convolutional self-attention single image rain removal method, which includes the following steps:

[0030] S1: data preprocessing;

[0031] S2: Build network model;

[0032] S3: training the network model;

[0033] S4: Network model optimization;

[0034] S5: Predict and output the image after rain removal.

[0035] During the data preprocessing process, the image data is preprocessed to obtain a rainy image and a rainless image, wherein the rainy image and the rainless image are respectively scenes with rain and without rain in the same environment.

[0036] The network model includes an encoding structure and a decoding structure. The encoding structure integrates an improved Transformer self-attention module and also embeds a multi-scale spatial feature fusion module. In addition to the conventional efficient convolution block (ECB), the decoding structure also integrates semantic features of corresponding scales in the encoding structure through the Skip Connection operation to guide the upsampling process in the decoding stage, establish long-distance feature dependencies, and facilitate the recovery of image details.

[0037] like Figure 2 As shown in the figure, the main body of the network consists of an encoder-decoder structure. In addition, it also integrates a cross-scale convolutional self-attention module and a multi-scale spatial feature fusion module improved on the basis of the original Transformer.

[0038] Improved Transformer self-attention module: In the 3rd and 4th stages of the downsampling phase, the improved Transformer self-attention module is integrated, and its calculation formula is:

[0039]

[0040] Where Q, K, V represent the input vector X∈R N×C After linear transformation W Q 、W K and WV ∈R C×C Mapped into corresponding query vector Q, key vector K and value vector V, is the scaling factor, softmax is the activation function, Represents the attention map, which is calculated using two-dimensional deep convolution in visual tasks

[0041]

[0042] Where ° represents the direct multiplication of elements at corresponding matrix positions. The improved Transformer attention module incorporates positional encoding, enabling it to model global features while also being sensitive to local similarities. This facilitates the removal of rain noise while preserving maximum background detail and texture.

[0043] Multi-scale spatial feature fusion module: In the image rain removal algorithm, the image is usually downsampled several times to discard redundant information in the image, but this will also cause some effective information to be lost, making it impossible to accurately locate the position of the rain streaks, resulting in incomplete rain streak removal and incomplete image background structure. In order to alleviate the problem of partial image feature loss during the downsampling process in the encoding stage, the present invention designs a multi-scale spatial feature fusion module embedded in the last stage of the encoding stage, aggregating contextual information from multiple scales to fully learn rain streak features of different sizes, so that the network model can cope with various complex rainfall situations in real environments. The specific structure of the multi-scale spatial feature fusion module is as follows: Figure 3 As shown in the figure, five parallel convolution operations are used to process the input features. First, a 1×1 convolution is used to reduce the dimension of the input feature map. Then, three 3×3 convolutions are set with different expansion factors of 2, 4, and 8 respectively. The image features are extracted with three different receptive fields to improve the model's ability to perceive rain streaks of different sizes. Secondly, an adaptive average pooling operation is used to reduce information redundancy. Finally, a 1×1 convolution is used to reduce the number of channels and the five feature maps of different scales are fused together. The effective information of different scales in the image will be fully learned.

[0044] During the process of training the network model, the rainy image is input into the network model for iterative training.

[0045] During the process of network model optimization, a hybrid loss function back-propagation optimization is used to iteratively update the network parameters of the network model so that the output result is close to the rain-free image, and the trained network model is saved.

[0046] Given a training loss objective function, this loss function is a mixture of the MAE loss function, the MS-SSIM loss function, the MSE loss function, and the TV loss function. The advantages of each loss function are used to make up for the defects of a single loss function, thereby enhancing the stability of the network. First, the MAE loss function and the MS-SSIM loss function are mixed according to certain weights. The formula is as follows:

[0047]

[0048] Among them L MS-SSIM is the MS-SSIM loss, L MAE is the MAE loss function, which is calculated as follows:

[0049] L MS-SSIM (P) = 1 - (MS-SSIM (p)) (4)

[0050]

[0051] Where P represents a pixel area, and p represents a pixel point in the pixel area P. Based on experience, α is set to 0.84.

[0052] Then, the MS-SSIM loss function is not particularly sensitive to consistency deviations, which can easily lead to changes in image brightness and color deviations; the TV loss function constrains the smoothness of the image by calculating the differences between adjacent pixels, making the output image relatively smooth. It can be used to solve the artifact problem caused by residual rain streaks in the image after rain removal, but it is not suitable for use alone. In rainy images, rain streaks and background detail textures mostly exist in high-frequency areas, and the MAE loss function will give a relatively large weight to the high-frequency part of the image, which will result in the retention of rain streaks while retaining details. Therefore, the MSE loss function and the TV loss function are used to remove rain streak artifacts. Finally, the hybrid loss function is as follows:

[0053] L Mix =L MS-SSIM-MAE +μ·L MSE +λ·L TV (6)

[0054] Where μ and λ are penalty factors, which are gradually adjusted according to the experiment and the obtained values ​​are 0.3 and 2×10 -8 The expressions of MSE loss function and TV loss function are as follows:

[0055]

[0056] L TV (p)=∑ i,j ((p i,j+1 -pi,j ) 2 +(p i+1,j -p i,j ) 2 ) β / 2 (8)

[0057] The obtained hybrid loss function is optimized and iterated on the network parameters of the network model through back propagation, so that the output result of the network gradually approaches the rain-free image, and the trained model is saved.

[0058] In the process of predicting and outputting the image after rain removal, the prepared test image data and the pre-trained model are loaded into the trained network model for forward calculation to obtain the image after rain removal of the test image.

[0059] Furthermore, the present invention provides a specific embodiment, using a synthetic dataset for experimental comparison:

[0060] In order to comprehensively verify the performance of the proposed technical solution, the present invention is compared with several currently advanced rain removal methods based on deep learning, specifically: MPRNet (2021), RCDNet (2020), JORDER-E (2020), DCSFN (2020), SPANet (2019), and RESCAN (2018).

[0061] See also Figure 4 , Figure 4 The subjective experimental results of different algorithms on the synthetic dataset Rain100H are shown. As can be seen from the figure, the proposed method can effectively remove rain streaks of different directions and densities, and generate rain-free images close to reality while retaining most of the details. In contrast, the images generated by other methods are relatively smooth and may even destroy the background content, such as Figure 4 (a) It can be seen that there are traces of smears on the woman's face. Figure 4 (b) The girl’s hair texture almost disappears, Figure 4 (c) The cross building is blurred. Among the three methods compared, only the MPRNet method retains some details. The DCSFN algorithm uses SSIM loss as the loss function, which makes the boundary of the structure clear, but also causes the residual of rain streaks, such as Figure 4In the DCSFN output image (c), rain streaks are clearly visible as artifacts in the sky. The RCDNet algorithm uses the MSE loss function, which heavily penalizes smooth areas in the image, blurring it. The image output by the algorithm of the present invention not only completely removes the rain streaks, but also renders the woman's face natural, the girl's hair texture clear, and the cross intact, fully preserving background details. This is due to the effective combination of the local feature modeling capabilities of convolution and the global feature modeling capabilities of self-attention. It also verifies the feasibility of the hybrid loss function in this solution.

[0062] In addition to comparing the subjective performance of each algorithm, to demonstrate the performance improvements achieved by the proposed algorithm in a statistically meaningful way, this paper uses two image quality assessment metrics, structural similarity (SSIM) and peak signal-to-noise ratio (PSNR), to objectively evaluate each algorithm. The closer the SSIM value is to 1, the greater the similarity between the two images; the larger the PSNR value, the less distortion there is in the image. Table 1 shows the SSIM and PSNR values ​​of each algorithm on different datasets. As can be seen from Table 1, the proposed algorithm is quite competitive compared to some currently advanced algorithms. From the perspective of image quality evaluation indicators alone, although on the Rain100L dataset, the proposed method is slightly inferior to RCDNet in terms of PSNR indicator and slightly lower than JORDER-E in terms of SSIM indicator, on the Rain100H dataset, the proposed method takes the lead. Compared with the latest MPRNet algorithm, the SSIM indicator is improved by 0.0153 and the PSNR indicator is improved by 0.95dB. The possible reason is that the weaker inductive bias in Transformer requires a larger training set. Transformer is suitable for large-scale datasets. Therefore, the proposed method has more advantages on the larger Rain100H dataset.

[0063]

[0064] Table 1. Comparison of evaluation metrics of different algorithms on synthetic datasets

[0065] In addition, in order to demonstrate the rain removal efficiency of the method of the present invention, the present invention also compares the average running time of different algorithms for processing a rainy image on the dataset Rain100H and the evaluation indicators PSNR and SSIM. The comparison results are as follows Figure 5 As shown in the figure, the algorithm proposed in this paper is comparable to RCDNet in terms of PSNR index, but its processing speed is about 50 times faster than RCDNet. Compared with the latest algorithm MPRNet, the algorithm proposed in this paper not only has higher evaluation indicators, but also has a significantly higher rain removal speed. This is due to the use of efficient convolution blocks in the method of this invention, which speeds up the inference speed of the model.

[0066] Furthermore, in order to verify the generalization ability of the algorithm proposed in this paper, the present invention also compares the algorithm in this paper with the latest algorithm MPRNet (2021) on a rainy dataset close to the real one. The comparison results are as follows: Figure 6 As shown in Figure 2, the two algorithms have comparable rain removal effects on images close to real rain, but the proposed method retains more details, such as Figure 6 In (a), the MPRNet algorithm removes a long white object from the background of the original image, while the proposed algorithm completely preserves the white object in the rain-removed image. Experiments demonstrate the strong generalization capabilities of our proposed algorithm, effectively removing rain streaks of varying degrees from synthetic rainy images and achieving excellent results on images of near-realistic rainy scenes.

[0067] Furthermore, for the method of integrating the cross-scale convolutional self-attention module in the encoding stage, two combination schemes are designed: Figure 7 As shown:

[0068] To compare the impact of the number of cross-scale convolutional self-attention modules combined with ordinary convolutions on the performance of the proposed network model, as well as the module's effectiveness in extracting features from images of varying resolutions, we trained two different combinations on the Rain100H dataset and demonstrated the evaluation metrics SSIM and PSNR for these two schemes on both datasets. As shown in Table 2, the evaluation metrics obtained after training on the Rain100H dataset using Combination A and Combination B are comparable, with Combination B slightly outperforming. However, Combination B contains fewer Transformer modules, which translates to fewer parameters and computational overhead. Therefore, this paper adopts Combination B, combining convolutional blocks with Transformers.

[0069]

[0070] Table 2. Comparison of evaluation indicators of combination A and combination B on the Rain100H dataset

[0071] Here, a hybrid loss function is used as the objective function to be optimized:

[0072] L Mix =L MS-SSIM-MAE +μ·L MSE +λ·L TV (9)

[0073] The above disclosure is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of the rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made in accordance with the claims of the present invention are still within the scope of the invention.

Claims

1. A multi-scale efficient convolutional self-attention single image rain removal method, characterized by: The following steps are involved: Data preprocessing; Build a network model; The network model includes an encoding structure and a decoding structure. The encoding structure integrates an improved Transformer self-attention module and embeds a multi-scale spatial feature fusion module. The decoding structure includes a conventional efficient convolution block and integrates semantic features of corresponding scales in the encoding structure. The improved Transformer self-attention module is calculated as follows: Where Q, K, V represent the input vector X∈R N×C After linear transformation W Q 、W K and W V ∈R C×C Mapped into corresponding query vector Q, key vector K and value vector V, is the scaling factor, softmax is the activation function, Represents the attention map, which is calculated using two-dimensional deep convolution in visual tasks in Indicates direct multiplication of elements at corresponding matrix positions; The multi-scale spatial feature fusion module processes input features using five parallel convolution operations. First, a 1×1 convolution is used to reduce the dimensionality of the input feature map. Then, three 3×3 convolutions are set with different expansion factors of 2, 4, and 8, respectively, to extract features from the image with three different receptive fields, improving the model's ability to perceive rain streaks of different sizes. Next, an adaptive average pooling operation is used to reduce information redundancy. Finally, a 1×1 convolution is used to reduce the number of channels and fuse the five feature maps of different scales together. This allows for the full learning of effective information at different scales in the image. training the network model; Network model optimization; During the network model optimization process, a hybrid loss function back-propagation optimization is used to iteratively update the network parameters of the network model so that the output result is close to the rain-free image, and the trained network model is saved; The hybrid loss function expression is as follows: L Mix =L MS-SSIM-MAE +μ·L MSE +λ·L TV The hybrid loss function is a mixture of MAE loss function, MS-SSIM loss function, MSE loss function and TV loss function, where μ and λ are penalty factors, which are 0.3 and 2×10 respectively. -8 ; Among them L MS-SSIM is the MS-SSIM loss, L MAE is the MAE loss function, which is calculated as follows: L MS-SSIM (P)=1-(MS-SSIM(p)) Where P represents a pixel area, p represents a pixel point in the pixel area P, and α is set to 0.84; The expressions of MSE loss function and TV loss function are as follows: The predicted output is the image after rain removal.

2. The multi-scale efficient convolutional self-attention single image rain removal method according to claim 1, characterized in that: During the data preprocessing process, the image data is preprocessed to obtain a rainy image and a rainless image, wherein the rainy image and the rainless image are respectively scenes with rain and without rain in the same environment.

3. The multi-scale efficient convolutional self-attention single image rain removal method according to claim 2, characterized in that: In the process of training the network model, the parameters of the pre-trained model are first loaded into the network model, wherein the pre-trained model is the network model trained before the network is improved, and then the rainy image is passed into the network model for iterative training.

4. The multi-scale efficient convolutional self-attention single image rain removal method according to claim 3, characterized in that: In the process of predicting and outputting the image after rain removal, the prepared test image data is loaded into the trained network model for forward calculation, and then the relatively optimal weights and bias parameters updated by the backpropagation process are calculated with the input to obtain the image after rain removal of the test image.

Images