A Single Image Rain Removal Method and System Based on Semi-Supervised Transfer Learning

Abstract

This disclosure relates to a method and system for deraining a single image based on semi-supervised transfer learning. The method includes: inputting a synthetic rain image and a real rain image into a stepwise optimization recurrent network, respectively, and extracting rain streak features to obtain synthetic rain streak images and real rain streak images; building models for the synthetic and real rain streak images using multi-scale convolutional sparse coding, and performing iterative semi-supervised training and iterative optimization on these models to obtain synthetic rain kernel sets and real rain kernel sets; minimizing the distance between the synthetic and real rain kernel sets to achieve a transfer learning process from the data domain of the synthetic rain image to the data domain of the real rain image. The system includes a semi-supervised training module, a multi-scale convolutional sparse coding module, and a transfer learning module connected sequentially. The method and system proposed in this disclosure are well-suited to real rainfall patterns, improving deraining performance and network generalization.

Description

Technical Field

[0001] This disclosure relates to the field of image deraining technology, and in particular to a method and system for deraining a single image based on semi-supervised transfer learning. Background Art

[0002] Single-image deraining algorithms are generally divided into model-driven and data-driven methods. Model-driven methods mainly rely on statistical analysis of rain streaks and background scenes, and solve for rain-free background images by designing and optimizing algorithms. This modeling approach depends on subjective human model design and can only perform deraining under specific scenarios. Due to the strong fitting ability of deep neural networks, and with the increase in data scale and computing power, data-driven deep learning-based single-image deraining algorithms have become the mainstream trend in recent years.

[0003] Deep learning-based methods typically employ supervised learning mechanisms, requiring a large number of paired rain / no-rain training data pairs. However, collecting a large number of real rain / no-rain image pairs is difficult and time-consuming. Therefore, most existing deep learning models are trained on synthetic paired datasets. However, synthetic data contains limited rain patterns, and there is a significant domain gap between synthetic and real rain, resulting in poor performance of deep network models trained on synthetic data in real-world tests. Furthermore, collecting individual real rain images has the advantage of low cost. Therefore, single-image deep learning techniques under semi-supervised learning mechanisms have attracted attention in academia.

[0004] However, on the one hand, existing semi-supervised algorithms generally have poor performance in deraining synthetic rain images, which is not as good as the performance of supervised depth state-of-the-art algorithms, and their performance in deraining real rain images is not much better than that of supervised depth single-image deraining algorithms. On the other hand, existing semi-supervised algorithms have difficulty effectively using accurate priors to train networks on real rain images, resulting in poor transfer learning effects.

[0005] Therefore, it is necessary to propose a solution to improve one or more problems existing in the above-mentioned related technical solutions.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] The first aspect of this disclosure provides a single-image rain removal method based on semi-supervised transfer learning, comprising the following steps:

[0008] The synthetic rain image and the real rain image are respectively input into the stepwise optimization recurrent network, and rain pattern features are extracted respectively to obtain the synthetic rain pattern image and the real rain pattern image.

[0009] Models are built for the synthetic rain pattern image and the real rain pattern image using multi-scale convolutional sparse coding, and the models are semi-supervised and iteratively optimized to obtain the synthetic rain kernel set and the real rain kernel set, respectively.

[0010] Minimize the distance between the synthetic rain kernel set and the real rain kernel set to achieve a transfer learning process from the data domain of the synthetic rain image to the data domain of the real rain image;

[0011] The synthetic rain pattern image includes multiple synthetic rain layer images; the real rain pattern image includes multiple real rain layer images.

[0012] In an exemplary embodiment of this disclosure, the steps of inputting the synthetic rain image and the real rain image into a stepwise optimization recurrent network, and performing feature extraction processing on the rain pattern image to obtain the synthetic rain pattern image and the real rain pattern image respectively, are as follows:

[0013] The progressively optimized recurrent network comprises multiple sequentially connected recurrent units with identical structures. Each recurrent unit includes at least a first convolutional layer, a long short-term memory layer, a residual block, and a second convolutional layer, all connected sequentially.

[0014] The first convolutional layer is used to receive the synthetic rain image and the real rain image, respectively;

[0015] The long short-term memory layer is used for feature transfer between the recurrent units;

[0016] The residual block is used to extract the rain pattern features at a depth;

[0017] The second convolutional layer is used to output the synthetic rain pattern image and the real rain pattern image;

[0018] Each of the recurrent units shares the same recurrent network parameters.

[0019] In an exemplary embodiment of this disclosure, the expression for the synthesized rain image includes:

[0020]

[0021] in, This represents a composite rain image; This represents the background image of the synthesized rain image after progressively optimizing the recurrent network to remove rain. f θ (·) represents the formal format for progressively optimizing a cyclic network; This represents a synthetic rain pattern image; the synthetic rain pattern image The expressions include:

[0022]

[0023] in, Indicates the synthetic rain kernel set; This represents the feature map corresponding to the synthesized rain pattern image; K represents two-dimensional convolution; s Indicates the number of synthetic rain nuclei; N sk This indicates the number of feature maps corresponding to the synthetic rain kernel;

[0024] The expression for the real rain image includes:

[0025]

[0026] in, Represents a real rain image; This represents the background image of the real rain image after progressively optimizing the recurrent network to remove rain. f θ (·) represents the formal format for progressively optimizing a cyclic network; Represents a real rain pattern image; the real rain pattern image The expressions include:

[0027]

[0028] in, Represents the true rain kernel set; A feature map representing the actual rain pattern image; K represents two-dimensional convolution; r N represents the number of actual rain cores. rk This represents the number of feature maps corresponding to the actual rain kernels.

[0029] In an exemplary embodiment of this disclosure, the steps of establishing models for the synthetic rain pattern image and the real rain pattern image using multi-scale convolutional sparse coding, and performing semi-supervised training and iterative optimization on the models to obtain the synthetic rain kernel set and the real rain kernel set, respectively, are as follows:

[0030] The semi-supervised training includes supervised training of the synthetic rain pattern image and unsupervised training of the real rain pattern image.

[0031] In an exemplary embodiment of this disclosure, the step of supervising training of the synthesized rain pattern image includes:

[0032] The paired synthetic rain image and rainless image are input into the stepwise optimization recurrent network, which outputs the background image of the synthetic rain image after rain removal and extracts the synthetic rain pattern image.

[0033] The difference between the background image and the corresponding clean background image of the synthesized rain image is minimized using a first loss function;

[0034] A second loss function is applied to the synthetic rain pattern image to ensure that the multi-scale convolutional sparse coding establishes the model for the synthetic rain pattern image; after iterative optimization, the synthetic rain kernel set is obtained.

[0035] The expression for the first loss function includes:

[0036]

[0037] Where L1 represents the first loss function; This represents the clean background image corresponding to the synthesized rain image; This represents the mean squared error loss function;

[0038] The expression for the second loss function includes:

[0039]

[0040] Where L2 represents the second loss function; β represents the hyperparameter; α represents the hyperparameter; and ||·||1 represents the loss form of L1.

[0041] In an exemplary embodiment of this disclosure, the step of iteratively optimizing the supervised training includes:

[0042] Update the feature map corresponding to the synthesized rain pattern image. Fixed network parameters f θ and the synthetic rain kernel set For the synthesized rain pattern image Perform singular value decomposition and take the components u of the singular value decomposition. s As the initial value of the synthetic rain core The feature map corresponding to the synthetic rain pattern image The update solution process includes:

[0043]

[0044] Update the synthetic rain kernel set Fix the network parameter f θ The feature map corresponding to the synthesized rain pattern image and the real rain kernel set The synthetic rain core set The iterative process includes:

[0045]

[0046] Using the proximal gradient descent method, let τ be the step size of the gradient descent, Prox||·||≤1 (·) represents the proximal operator, yielding the iterative optimization result of the synthetic rain kernel set; the iterative optimization result of the synthetic rain kernel set includes:

[0047]

[0048] Where γ represents the hyperparameter; T represents the transpose of the matrix; t+0.5 represents the intermediate time of the iterative optimization of the synthetic rain kernel set; and t+1 represents the t+1 time of the iterative optimization of the synthetic rain kernel set.

[0049] In an exemplary embodiment of this disclosure, the step of performing unsupervised training on the real rain pattern image includes:

[0050] The real rain image is input into the stepwise optimization recurrent network, and the background image of the real rain image after rain removal is output, and the real rain pattern image is extracted.

[0051] A third loss function is applied to the background image of the real rain image to smooth the background image of the real rain image;

[0052] A fourth loss function is applied to the real rain pattern image to ensure that the multi-scale convolutional sparse coding establishes the model for the real rain pattern image. After iterative optimization, the real rain kernel set is obtained.

[0053] The expression for the third loss function includes:

[0054]

[0055] Where L3 represents the third loss function; TV represents global variational regularization; The background image represents the real rain image after progressively optimizing the recurrent network to remove rain.

[0056] The expression for the fourth loss function includes:

[0057]

[0058] Where L4 represents the fourth loss function; β represents the hyperparameter; Let represent the mean squared error loss function; ||·||1 represents the L1 loss form; α represents the hyperparameter.

[0059] In an exemplary embodiment of this disclosure, the step of iteratively optimizing the unsupervised training includes:

[0060] Update the feature map corresponding to the real rain pattern image. Fixed network parameters f θ and the real rain kernel set For the real rain pattern image Perform singular value decomposition and take the components u of the singular value decomposition. r As the initial value of the synthetic rain core The feature map corresponding to the real rain pattern image The update solution process includes:

[0061]

[0062] Update the real rain kernel set Fix the network parameter f θ The feature map corresponding to the real rain pattern image and the synthetic rain kernel set The real rain core set The iterative process includes:

[0063]

[0064] Optimize the network parameter f θ The expressions include:

[0065]

[0066] Where l(θ) represents the optimization result of the network parameters; μ represents the hyperparameters.

[0067] In an exemplary embodiment of this disclosure, the process of minimizing the distance between the synthetic rain kernel set and the real rain kernel set to achieve transfer learning from the data domain of the synthetic rain image to the data domain of the real rain image includes:

[0068] A consistency learning method is adopted, which uses the fifth loss function to minimize the Euclidean distance between the synthetic rain kernel set and the real rain kernel set, thereby realizing the transfer learning process from the data domain of the synthetic rain image to the data domain of the real rain image.

[0069] The expression for the fifth loss function includes:

[0070]

[0071] Where L5 represents the fifth loss function; Represents the true rain kernel set; Indicates the synthetic rain kernel set; This represents the mean squared error loss function.

[0072] A second aspect of this disclosure provides a single-image rain removal system based on semi-supervised transfer learning. The single-image rain removal system includes a semi-supervised training module, a multi-scale convolutional sparse coding module, and a transfer learning module connected sequentially.

[0073] The semi-supervised training module includes a progressively optimized recurrent network, which receives synthetic rain image information and real rain image information, performs semi-supervised training on the synthetic rain image information and the real rain image information, and inputs the training results into the multi-scale convolutional sparse coding module.

[0074] The multi-scale convolutional sparse coding module is used to model and iteratively optimize the training results, and input the synthetic rain kernel set and the real rain kernel set into the transfer learning module;

[0075] The transfer learning module is used to minimize the Euclidean distance between the synthetic rain kernel set and the real rain kernel set, thereby realizing the transfer learning process from the data domain of the synthetic rain image to the data domain of the real rain image.

[0076] This disclosure proposes a single-image rain removal method based on semi-supervised transfer learning. This method utilizes multi-scale sparse coding within a semi-supervised transfer learning framework for single-image rain removal to accurately simulate the intrinsic features of rain streaks, thus improving the transfer performance from synthetic and real rain streaks. By minimizing the distance between the synthetic and real rain kernel sets, transfer learning from the data domain of synthetic rain images to the data domain of real rain images is achieved, improving the algorithm's generalization ability. The semi-supervised transfer learning framework of this disclosure uses a progressively optimized recurrent network as the backbone network. When the training and test domains are the same, the progressively optimized recurrent network achieves state-of-the-art (SOTA) performance. When the training and test domains are different, transfer learning can be applied to minimize the gap, thus demonstrating superior rain removal performance for both synthetic and real rain image data. Attached Figure Description

[0077] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0078] Figure 1 This diagram illustrates the steps of a single-image deraining method based on semi-supervised transfer learning in an exemplary embodiment of this disclosure.

[0079] Figure 2 A schematic diagram of the structure of a single-image rain removal method based on semi-supervised transfer learning in an exemplary embodiment of this disclosure is shown.

[0080] Figure 3 A flowchart illustrating a single-image rain removal method based on semi-supervised transfer learning in an exemplary embodiment of this disclosure is shown.

[0081] Figure 4 A schematic diagram showing a comparison of the number of iterations and the rain removal effect between the synthetic rain map and the real rain map in an exemplary embodiment of this disclosure;

[0082] Figure 5 A block diagram of a single-image deraining system based on semi-supervised transfer learning in an exemplary embodiment of this disclosure is shown. Detailed Implementation

[0083] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0084] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0085] The first aspect of this exemplary implementation provides a single-image rain removal method based on semi-supervised transfer learning, such as... Figure 1 As shown, the following steps may be included:

[0086] Step S101: Input the synthetic rain image and the real rain image into the Progressive Recurrent Network (PReNet) respectively, and extract the rain pattern features to obtain the synthetic rain pattern image and the real rain pattern image respectively.

[0087] Step S102: Build models for synthetic rain pattern images and real rain pattern images using Multiscale Convolutional Sparse Coding (MS-CSC), and perform semi-supervised training and iterative optimization on the models to obtain synthetic rain kernel sets and real rain kernel sets respectively.

[0088] Step S103: Minimize the distance between the synthetic rain kernel set and the real rain kernel set to realize the transfer learning process from the data domain of the synthetic rain image to the data domain of the real rain image.

[0089] It's important to note that since rain streaks can be modeled as the sum of multiple rain layers, each layer within a rain streak may share similar local repetitive patterns (including their shape, thickness, and orientation) at different locations. These patterns can be represented using a convolutional dictionary learning model. Therefore, synthetic rain streak images consist of multiple synthetic rain layer images; similarly, real rain streak images consist of multiple real rain layer images.

[0090] This disclosure proposes a single-image rain removal method based on semi-supervised transfer learning. This method utilizes multi-scale sparse coding within a semi-supervised transfer learning framework for single-image rain removal to accurately simulate the intrinsic features of rain streaks, thus improving the transfer performance from synthetic and real rain streaks. By minimizing the distance between the synthetic and real rain kernel sets, transfer learning from the data domain of the synthetic rain image to the data domain of the real rain image is achieved, improving the algorithm's generalization ability. The semi-supervised transfer learning framework uses a progressively optimized recurrent network as the backbone network. When the training and test domains are the same, the progressively optimized recurrent network achieves state-of-the-art (SOTA) performance. When the training and test domains are different, transfer learning can be applied to minimize the gap, thus demonstrating superior rain removal performance for both synthetic and real rain image data.

[0091] The steps of the method described above in this example embodiment will now be explained in more detail.

[0092] In this embodiment, the PReNet neural network is used as the backbone network. The PReNet model is trained under a semi-supervised mechanism. Both synthetic and real rain images are input into the PReNet neural network to extract rain pattern features, resulting in synthetic and real rain pattern images, respectively. MS-CSC is used to model the synthetic and real rain pattern images, and the model is iteratively semi-supervised to obtain synthetic and real rain kernel sets, respectively. The distance between the synthetic and real rain kernel sets is minimized. This effectively adapts to real rainfall patterns and improves the network's generalization ability. Furthermore, rain pattern extraction from synthetic rain images is performed under the supervision of a clean background image, while extraction from real rain images is unsupervised. A loss function is used...

[0093]

[0094] This model is then trained. The PReNet neural network can perform internal loop calculations, achieving good rain removal results without increasing the network parameters.

[0095] In step S101, as Figure 2As shown, the PReNet neural network includes the supervised and unsupervised parts mentioned above; and the PReNet neural network includes multiple sequentially connected recurrent units with the same structure, each recurrent unit including at least a first convolutional layer, a long short-term memory layer, a residual block, and a second convolutional layer connected in sequence.

[0096] It should be noted that the first convolutional layer (Conv) is used to receive the input image, that is, it can receive synthetic rain images and real rain images respectively.

[0097] Long Short-Term Memory (LSTM) layers are used for feature transfer between recurrent units.

[0098] A residual block (ResBlock) is at least one or a combination of several residual blocks used for deep extraction of rain pattern features.

[0099] The second convolutional layer is used to output both synthetic rain pattern images and real rain pattern images.

[0100] Furthermore,

[0101] In step S101, the expression for the synthesized rain image is:

[0102]

[0103] in, This represents a composite rain image; This represents the background image of the synthesized rain image after progressively optimizing the recurrent network to remove rain. f θ (·) represents the formal format for progressively optimizing a cyclic network; This represents a composite rain pattern image.

[0104] Here, the rain pattern image is synthesized. The expression is:

[0105]

[0106] in, Indicates the synthetic rain kernel set; This represents the feature map corresponding to the synthesized rain pattern image; K represents two-dimensional convolution; s Indicates the number of synthetic rain nuclei; N sk This indicates the number of feature maps corresponding to the synthetic rain kernel.

[0107] In step S101, the expression for the real rain image is:

[0108]

[0109] in, Represents a real rain image; This represents the background image of the real rain image after progressively optimizing the recurrent network to remove rain. f θ (·) represents the formal format for progressively optimizing a cyclic network; Represents a true image of rain patterns.

[0110] Here is a real rain pattern image. The expression is:

[0111]

[0112] in, Represents the true rain kernel set; A feature map representing the actual rain pattern image; K represents two-dimensional convolution; r N represents the number of actual rain cores. rk This represents the number of feature maps corresponding to the actual rain kernels.

[0113] In step S102, as Figure 3 and Figure 4 As shown, this embodiment includes supervised training of synthetic rain pattern images and unsupervised training of real rain pattern images.

[0114] Furthermore, the supervised training process for the synthesized rain pattern images is as follows:

[0115] First, pairs of synthetic rain images and rainless images are input into the PReNet neural network (i.e., a progressively optimized recurrent network), which outputs the background image of the synthetic rain image after rain removal and extracts the synthetic rain pattern image.

[0116] Next, the difference between the background image of the synthesized rain image and its corresponding clean background image is minimized using the first loss function. Here, the expression for the first loss function is:

[0117]

[0118] Where L1 represents the first loss function; This represents the clean background image corresponding to the synthesized rain image; This represents the mean squared error loss function.

[0119] Next, a second loss function is applied to the synthesized rain strife image to ensure that MS-CSC accurately models the synthesized rain strife image. Here, the expression for the second loss function is:

[0120]

[0121] Where L2 represents the second loss function; β represents the hyperparameter; α represents the hyperparameter; and ||·||1 represents the loss form of L1.

[0122] Finally, supervised training is completed through iterative optimization by combining the first and second loss functions.

[0123] Furthermore, since the PReNet neural network can perform internal loop computation without increasing the network parameters, the steps for iterative optimization during supervised training are as follows:

[0124] First, update the feature map corresponding to the synthesized rain swastika image. Fixed network parameters f θ and synthetic rain core set Synthetic rain pattern image Perform singular value decomposition and extract the singular value decomposition components u. s As the initial value for synthesizing rain kernels Feature map corresponding to the synthesized rain pattern image The update solution process is as follows:

[0125]

[0126] Secondly, update the synthetic rain core set. Fixed network parameters f θ Feature maps corresponding to synthetic rain pattern images and real rain core collection Synthetic Rain Core Set The iterative process includes:

[0127]

[0128] Finally, using the proximal gradient descent method, let τ be the step size of the gradient descent, Prox ||·||≤1 (·) represents the proximal operator, yielding the iterative optimization result of the synthetic rain kernel set; the iterative optimization result of the synthetic rain kernel set is:

[0129]

[0130] Where γ represents the hyperparameter; T represents the transpose of the matrix; t+0.5 represents the intermediate time of the iterative optimization of the synthetic rain kernel set; and t+1 represents the t+1 time of the iterative optimization of the synthetic rain kernel set.

[0131] Furthermore, the steps for unsupervised training on real rain pattern images are as follows:

[0132] First, the real rain image is input into the PReNet neural network, which outputs the background image of the real rain image after the rain has stopped and extracts the real rain pattern image.

[0133] Next, a third loss function is applied to the background image of the real rain image, which smooths the background image of the real rain image.

[0134] Here, the expression for the third loss function is:

[0135]

[0136] Where L3 represents the third loss function; TV represents global variational regularization; The background image represents the real rain image after the recurrent network is gradually optimized to remove rain.

[0137] Next, a fourth loss function is applied to the real rain pattern image to ensure that MS-CSC accurately models the real rain map.

[0138] Here, the expression for the fourth loss function is:

[0139]

[0140] Where L4 represents the fourth loss function; β represents the hyperparameter; Let represent the mean squared error loss function; ||·||1 represents the L1 loss form; α represents the hyperparameter.

[0141] Finally, unsupervised training is completed through iterative optimization using the third and fourth loss functions combined.

[0142] Furthermore, the steps for iteratively optimizing this unsupervised training are as follows:

[0143] First, update the feature map corresponding to the real rain pattern image. Fixed network parameters f θ and real rain core collection For real rain streak images Perform singular value decomposition and extract the singular value decomposition components u. r As the initial value for synthesizing rain kernels Feature map corresponding to real rain streak image The update solution process is as follows:

[0144]

[0145] Secondly, update the real rain core set. Fixed network parameters f θ Feature maps corresponding to real rain streaks and synthetic rain core set Real Rain Core Collection The iterative process is as follows:

[0146]

[0147] in,

[0148]

[0149]

[0150]

[0151] Finally, optimize the network parameter f. θ The expression is:

[0152]

[0153] Where l(θ) represents the optimization result of the network parameters; μ represents the hyperparameters.

[0154] In step S103, as Figure 3 As shown, the consistency learning method is adopted, and the fifth loss function is used to minimize the Euclidean distance between the synthetic rain kernel set and the real rain kernel set, so as to realize the transfer learning process from the data domain of synthetic rain image to the data domain of real rain image.

[0155] Here, the expression for the fifth loss function is:

[0156]

[0157] Where L5 represents the fifth loss function; Represents the true rain kernel set; Indicates the synthetic rain kernel set; This represents the mean squared error loss function.

[0158] like Figure 4 As shown, with the increase of the number of iterations, the real rain kernel and the synthetic rain kernel become closer and closer. This indicates that the method proposed in this disclosure reduces the gap between the data domain of the synthetic rain image and the data domain of the real rain image, thereby achieving better generalization.

[0159] The second aspect of this exemplary implementation provides a single-image deraining system based on semi-supervised transfer learning, such as... Figure 5 As shown, the system includes a semi-supervised training module, a multi-scale convolutional sparse coding module, and a transfer learning module connected in sequence; wherein,

[0160] The semi-supervised training module includes a progressively optimized recurrent network, which receives synthetic rain image information and real rain image information, performs semi-supervised training on the synthetic rain image information and real rain image information, and inputs the training results into the multi-scale convolutional sparse coding module.

[0161] The multi-scale convolutional sparse coding module is used to model the training results and input the synthetic rain kernel set and the real rain kernel set into the transfer learning module;

[0162] The transfer learning module is used to minimize the Euclidean distance between the synthetic rain kernel set and the real rain kernel set, realizing the transfer learning process from the data domain of the synthetic rain image to the data domain of the real rain image.

[0163] In summary, this disclosure, through feature analysis of rain patterns, reveals two inherent characteristics in a single image: First, the rain pattern, where the rain nuclei exhibit repetitive local patterns and are sparsely distributed across various locations; second, due to varying camera distances, the rainfall pattern possesses a multi-scale structure. To fully capture these features, the rain pattern image of a single image is modeled as MS-CSC, and a simple and efficient PReNet neural network is selected as the backbone network. This reduces the distance between the data domain of the synthesized rain image and the data domain of the real rain image, effectively extracting the distribution of rain patterns.

[0164] It should be noted that although several units of the system for executing actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units. Some or all of the units can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0165] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A single-image rain removal method based on semi-supervised transfer learning, characterized in that, Includes the following steps: The synthetic rain image and the real rain image are respectively input into the stepwise optimization recurrent network, and rain pattern features are extracted respectively to obtain the synthetic rain pattern image and the real rain pattern image. Models are built for the synthetic rain pattern image and the real rain pattern image using multi-scale convolutional sparse coding, and the models are semi-supervised and iteratively optimized to obtain the synthetic rain kernel set and the real rain kernel set, respectively. The semi-supervised training includes supervised training of the synthetic rain pattern image and unsupervised training of the real rain pattern image. Minimize the distance between the synthetic rain kernel set and the real rain kernel set to achieve a transfer learning process from the data domain of the synthetic rain image to the data domain of the real rain image; The synthetic rain pattern image includes multiple synthetic rain layer images; the real rain pattern image includes multiple real rain layer images. The expression for the synthesized rain image includes: （1） in, This represents a composite rain image; This represents the background image of the synthesized rain image after progressively optimizing the recurrent network to remove rain. ; A formal format for progressively optimizing a cyclic network; This represents a synthetic rain pattern image; the synthetic rain pattern image The expressions include: （2） in, Indicates the synthetic rain kernel set; This represents the feature map corresponding to the synthesized rain pattern image; Represents two-dimensional convolution; K s Indicates the number of synthesized rain cores; N sk This indicates the number of feature maps corresponding to the synthetic rain kernel; The expression for the real rain image includes: （3） in, Represents a real rain image; This represents the background image of the real rain image after progressively optimizing the recurrent network to remove rain. ; A formal format for progressively optimizing a cyclic network; Represents a real rain pattern image; the real rain pattern image The expressions include; （4） in, Represents the true rain kernel set; A feature map representing the actual rain pattern image; Represents two-dimensional convolution; K r Indicates the actual number of rain cores; N rk This represents the number of feature maps corresponding to the actual rain kernels.

2. The single-image rain removal method based on semi-supervised transfer learning according to claim 1, characterized in that, In the step of inputting the synthetic rain image and the real rain image into a stepwise optimization recurrent network, and performing feature extraction processing on the rain pattern image to obtain the synthetic rain pattern image and the real rain pattern image respectively, The progressively optimized recurrent network comprises multiple sequentially connected recurrent units with identical structures. Each recurrent unit includes at least a first convolutional layer, a long short-term memory layer, a residual block, and a second convolutional layer, all connected sequentially. The first convolutional layer is used to receive the synthetic rain image and the real rain image, respectively; The long short-term memory layer is used for feature transfer between the recurrent units; The residual block is used to extract the rain pattern features at a depth; The second convolutional layer is used to output the synthetic rain pattern image and the real rain pattern image; Each of the recurrent units shares the same recurrent network parameters.

3. The single-image rain removal method based on semi-supervised transfer learning according to claim 1, characterized in that, The steps for supervised training of the synthesized rain pattern image include: The paired synthetic rain image and rainless image are input into the stepwise optimization recurrent network, which outputs the background image of the synthetic rain image after rain removal and extracts the synthetic rain pattern image. The difference between the background image and the corresponding clean background image of the synthesized rain image is minimized using a first loss function; A second loss function is applied to the synthesized rain pattern image to ensure that the multi-scale convolutional sparse coding establishes the model for the synthesized rain pattern image; after iterative optimization, the synthesized rain kernel set is obtained; The expression for the first loss function includes: （5） in, Represents the first loss function; This represents the clean background image corresponding to the synthesized rain image; This represents the mean squared error loss function; The expression for the second loss function includes: （6） in, This represents the second loss function; Indicates hyperparameters; Indicates hyperparameters; express The form of loss.

4. The single-image rain removal method based on semi-supervised transfer learning according to claim 3, characterized in that, The steps for iteratively optimizing the supervised training include: Update the feature map corresponding to the synthesized rain pattern image. Fixed network parameters and the synthetic rain kernel set For the synthesized rain pattern image Perform singular value decomposition and extract the components of the singular value decomposition. As the initial value of the synthetic rain core The feature map corresponding to the synthesized rain pattern image The update solution process includes: （7） Update the synthetic rain kernel set : Fix the network parameters The feature map corresponding to the synthesized rain pattern image and the real rain kernel set The synthetic rain core set The iterative process includes: （8） Using the proximal gradient descent method, let Let be the step size of the gradient descent. Using a proximal operator, the iterative optimization results of the synthetic rain kernel set are obtained; the iterative optimization results of the synthetic rain kernel set include: （9） Where γ represents the hyperparameter; T represents the transpose of the matrix; t+0.5 represents the intermediate time of the iterative optimization of the synthetic rain kernel set; and t+1 represents the t+1 time of the iterative optimization of the synthetic rain kernel set.

5. The single-image rain removal method based on semi-supervised transfer learning according to claim 1, characterized in that, The steps for unsupervised training on the real rain pattern image include: The real rain image is input into the stepwise optimization recurrent network, and the background image of the real rain image after rain removal is output, and the real rain pattern image is extracted. A third loss function is applied to the background image of the real rain image to smooth the background image of the real rain image; A fourth loss function is applied to the real rain pattern image to ensure that the multi-scale convolutional sparse coding establishes the model for the real rain pattern image. After iterative optimization, the real rain kernel set is obtained. The expression for the third loss function includes: （10） in, Represents the third loss function; TV Indicates global variational regularity; The background image represents the real rain image after progressively optimizing the recurrent network to remove rain. The expression for the fourth loss function includes: （11） in, This represents the fourth loss function; Indicates hyperparameters; This represents the mean squared error loss function; express The form of loss; This represents hyperparameters.

6. The single-image rain removal method based on semi-supervised transfer learning according to claim 5, characterized in that, The steps for iteratively optimizing the unsupervised training include: Update the feature map corresponding to the real rain pattern image. Fixed network parameters and the real rain kernel set For the real rain pattern image Perform singular value decomposition and extract the components of the singular value decomposition. As the initial value of the synthetic rain core The feature map corresponding to the real rain pattern image The update solution process includes: （12） Update the real rain kernel set : Fix the network parameters The feature map corresponding to the real rain pattern image and the synthetic rain kernel set The real rain core set The iterative process includes: （13） Optimize the network parameters The expressions include: （14） Where l(θ) represents the optimization result of the network parameters; μ represents the hyperparameters.

7. The single-image rain removal method based on semi-supervised transfer learning according to claim 1, characterized in that, The process of minimizing the distance between the synthetic rain kernel set and the real rain kernel set to achieve transfer learning from the data domain of the synthetic rain image to the data domain of the real rain image includes: A consistency learning method is adopted, which uses the fifth loss function to minimize the Euclidean distance between the synthetic rain kernel set and the real rain kernel set, thereby realizing the transfer learning process from the data domain of the synthetic rain image to the data domain of the real rain image. The expression for the fifth loss function includes: （15） in, This represents the fifth loss function; Represents the true rain kernel set; Indicates the synthetic rain kernel set; This represents the mean squared error loss function.

8. A single-image rain removal system based on semi-supervised transfer learning, characterized in that, The system is used to perform the method as described in any one of claims 1 to 7, wherein the single-image rain removal system comprises a semi-supervised training module, a multi-scale convolutional sparse coding module, and a transfer learning module connected in sequence; wherein... The semi-supervised training module includes a progressively optimized recurrent network, which receives synthetic rain image information and real rain image information, performs semi-supervised training on the synthetic rain image information and the real rain image information, and inputs the training results into the multi-scale convolutional sparse coding module. The multi-scale convolutional sparse coding module is used to model and iteratively optimize the training results, and input the synthetic rain kernel set and the real rain kernel set into the transfer learning module; The transfer learning module is used to minimize the Euclidean distance between the synthetic rain kernel set and the real rain kernel set, thereby realizing the transfer learning process from the data domain of the synthetic rain image to the data domain of the real rain image.

Images