A method for detecting and removing rain streaks in light field images based on transfer learning
By building a self-supervised network and a 3D recursively generated adversarial network, combining Gaussian process and LSTM layer, the problem of poor rain removal effect in real scenes is solved, and high-quality rain stripe detection and removal in light field images are achieved , enhances the generalization ability of the network.
Patent Information
- Application Number
- CN202110094780.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-01-25
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-01-25
AI Technical Summary
The existing deep learning-based image rain removal method is not effective in real scenes, and it is difficult to effectively detect and remove using depth information and rain stripe position relationship, especially in light field images.
Using a transfer learning-based method, a self-supervised network is built, and the depth map calculation module, a rain stripe detection module and a rainwater removal module are used to generate an adversarial network using Gaussian process and 3D recursion, combining the LSTM layer and discriminator to extract and remove light field images. The rain stripes in the middle enhance the generalization ability of the network.
It realizes more accurate detection and removal of rain stripes in real scenes, improves the quality of rain-free graphs and network generalization capabilities, and improves the details of the generator output.
Smart Images

Figure CN112767274B_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a light field image rain streak detection and removal method based on transfer learning, and belongs to the technical field of computer image processing. Background Art
[0002] Image deraining has always been a difficult and hot topic in the field of computer vision. Recent deep learning-based image deraining methods have shown excellent performance in terms of both reconstruction error and visual quality. However, due to various challenges in obtaining real scene image deraining datasets, most of these methods are only trained on synthetic data and are difficult to generalize to real scene images.
[0003] In addition, existing rain removal methods are mainly applied to single images or frame sequences. However, it is difficult to effectively detect and remove rain using depth information or the position relationship of rain streaks in either single images or frame sequences, and there is no good effect in real scenes.
[0004] Light field images (LFI) can capture multiple sub-aperture views at a time, record rich structure and texture information of the target scene, make it easier to estimate the depth map, and the positions of rain streaks in the sub-aperture views are highly correlated in space.
[0005] Therefore, in order to enhance the generalization ability of the network and take advantage of LFI to solve the above problems, the present invention designs a light field image rain streak detection and removal method based on transfer learning. Summary of the invention
[0006] The purpose of the present invention is to provide a method for detecting and removing rain streaks in light field images based on transfer learning. By using transfer learning to construct a self-supervised network, rain streaks in real scenes can be more accurately extracted and high-quality rain-free images can be obtained, thereby enhancing the generalization ability of the network.
[0007] In order to achieve the above-mentioned invention object, the present invention adopts the following technical scheme:
[0008] A method for detecting and removing rain streaks in a light field image based on transfer learning, the method comprising a depth map calculation module, a rain streak detection module and a rain removal module;
[0009] Wherein, the depth map calculation module obtains the depth map by calculation;
[0010] The rain streak detection module detects the synthetic data rain streak pattern, and at the same time, uses a self-supervised network based on a Gaussian process to detect the real scene data rain streak pattern;
[0011] The rain removal module connects the rainy 3DEPI volume block, the depth map obtained by the depth map calculation module and the rain streak map extracted by the rain streak detection module in series, and then inputs them into the 3D recursive generative adversarial network to remove rain and repair the background. The training is repeated until a high-quality rain-free image is obtained.
[0012] Furthermore, the depth map calculation module first uses a multi-stream neural network to extract information of light field sub-viewpoints from different directions, and then enters a fusion network to calculate the depth map.
[0013] Furthermore, the depth map includes a synthetic data depth map and a real scene depth map; wherein the synthetic data depth map is obtained by synthetic data entering the depth map calculation module; the real scene depth map is obtained by real scene data entering the depth map calculation module.
[0014] Furthermore, the rain streak detection module takes a 3DEPI volume block formed by stacking a row of sub-viewpoints of the light field image as input, uses a residual network to extract features of rain streaks, and in each layer of convolution, the extracted synthetic data features and real scene data features are modeled through a Gaussian process to obtain a pseudo-true value of the real scene data, which is further used to supervise the network to extract features of the real scene data.
[0015] Furthermore, the rain streak map includes a synthetic data rain streak map and a real scene rain streak map; wherein the synthetic data rain streak map is obtained by synthetic data entering the rain streak detection module; the real scene rain streak map is obtained by real data entering the rain streak detection module.
[0016] Furthermore, the rainwater removal module is provided with an LSTM layer and a discriminator structure.
[0017] Furthermore, the LSTM layer is used to propagate features between each iterative convolutional layer.
[0018] Furthermore, in each iteration, a long short-term memory network is used to transfer features.
[0019] Furthermore, the loss function of the discriminator is:
[0020]
[0021] Among them, y is the clean image output by the generator, y gt is the corresponding true value, and D(·) is the convolution operation of the discriminator network.
[0022] The advantages and beneficial effects of the present invention compared to the prior art are:
[0023] (1) The rain streak detection module of the present invention obtains the pseudo-true value of the real scene data by constructing a 3D residual network based on a Gaussian process. The pseudo-true value is further used to supervise the network to extract the features of the real scene data. The real scene data can be used to improve the generalization ability of the network.
[0024] (2) The rain removal module of the present invention is a 3D recursive generative adversarial network. The introduction of the LSTM layer can propagate the features between each iterative convolutional layer, which is conducive to removing rain streaks. The discriminator structure added at the end of the network can guide the generator to remove rain and improve the blurring of details in the rain-free image output by the generator.
[0025] (3) Compared with similar works, the present invention uses transfer learning to construct a self-supervised network, which can more accurately extract rain streaks in real scenes and obtain high-quality rain-free images with good generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 This is a flow chart of the method for detecting and removing rain streaks in light field images based on transfer learning;
[0027] Figure 2 Calculate the flow chart for the depth map;
[0028] Figure 3 This is the structure diagram of the rain streak detection network;
[0029] Figure 4 A diagram of the network structure for stormwater removal;
[0030] Figure 5 The results of synthetic data experiments are shown in Figure 1. The evaluation index is PSNR / SSIM. (a) is the input light field image with rain; (b) is the rain-free image obtained by this method; (c) is the true value image of the light field image; (d) is the rain streak image obtained by this method; (e) is the true value image of the rain streak; (f) is the depth image obtained by this method.
[0031] Figure 6 The figures are the experimental results of real scene data, where (a) is the input light field image with rain; (b) is the rain-free image obtained by this method; (c) is the rain streak image obtained by this method. DETAILED DESCRIPTION
[0032] The present invention is further described below in conjunction with the accompanying drawings and preferred embodiments. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and process are given, but the protection scope of the present invention is not limited to the following embodiments.
[0033] like Figure 1FIG. 1 is a flow chart of a method for detecting and removing rain streaks from a light field image based on transfer learning of the present invention. The method for detecting and removing rain streaks from a light field image based on transfer learning of the present invention includes the following three modules:
[0034] 1. Depth map calculation module
[0035] The depth map calculation module of the present invention uses the existing depth map calculation network EPINET for calculation, such as Figure 2 The depth map calculation flow chart is shown in FIG. The depth map calculation network EPINET takes 3DEPI volume blocks based on four directions of horizontal, vertical, left diagonal and right diagonal as input, first constructs a multi-stream network to extract features in each direction, then connects the features extracted in each direction together and inputs them into the fusion network to calculate the depth map.
[0036] Furthermore, the depth map calculation module sequentially inputs the synthetic data and the real scene data to obtain the synthetic data depth map D s And the real scene data depth map D r , the calculation formula is as follows:
[0037]
[0038] Among them, f EPINET (·) represents the convolution operation of the EPINET network; B EPI Indicates the true value of the synthetic data without rain; since there is no true value without rain in the real scene, the 3D EPI volume block with rain is directly used. EPI as input.
[0039] 2. Rain streak detection module
[0040] 2.1 Network Structure
[0041] In the rain streak detection module, the present invention constructs a 3D residual network based on Gaussian Process (GP), such as Figure 3 The figure shows the structure of the rain streak detection network. The input I size is (3, 9, 128, 128), where 3 represents three channels, 9 represents the depth, i.e. the number of sub-viewpoints, and 128×128 is the sub-viewpoint size. The middle part of the network consists of 5 layers of standard residual blocks, the convolution kernel size is set to 3×3×3, the activation function is set to the linear rectifier function (ReLU), and the activation function of the last layer is the Sigmoid function. Each layer of residual blocks is followed by a GP module, and the extracted synthetic data features and real scene data features are modeled to obtain the pseudo-true value of the real scene data, which is further used to supervise the network to extract the real scene data features.
[0042] Figure 3 middle, Is is the input synthetic data, I r is the input real scene data, R s is the output synthetic data rain streak, R r is the rain streak of the real scene data output, F zs [i] stores all intermediate feature vectors extracted from the i-th layer The matrix of .
[0043] 2.2 Synthetic Data Training Phase
[0044] In this stage, the rain streak detection network is trained using synthetic data to minimize the supervised loss function:
[0045]
[0046] Among them, λ p is a constant, is the l1 loss, is the perceptual loss, defined as:
[0047]
[0048]
[0049] in, is the true value of rain streaks, and VGG(·) is the convolution operation of the pre-trained VGG-16 network.
[0050] In addition to minimizing the loss function, during the training process, all the intermediate feature vectors of the synthetic data are Stored in matrix F zs [i], that is N s is the total number of synthetic data.
[0051] 2.3 Real scene training phase
[0052] After learning weights using synthetic data, we use real scene data to update the weights. We use real scene data to improve the generalization ability of the network. Specifically, we first use GP to jointly model the synthetic data F zs and real scene data z r . Then by minimizing z r and pseudo-truth value z r,pseudo The error between and provides supervision for the intermediate latent space of the network.
[0053] Modeling process: First, use synthetic data z s Represents the observed samples of the real scene data corresponding to the kth training
[0054]
[0055] Among them, α n is a constant, ε is the added noise
[0056] Using formula (2-4), GP is used to jointly model the distribution of synthetic data and real scene data. The conditional joint probability distribution will form the following conditional multivariate Gaussian distribution for the real scene data:
[0057]
[0058] Depend on Get the pseudo-true value of the real scene data feature of the kth training. The GP module connected to each residual block of the network can get a pseudo-true value In order to retain the Gaussian distribution parameters obtained by the GP module in the shallow network, the pseudo-true value obtained is passed backward, and the formula is expressed as:
[0059]
[0060] Therefore, the self-supervised loss function used in the real scene training process is defined as follows:
[0061]
[0062] 2.4 Network Overall Loss Function
[0063] The overall loss function used to train the network is defined as follows:
[0064]
[0065] Among them, λ r is a predefined weight used to adjust and of the total losses.
[0066] 3. Rainwater removal module
[0067] 3.1 Network Structure
[0068] The rain removal module is a 3D recursive generative adversarial network, such as Figure 4The figure shows the structure of the rain removal network. The generator uses a recursive residual network (ResNet) with 4 iterations. Each residual layer (ResBlock) contains 2 convolutional layers, which are then activated using the ReLU function. The convolution kernel size of all convolutional layers is 3×3×3, and no downsampling or upsampling operations are required. A long short-term memory network (LSTM) layer is introduced. All convolutions of the LSTM have 32 input channels and 32 output channels. This layer can propagate the features between each iterative convolutional layer, which is conducive to removing rain streaks. At the same time, a discriminator structure is added at the end of the network to guide the generator to remove rain and improve the blurred details of the rain-free image output by the generator.
[0069] Figure 4 In the figure, I is the input rainy 3DEPI volume block, D is the depth map obtained by the depth map calculation module, and R is the rain streak map extracted by the rain streak detection module. It should be noted that I, D, and R here include both synthetic data and real data.
[0070] 3.2 Loss Function
[0071] Existing image deraining methods use an additive model, that is, the rain image x is considered to be the superposition of the rain component r of the clean image y:
[0072] x=y+r (3-1)
[0073] Then, from formula (3-1), we can get y = xr. In the rain streak detection module, the detected rain streak image r is obtained. Since the real scene data has no true value, the input rain image x is subtracted from the rain streak image r and normalized as the pseudo-true value of the clean image of the real scene data.
[0074] The rainy 3DEPI volume block I, the depth map D obtained by the depth map calculation module, and the rain streak map R extracted by the rain streak detection module are connected in series and input into the generator for rain removal. The loss function of the generator is the structural similarity loss (SSIM) and the perceptual loss formula is shown in the following formulas (3-2) and (3-3), respectively:
[0075]
[0076]
[0077] Among them, y is the clean image output by the generator, y gt is the corresponding truth value.
[0078] The discriminator receives the output of the generator and evaluates whether it looks like a real clear rain-free scene. The discriminator loss function is:
[0079]
[0080] Where D(·) is the convolution operation of the discriminator network.
[0081] In summary, the overall loss function used to train the network is defined as follows:
[0082]
[0083] where λ p,g ,λ gan They are The weight of the total loss.
[0084] In order to fully illustrate the effect of the method of the present invention on rain streak detection and removal, the present invention is demonstrated through the following specific experiments:
[0085] The angular resolution of the input LFI is 9×9 and the spatial resolution is 512×512. Training was performed on a PC equipped with an i7-7700K CPU, 24GB of memory and an NVIDIARTX TITAN GPU, 24GB. Due to the memory size limitation of the NVIDIARTX TITAN GPU, the rainy LFI was cut into small blocks of size 128×128 with a step size of 64. The network was implemented using pytorch1.5.0 and Python 3.6.0, with the number of iterations set to 500 and the batch size set to 2. Adam optimization was used to train the network, with the learning rate of the rain streak detection module set to 0.0002, the learning rate of the rain removal module generator set to 0.0001, and the discriminator learning rate set to 0.0004. In formula (2-1), λ p Set to 0.04; λ in formula (2-8) r Set to 0.015; in formula (3-5) Set to 0.04 and 0.01 respectively.
[0086] First, the depth map is obtained by the depth map calculation module, and then the synthetic data rain streak map is detected by the rain streak detection module. At the same time, the Gaussian process module is used to self-supervise the detection of the real scene data rain streak map. Finally, the rain removal module is entered. The rainy 3DEPI volume block, the depth map obtained by the depth map calculation module and the rain streak map extracted by the rain streak detection module are connected in series and input into the 3D recursive generative adversarial network to remove rain. The training is repeated until a high-quality rain-free image is obtained.
[0087] Among them, for synthetic data, two typical evaluation indicators, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), are used to quantitatively evaluate the light field image rain streak detection and removal method based on transfer learning proposed in this invention. Figure 5 The following is the result of the synthetic data experiment. Figure 5It can be seen that in column (a), when the input has dense rain streaks and thick fog, the present invention can accurately extract rain streaks, as shown in column (d). From the rain removal results in column (b), it can be seen that the rain-free image obtained by the present invention has a high PSNR value and SSIM value, and can effectively remove rain streaks, repair the background, and obtain a high-quality rain-free image.
[0088] For real scene data, the proposed method is qualitatively evaluated, such as Figure 6 This is the experimental result diagram of real scene data. Figure 6 It can be seen from columns (b) and (c) that the present invention can also accurately extract rain streaks in real scenes and obtain clean rain-free images, which proves that the present invention has good generalization ability.
[0089] The above is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited thereto. All equivalent changes, simplifications and modifications made according to the scope of the present invention without violating the principles of the present invention shall fall within the scope of the present invention.
Claims
1. A method for detecting and removing rain streaks in light field images based on transfer learning, characterized in that: The method includes a depth map calculation module, a rain streak detection module and a rain removal module; The depth map calculation module obtains the depth map by calculation; the depth map includes a synthetic data depth map and a real scene depth map; the synthetic data depth map is obtained by synthetic data entering the depth map calculation module; the real scene depth map is obtained by real scene data entering the depth map calculation module; The rain streak detection module detects the synthetic data rain streak image and constructs a 3D residual network based on Gaussian process. The middle part of the network is composed of residual blocks, the activation function is set to linear rectification function, the activation function of the last layer is Sigmoid function, and each layer of residual block is connected to GP module. At the same time, the Gaussian process module is used to self-supervise the detection of the real scene data rain streak image. The rain streak detection module uses a 3DEPI volume block formed by stacking a row of sub-viewpoints of the light field image as input, uses a residual network to extract the features of the rain streaks, and in each convolution layer, uses a Gaussian process to model the extracted synthetic data features and real scene data features to obtain a pseudo-true value of the real scene data, which is further used to supervise the network to extract the features of the real scene data; The rain removal module connects the rainy 3DEPI volume block, the depth map obtained by the depth map calculation module, and the rain streak map extracted by the rain streak detection module in series, and then inputs them into the 3D recursive generative adversarial network to remove rain and repair the background, and repeatedly trains until a high-quality rain-free image is obtained; The rainwater removal module is provided with an LSTM layer and a discriminator structure; the LSTM layer is used to propagate features between each iterative convolutional layer.
2. According to the method for detecting and removing rain streaks in light field images based on transfer learning in claim 1, it is characterized in that: The depth map calculation module first uses a multi-stream neural network to extract information of light field sub-viewpoints from different directions, and then enters a fusion network to calculate the depth map.
3. According to the method for detecting and removing rain streaks in light field images based on transfer learning in claim 1, it is characterized in that: The rain streak map includes a synthetic data rain streak map and a real scene rain streak map; wherein the synthetic data rain streak map is obtained by synthetic data entering a rain streak detection module; the real scene rain streak map is obtained by real data entering a rain streak detection module.
4. According to the method for detecting and removing rain streaks in light field images based on transfer learning in claim 1, it is characterized in that: In each iteration, the long short-term memory network is used to transfer features.
5. According to the method for detecting and removing rain streaks in light field images based on transfer learning in claim 1, it is characterized in that: The loss function of the discriminator is: L gan =-log(D(y))-log(1-D(y gt )) Among them, y is the clean image output by the generator, y gt is the corresponding true value, and D(·) is the convolution operation of the discriminator network.
