A bidirectional rain control image processing method from removal to rendering

Abstract

The present invention discloses a two-way rain control image processing method from removal to rendering. The steps of the method are: constructing a database including a training set and a test set; inputting a rain image into a background extraction network and a high-frequency rain removal network, and outputting a background image containing background information and a fog image containing background and fog information respectively; constructing a loss function of a main control network, and the loss function includes a directional gradient histogram loss function and an autocorrelation loss function; inputting a rain image, a background image, and a fog image into the main control network, and using a multi-branch structure to train and output rain images of different levels from no rain to heavy rain; in the test phase, through feature space interpolation, rain images of any intermediate different rainfall intensity levels can be obtained. The method of the present invention can realize the input of a single rain image, and the user can control the acquisition of any intermediate intensity rain image from completely no rain to heavy rain, and in this change process, scene-specific rain features such as the direction, appearance, and distribution of rain streaks are retained.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and deep learning, and in particular relates to a bidirectional rain control image processing method from removal to rendering. Background Art

[0002] In the field of image processing, the conversion of weather conditions is an important and challenging topic. Specifically, the research on image conversion between rainy and rainless weather includes two aspects: rain removal and rain rendering. The aspect of rain removal focuses on the study of rain removal for rainy images. However, although some rain removal methods can generate high-quality rain removal images, they cannot obtain images of different intermediate rain intensities through control. In addition, some multi-stage step-by-step rain removal methods have been proposed. However, the intermediate stage output of these methods is a partial rain removal image, which is inconsistent with the effect of medium-intensity rain images. The aspect of rain rendering can achieve the rendering of simulated rain on rainless images. However, rain rendering methods are highly dependent on physical parameters such as scene structure, lighting conditions, wind speed and direction, and camera settings. These limitations make it impossible to render rain effects with the same characteristics based on the input rain image. In general, both the current methods of rain removal and rain rendering only focus on directly converting from one weather condition to another, leaving an innovative problem, that is, how to achieve two-way continuous control from complete rain removal to generating heavy rain by inputting a single rain image.

[0003] Different intensities of rain in an image have different performances in terms of rain streak density, fog thickness and other features. At the same time, some scene-specific features should remain unchanged under different rainfall intensities, including local rainfall direction, appearance and distribution of rain streaks, etc. These features that do not change with intensity mainly depend on wind direction, illumination direction and camera parameters (such as focal length, focus and exposure time, etc.). Simply generating rain maps of different rainfall intensities will ignore these intensity-related and irrelevant rain features, making the rain changing process lack authenticity. Therefore, it is an urgent problem to provide a rain control image processing method that can change the intensity-related rain features and retain the intensity-independent rain features while adjusting the rain intensity. Summary of the invention

[0004] In view of the problem that current image weather conversion methods can only achieve one-way rain removal or rain rendering, the present invention provides a two-way rain control image processing method from removal to rendering to generate rain maps with different continuous intensities of rainfall from complete rain removal to heavy rainfall, and retain scene-specific rain characteristics during the rain control process.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A two-way rain control image processing method from removal to rendering comprises the following steps:

[0007] Step 1, construct a database, including a training set and a test set, each sample in the training set includes a rainless background image, a fog image, and rain images of multiple different rainfall intensity levels;

[0008] Step 2: Input the original rain image in the training set into the background extraction network, the network outputs the background image containing background information and transmits it to the main control network; input the original rain image into the high-frequency rain removal network, the network eliminates the high-frequency rain streaks in the rain image, outputs the fog image containing background and fog information, and transmits it to the main control network;

[0009] Step 3, constructing a loss function of the main control network, the loss function including a directional gradient histogram loss function and an autocorrelation loss function;

[0010] Step 4, the main control network uses a multi-branch structure to train and output rain maps of different rainfall intensity levels according to the original rain map, the background map containing background information in step 2, and the fog map containing background and fog information;

[0011] Step 5, in the test phase, the rain map in the test set is input into the trained main control network, and the background image and the fog image are output respectively through the background extraction network and the high-frequency rain removal network, and transmitted to the main control network; in the main control network, any intermediate level rain map between the different rainfall intensity levels in step 1 can be obtained through feature space interpolation.

[0012] Compared with the prior art, the present invention has the following advantages:

[0013] (1) The image processing method of the present invention can control rain intensity. For a single input rain image, it can realize two-way continuous rain control from rain removal to rain rendering, thereby increasing the diversity and flexibility of image weather conversion tasks.

[0014] (2) In the process of controlling rain patterns with different intensities, the present invention can retain scene-specific rain features such as the direction, appearance, and distribution of rain streaks, thereby generating a more natural and realistic rain pattern effect. The main control network loss function of the present invention includes a directional gradient histogram loss function and an autocorrelation loss function, which can constrain the input and output rain patterns to have the same rain streak direction features and reduce periodic repeated rain streaks.

[0015] (3) The training of different types of rain image data sets increases the universality of the present invention in changing various types of rain. In the test process of simulation and real rain images, the present invention can achieve different intensities of rain control from rain removal to rain rendering, which further verifies the effectiveness of the method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic flow diagram of the method of the present invention;

[0017] Figure 2 It is the overall network structure diagram of the method of the present invention;

[0018] Figure 3 It is an introduction diagram of each network layer of the method of the present invention;

[0019] Figure 4 It is a graph of the rain removal experiment results of the method of the present invention on the Rain12000 and RainLevel5 datasets.

[0020] Figure 5 It is a graph of the rain variation experiment results of the method of the present invention on the RainLevel5 dataset.

[0021] Figure 6 This is a diagram showing the result of a rain variation experiment using the method of the present invention on a real rain map.

[0022] Figure 7 It is a graph of the experimental results of interpolation of any intermediate rain level obtained by using the method of the present invention. DETAILED DESCRIPTION

[0023] The following is a detailed description of an embodiment of the present invention. This embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation method and a specific operation process are given, but the protection scope of the present invention is not limited to the following implementation.

[0024] Figure 1 : is a training flow chart of this embodiment, which includes the following steps:

[0025] (1) Construct a database, including a training set and a test set. Each sample in the training set includes a rainless background image, a fog image with only fog effect rendered on the background image, and a rain image with multiple different rain intensity levels with both fog and rain rendered. The test set includes simulated rain images and real rain images. The simulated rain images include a rainless background image, a fog image, and multiple rain images with different rain intensity levels. The real rain images include a single real scene rain image.

[0026] (2) The rain image in the training set is input into the background extraction network, and a background image containing background information is output; the rain image is input into the high-frequency rain removal network, and a fog image containing background and fog information is output.

[0027] (3) Constructing the loss function of the main control network, which includes the directional gradient histogram loss function and the autocorrelation loss function; wherein, the directional gradient histogram loss is constructed to constrain the unified direction, and the autocorrelation loss is constructed to reduce repeated rain streaks.

[0028] (4) Input a rain map, a background map containing background information, and a fog map containing background and fog information into the main control network, and use a multi-branch structure to train and output rain maps of different levels from no rain to heavy rain.

[0029] (5) In the testing phase, the test rain image is input into the trained model, and the background extraction network and the high-frequency rain removal network are used to output the background and fog information respectively. In the main control network, any intermediate level rain image can be obtained through feature space interpolation.

[0030] 1. In this embodiment, the database part involved in step (1) is described in detail:

[0031] This method constructs a dataset named RainLevel5 to train the rain intensity control network. The dataset contains 29750 pairs of rain images and clean background images, of which 26870 pairs are used for training and 2880 pairs are used for testing. The clean background image is taken from the Cityscapes dataset, and the rain image contains 5 intensity levels: 25, 50, 75, 100, and 200 mm / hr. These rain images are synthesized by a rendering method based on physical principles and a Photoshop-based drawing software. Through these two synthesis methods, rain streaks of different appearance types can be generated to ensure the generalization ability of the trained network. In order to simulate the rainy day effect more realistically, fog of the corresponding level is also rendered into the rain image, and fog images without rain streaks but only containing background and fog information are also rendered to train the high-frequency rain removal network. In this embodiment, the datasets Rain12000 and RainLevel5 are trained and tested, and the deraining results of this method and other methods are compared using structural similarity (SSIM) and peak signal-to-noise ratio (PSNR). In the rain variation experiment, this method demonstrated the rain variation effect on the simulated rain image and real rain image of the RainLevel5 test set.

[0032] 2. In this embodiment, the background extraction network and high-frequency rain removal network involved in step (2) are described in detail:

[0033] The background extraction network is used to extract background information and provide the intermediate information to the main control network for rain rendering. The design structure of the background extraction network is Generative Adversarial Networks (GAN), in which the generator part consists of coarse processing and fine processing networks, and the discriminator (Discriminator, D) is used to evaluate the similarity between the output of the generator and the target rain-free image. The original rain image is input, and the rain streaks are initially removed by the coarse processing network. Then the original rain image and the preliminary rain streak removal image are input to the fine processing network, and finally the refined rain removal image is output. Both the coarse processing network and the fine processing network are codec structures. In order to achieve better rain streak removal, 5 groups of residual and dense modules are inserted in the middle of the codec network in the coarse processing network. In order to fully display the background details in the rain image generation process, the skip-layer features in the fine processing network of the background extraction network are transmitted to the corresponding skip-layer of the main control network. In addition, the refined derained image is not only transmitted to the input of the main control network, but also transmitted to the rain-free branch of the parallel gating module in the main control network, which is used to generate rain-free images. The nearest neighbor interpolation is used to downsample the refined derained image to the feature size corresponding to these branches. Since the background extraction network is designed to extract background information, both rain and fog information are removed in this process. Therefore, a high-frequency rain removal network is designed, which only eliminates high-frequency rain streaks and retains fog information, which is provided to the main control network, thereby achieving scene-specific fog effect generation in the process of rendering rain images.

[0034] 3. In this embodiment, the loss function involved in step (3) is described in detail:

[0035] In the process of generating rain images of different levels, scene-specific physical properties such as the direction of rain streaks should be preserved. In order to preserve this directional information, given that the histogram of oriented gradients (HOG) can learn the direction of rain streaks, this method proposes to use the HOG loss function as one of the loss functions, thereby constraining the input and output rain images to have the same distribution of rain streak directions. Specifically, this method uses the Sobel operator to calculate the horizontal and vertical gradient maps (G x ,G y ), the direction of the gradient map is determined by Since the distribution of rain direction in local areas is mostly consistent, the image is divided into K units of the same size. In this method, K is 512. In each unit, the direction [0,π) is divided into 180 intervals to statistically weight the direction histogram. The weight is the size of the gradient, that is, In order to better maintain the invariance to noise and illumination, the L1 norm is used to normalize the original direction histogram O org :O=O org / ||O org|| 1. The normalized direction histogram of the true and actual output of the kth unit is marked as and In order to measure the difference between the two distributions, the average KL divergence (KullbackLeibler divergence) is used to calculate the HOG loss function L in the network h :

[0036]

[0037] During the experiment, the generated rain streaks appeared to repeat periodically in the horizontal and vertical directions. To solve this problem, this method proposed an autocorrelation loss function to reduce the repeatability of rain streaks. In order to reduce the influence of the background, especially in scenes with periodic patterns, the rain streak map is used for further calculations, which is obtained by subtracting the background map from the input rain map. Since most rain streaks are vertical stripes, this method uses the horizontal gradient of the rain streak map to calculate the autocorrelation. In the experiment, since the peaks only appear significantly in the horizontal and vertical directions of the autocorrelation function, this method only takes the normalized autocorrelation coefficients ac in the horizontal and vertical directions x , ac y To construct the loss function. Due to the periodic repetition of rain streaks, obvious peaks appear in the autocorrelation coefficient. In order to reduce this repetitiveness, this method designs an autocorrelation loss function to detect and constrain the first N peaks:

[0038]

[0039] where max i represents the index of the i-th largest coefficient, Represents the autocorrelation coefficient ac x and ac y Considering that the autocorrelation coefficients close to zero are relatively high, these coefficients will affect the calculation of the mean, so ac x and ac y The near-zero region coefficients of are not used to calculate the average value. The near-zero region in this method is defined as the region from the zero point to the first local minimum.

[0040] 4. In this embodiment, the main control network part involved in step (4) is described in detail:

[0041] The main control network is the main network for realizing rain intensity control. It is also designed based on the generative adversarial network (GAN). The generator part includes a codec network, a parallel gating module and a noise enhancement module. The discriminator part is used to evaluate the similarity between the generator output and the target level rain map. The original rain features processed by the encoder of the codec network are input to the parallel gating module. The multi-branch structure of the parallel gating module outputs rain features of different intensities respectively, and then the decoder of the codec network outputs the final rain map of different rainfall intensity levels. The overall network structure of the present invention is as follows: Figure 2 As shown in the figure, the specific introduction of each network layer is as follows Figure 3 As shown, Figure 3 Here, C and DC represent convolution and deconvolution respectively, ReLU is the activation function Rectified Linear Unit, and BN is the normalization operation Batch Normalization.

[0042] The parallel gating module in the main control network is mainly used to realize the control of rain of different intensities. The parallel gating module is located in different jump layers of the codec network. During the training process, the method realizes discrete level training through the parallel gating module, that is, outputs rain maps of several specific intensity levels in turn. During the test process, the rain map of any intermediate intensity is obtained by weighting between adjacent gating structures. In order to better utilize the features of different scales, the parallel gating module is designed in the jump layers of different scales of the codec. Each parallel gating module consists of multiple branches, and different branches correspond to rain maps of different intensities from no rain to the maximum level of rain. The structure of each parallel gating module can be expressed as f(B)=∑ i g i t i (B),∑ i g i = 1, where B is the rain feature in the skip layer, t(·) is a 1×1 convolution, which is used to convert B into a certain intensity of rain, g i is the gating parameter that controls the weight of each branch. In the training stage, the output rain map is a rain map of N discrete intensities, that is, i = 0, 1, ..., N-1. In this method, N is 6, and the corresponding gating parameter {g i}(i∈0,1,...,N-1) is binary, and when the i-th level rain map is output, only the corresponding i-th branch is opened, that is, g i = 1, and the rest of the branches are closed. During the test phase, the gating module enables the network to generate a continuous intensity rain map through a linear combination of two adjacent gating branches. Specifically, for any intermediate level s between two specific discrete levels i and i1, the interpolated rain map can be output by the following method:

[0043]

[0044] Among them l i , l i+1 and l s Indicates the corresponding level of rain intensity.

[0045] In the main control network, the noise enhancement module is used to add noise multiplied by a learnable parameter to the convolution and deconvolution layers of the codec network to enhance the randomness of the rain streak distribution and the diversity of the texture; and when outputting each intensity of rain, the noise is multiplied by a scaling parameter of the corresponding size to further enhance the adjustment of the interpolation effect, thereby generating a rain map with a more reasonable level. The specific processing steps are: first, the Gaussian noise image is repeatedly expanded at the channel level, and the number of channels is the same as the number of channels added to the target feature, then the expanded noise image is multiplied by the scaling parameter corresponding to the target rain intensity level, and then each channel of the noise image is multiplied by a learnable parameter, and finally this weighted noise image is added to the corresponding feature in the main control network. During the training process, when the target output rain intensity level changes from 0 to N-1, the noise is multiplied by the scaling parameters 0, 1 / (N-1), 2 / (N-1),..., 1 respectively. During the test process, the noise is multiplied by s / (N-1) to generate an s-level rain map with intensity levels between 0 and N-1.

[0046] The training of the overall network is divided into two steps: I. training the background extraction network and extracting background information, II. fixing the background extraction network parameters and training the high-frequency rain removal network and the main control network together.

[0047] Specifically, in the first stage, in addition to the GAN loss In addition, MAE loss L l1 , sensory loss L p and SSIM loss L s It is also used to constrain the coarse processing network C1 and the fine processing network C2 of the background extraction network. The target real background image is represented as C gt , then the generator loss function of training the background extraction network is:

[0048]

[0049] in Denotes the discriminator network D from the background subtraction network b The adversarial loss, D b Trained using the discriminator loss function: λ s ,λ p are the corresponding weight coefficients, which are 100, 100 and 300 respectively.

[0050] In the second stage, the high-frequency rain removal network is trained together with the main control network, and its generator loss function is:

[0051]

[0052] in I and I gt is the output image of the main control network and the target real image, L h is the HOG loss function, L ac is the autocorrelation loss function, I′ and I′ gt Represents the output of the high-frequency rain removal network and the ground-truth target image with only background and fog information. λ h ,λ ac are the weight coefficients of the corresponding loss functions, which are 100, 0.01, and 10 respectively. The discriminator network D of the main control network m Training using the discriminator loss function:

[0053] The network of this method uses the Adam optimizer, and the learning rate is initialized to 0.0002. The total number of training epochs for the background extraction network is 15, and the learning rate is divided by 2 after 10 epochs. The total number of training epochs for the high-frequency rain removal network and the main control network is 6.

[0054] 5. In this embodiment, the network test part involved in step (5) is described in detail:

[0055] In the test phase, the trained network parameters are first loaded, and then the test rain image is input into the network. The test rain image is passed through the background extraction network to obtain the background image, and then through the high-frequency rain removal network to obtain the fog image. Figure 1 Input the main control network and use the method in the test process mentioned in step (4) to adjust the parameters to obtain any level of rain map.

[0056] See also Figure 4 The proposed method compares the derained image output by the background extraction network with the deraining methods such as RESCAN, UMRL, PReNet, SEMI, JORDER, MSPFN and RCDNet, and uses the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) to measure the similarity between the derained image and the target rain-free image. The comparison is conducted on the datasets RainLevel5 and Rain12000, and the analysis results show that compared with other deraining methods, the proposed method achieves better deraining indicators on both datasets.

[0057] See also Figure 5The present invention demonstrates the variable rain effect on the dataset RainLevel5, proving the ability of the present method to achieve two-way rain control and maintain scene-specific rain characteristics. Input a rain map of any rain intensity in the white box, and the present method can output a rain map with a lower or higher intensity than the original intensity, so as to achieve a two-way rain control effect of controllable rain removal and rain rendering. Figure 5 , scene-specific rain characteristics can also be preserved during the transformation process. For example, the appearance of rain streaks in the input and output rain images remains consistent, and the direction and distribution of the input rain are also well learned and preserved. In addition, the fog effect in the original rain image is preserved during the transformation process, and the thickness of the fog changes with the rain intensity, which further enhances the realism of the output rain image. See Figure 6 ,The present invention demonstrates the bidirectional rain control effect on a real rain ,map, which further proves that the method can input a single rain map, output a rain removal map, or generate a rain map of ,arbitrary intensity, and the generated rain can retain the rain characteristics of a specific ,scene.

[0058] See also Figure 7 The present invention adjusts g in formula (3) i , using the spatial interpolation method, the output of any intermediate level rain map is realized. Figure 7 As shown in the figure, the intensity of the input rain map is 100 mm / hr. In addition to outputting rain maps of determined rain intensities (100 mm / hr and 200 mm / hr), this method can also generate rain maps of any intermediate interpolation levels (100 mm / hr to 200 mm / hr, and this method shows three results: 125 mm / hr, 150 mm / hr, and 175 mm / hr), and these interpolation results also retain the same rain characteristics as the input rain map.

Claims

1. A two-way rain control image processing method from removal to rendering, characterized in that: The following steps are involved: Step 1, construct a database, including a training set and a test set, each sample in the training set includes a rainless background image, a fog image, and rain images of multiple different rainfall intensity levels; Step 2: Input the original rain image in the training set into the background extraction network, the network outputs the background image containing background information and transmits it to the main control network; input the original rain image into the high-frequency rain removal network, the network eliminates the high-frequency rain streaks in the rain image, outputs the fog image containing background and fog information, and transmits it to the main control network; Step 3, constructing a loss function of the main control network, the loss function including a directional gradient histogram loss function and an autocorrelation loss function; Step 4, the main control network uses a multi-branch structure to train and output rain maps of different rainfall intensity levels using the loss function according to the original rain map, the background map containing background information in step 2, and the fog map containing background and fog information; Step 5: During the test phase, the rain map in the test set is input into the trained main control network, and the background extraction network and the high-frequency rain removal network are used to output the background map and the fog map respectively, and transmit them to the main control network; in the main control network, any intermediate level rain map between different rainfall intensity levels is obtained through feature space interpolation.

2. A two-way rain control image processing method from removal to rendering according to claim 1, characterized in that: In step 1, the test set includes simulated rain images and real rain images, wherein the simulated rain images include rainless background images, fog images, and rain images of multiple different rainfall intensity levels, and the real rain images include a single real scene rain image.

3. The two-way rain control image processing method from removal to rendering according to claim 1, characterized in that: In step 3, the steps for constructing the directional gradient histogram loss function are: inputting the original rain map into the main control network to obtain the output rain map, calculating the directional gradient histogram of the output rain map and the corresponding target rain map, and then using the KL divergence to measure the difference between the two gradient directional histograms as the final loss value.

4. The two-way rain control image processing method from removal to rendering according to claim 3 is characterized in that: Histogram of Oriented Gradients Loss Function L h : Among them, K represents the unit of image division, the true direction histogram and the normalized direction histogram of the actual output k-th unit are marked as and KL() stands for KL divergence.

5. The two-way rain control image processing method from removal to rendering according to claim 1, characterized in that: Autocorrelation loss function L ac : Where N represents the number of the largest peaks in the autocorrelation coefficient, ac x,y Represents the autocorrelation coefficient ac x and ac y , max i represents the index of the i-th largest peak, Represents the autocorrelation coefficient ac x and ac y The average value of .

6. The two-way rain control image processing method from removal to rendering according to claim 1, characterized in that: In step 4, the main control network includes a codec network, a parallel gating module and a noise enhancement module; the parallel gating module is located in different jump layers of the codec network, the original rain features processed by the encoder of the codec network are input to the parallel gating module, the multi-branch structure of the parallel gating module outputs rain features of different intensities respectively, and then the decoder of the codec network outputs the final rain map of different rainfall intensity levels; The noise enhancement module is used to add the noise multiplied by the learnable parameter to the convolution and deconvolution layers of the codec network, and multiply the noise by a scaling parameter of a corresponding size when outputting a rain map of each rainfall intensity level.

7. A two-way rain control image processing method from removal to rendering according to claim 6, characterized in that: The processing steps of the noise enhancement module are as follows: first, the Gaussian noise image is repeatedly expanded at the channel level, and the number of channels is the same as the number of channels of the target added features, then the expanded noise image is multiplied by a scaling parameter corresponding to the target rain intensity level, and then each channel of the noise image is multiplied by a learnable parameter, and finally this weighted noise image is added to the corresponding feature in the main control network.

8. The two-way rain control image processing method from removal to rendering according to claim 6, characterized in that: In the step 5, first, the background extraction network is used to output a background image containing background information, and the background image is compared with the actual background image in the test set to obtain structural similarity and peak signal-to-noise ratio for measuring the similarity of the two images to quantify the deraining effect; then the rain image in the test set is input into the high-frequency rain removal network, and the network outputs a fog image containing background information and fog information; the main control network controls the feature space interpolation through parameters according to the test rain image, the input background image and the fog image, and performs weighted linear addition on the features of the two adjacent level branches in the parallel gating module, thereby obtaining any intermediate level rain image.

9. The two-way rain control image processing method from removal to rendering according to claim 6, characterized in that: The background extraction network includes a coarse processing network and a fine processing network. The input original rain image is first processed by the coarse processing network to preliminarily remove rain streaks. Then, the original rain image and the rain image after the preliminary removal of rain streaks are input into the fine processing network to obtain a refined rain removal image. The refined derained image is then transferred to the input of the main control network and the rain-free branch of the parallel gating module; The jump layer characteristics in the fine processing network are transmitted to the corresponding jump layer of the main control network.

Images