Physical perception-oriented unsupervised rain removal network construction method

Abstract

The invention provides a physical perception-oriented unsupervised rain removal network construction method. The method comprises the steps of S1, constructing a rainwater image generation branch comprising two sub-networks; s2, constructing a clear image generation branch; s3, obtaining a clear image feature B'by using a physical model according to the input feature O and the rainwater feature R, and calculating the clear consistency loss of the clear image B 'and B; s4, adding the clear image B and the rainwater layer R to obtain a rainwater image O ', and calculating content consistency loss with the input image O; and S5, constructing bidirectional global local contrast loss by using the non-paired data set and the images output by the two networks so as to achieve a better image reconstruction effect. According to the method, the problem that the physical interpretability of information in a feature space is insufficient due to the fact that most existing unsupervised rain removal methods do not make full use of a physical model for modeling in the feature space is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method for constructing an unsupervised rain removal network for physical perception. Background Art

[0002] In recent years, the intensification of environmental pollution and frequent severe weather conditions have caused the images captured by optical sensors to often have quality problems such as color distortion, blur, and low contrast under rainy conditions. These distortion phenomena not only affect the overall quality of the image, but also bring difficulties to image analysis. With the rapid development of deep learning technology, advanced visual tasks such as target detection, target tracking, and image segmentation have played an increasingly important role in fields such as autonomous driving, drone inspection, and security monitoring. Therefore, it is particularly important to develop an efficient image deraining algorithm to obtain clear images as input for advanced visual tasks. By solving the problem of image deraining, the efficiency and accuracy of a variety of real-world applications can be improved, thereby improving people's production and living quality.

[0003] At present, image deraining methods can be divided into two categories: artificial prior-based methods and deep learning-based methods. Most of the early deraining research mainly relied on the prior information of the image itself to effectively remove rain streaks, such as layer prior, low-rank appearance model and frequency domain decomposition model. In recent years, deep learning-based methods have been widely used in many fields such as image deraining, image classification and face recognition. In the field of image deraining, convolutional neural network (CNN) is widely used for natural image deraining and has made significant progress in this field. Learning-based deraining methods can be divided into two categories: supervised deraining and unsupervised deraining. Traditional supervised deraining algorithms rely on a large number of paired rainy and rain-free images for training. However, it is quite challenging to obtain these paired data in practical applications. In order to solve this problem, researchers began to explore image deraining algorithms based on unsupervised learning. These methods can train models without paired data and achieve effective removal of rain streaks. Many researchers have proposed many unsupervised methods based on the CycleGAN framework. Jin et al. published "Unsupervised single image deraining with self-supervised constraints", which proposed an unsupervised single image deraining method using self-supervised constraints. Through the intrinsic statistical characteristics of unpaired data, the rain streaks were effectively removed. Yu et al. published "Unsupervised image deraining: Optimization model-driven deep cnn", which used the statistical characteristics of raindrop directions to construct an optimization model to separate the rain layer and the image layer, thereby improving the network's ability to reconstruct clear images. Chen et al. published "Unpaired deep image deraining using dual contrastive learning", which encouraged the generator to receive data that was more in line with expectations through the use of a bidirectional collaborative branch, thus avoiding the generator from being affected by data with different distributions, thus better promoting rain removal and helping image restoration. Although these methods have achieved certain deraining effects on natural images, they all ignore the interpretability of information in the feature space and do not fully model the physical model in the feature space, resulting in the inability to obtain feature information useful for reconstructing clear images. In the end, there is a large amount of rain residue in the generated image. Summary of the invention

[0004] The present invention provides a method for constructing an unsupervised rain removal network for physical perception, which solves the problem that most of the current unsupervised rain removal methods do not fully utilize physical models for modeling in feature space and the physical interpretability of information in feature space is insufficient, thereby resulting in a decrease in the network's ability to remove rain and deviations in the color and texture details of the generated image.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: A method for constructing a physical-aware unsupervised rain removal network includes the following steps: S1. Construct a rainwater layer generation branch for generating the rainwater layer R, including two subnets. The first subnet is used to extract the density information of rainwater, and the second subnet is used to extract the location information of rainwater. S2, build a clear image generation branch to generate a clear image B, and use a discriminator to distinguish the generated clear image from the unpaired real image. The branch consists of an end-to-end image generation network, including a coding layer feature extraction module, a physical perception multi-branch unit, and a decoding layer image restoration module; S3. A clear image feature B' is obtained by using a physical model from the rain image O input by the network and the rain layer R obtained by the rain image generation network; the clear image B obtained by the end-to-end image generation network is added to the rain layer R to obtain the rain image O'; a clear consistency loss constraint is established using the clear image B and the reconstructed clear image feature B', and a discrimination loss constraint is established using the unpaired real image; an image content consistency loss constraint is established using the reconstructed rain image O' and the input rain image O; a bidirectional global contrast loss constraint and a bidirectional local contrast loss constraint are established using the input rain image, the clear image, the reconstructed clear image and the reconstructed rain image; S4. Establish the overall network loss function according to all the loss constraints established in S3, with the goal of minimizing the overall network loss function and optimizing network performance.

[0006] The rain image generation branch constructed by the above S1 is as follows: Subnet 1: first layer of feature extraction of coding layer → second layer of feature extraction of coding layer → third layer of feature extraction of coding layer → fourth layer of feature extraction of coding layer → first layer of recovery module of decoding layer → second layer of recovery module of decoding layer → third layer of recovery module of decoding layer → output raindrop density information D; Subnet 2: first layer of feature extraction of coding layer → second layer of feature extraction of coding layer → third layer of feature extraction of coding layer → fourth layer of feature extraction of coding layer → first layer of recovery module of decoding layer → second layer of recovery module of decoding layer → third layer of recovery module of decoding layer → output raindrop position information P; Synthesizing the final raindrop features: D P→get the final fine rainwater layer R.

[0007] The clear image generation branch constructed in the above S2 is as follows: Input image O→first layer of feature extraction of coding layer→second layer of feature extraction of coding layer→physical perception multi-branch unit→third layer of feature extraction of coding layer→physical perception multi-branch unit→fourth layer of feature extraction of coding layer→first layer of recovery module of decoding layer→second layer of recovery module of decoding layer→third layer of recovery module of decoding layer→output reconstructed image B→discriminator constraint ensures that B is a clear image; Constructing a physical-aware multi-branch unit: Input encoding feature F→3*3 Convolution→Activation function Relu→3*3 Convolution→Activation function sigmoid→Feature A with 1 channel→Feature A*Input feature F→Rainwater location information feature F P ; Input encoding feature F→3*3 Convolution→3*3 Convolution→Activation function Relu→3*3 Convolution→Activation function sigmoid→Feature B with 1 channel number→Feature B*Input feature F→Rainwater density information feature F D ; Feature F P , feature F D → Element-wise multiplication → Rain feature F R ; Input encoding feature F→3*3 Convolution→Activation function Relu→3*3 Convolution→Activation function sigmoid→Feature C with 1 channel→Feature C*Input feature F→Feature M→Global average pooling→1*1Convolution→Activation function Relu→1*1Convolution→Activation function sigmoid→ Feature D→Feature D*Feature M→Clear image feature F B ; Input encoding feature F, rain feature F R →Element-wise subtraction →Feature F B ’ Adaptive weight allocator assigns weights , F B →Multiply pixel by pixel; Adaptive weight allocator assigns weights , F B ’ →Multiply pixel by pixel; The two features after weighted multiplication → fusion addition → clear image perception feature J after fusion.

[0008] In S3 above, the loss constraint is constructed as follows: End-to-end image generation image B → adversarial loss → ensure that the generated B is a clear image → B, reconstructed clear image B' → clear consistency loss → improve image clarity → input image O, reconstructed rain image O' → content consistency constraint → ensure content detail consistency → end-to-end image generation B, reconstructed clear image B', input image O, reconstructed rain image O' → bidirectional global-local contrast loss.

[0009] The above five loss function constraints are: 1) Adversarial loss, G B represents the image generator, which mainly converts rainy images into clear images, and D is the discriminator, which mainly distinguishes the generated clear images from the real rain-free images. The specific adversarial loss is defined as follows: ; 2) Content consistency loss, ensuring that the image generator G B The generated clear image is consistent with the rain image input in the dataset in terms of content details. The definition of content consistency loss is as follows: ; Among them, G R represents the rainwater generator and RM represents the rainwater model.

[0010] 3) Clear consistency loss: In order to improve the clarity of the generated clear image, clear consistency loss is used to supervise the image generator G B The consistency between the generated clear image and the clear image reconstructed by the rain model RM. The definition of the clear consistency loss is as follows: ; 4) Bidirectional global contrast loss, which can effectively compensate for the constraints on the clarity and overall color of the generated image. In this way, the network's ability to learn the overall background content features can be significantly improved, and the deraining performance can be further improved. The specific formula is as follows: ; ; ; in and It is a hyperparameter used to balance different losses.

[0011] 5) Bidirectional local contrast loss: The bidirectional local contrast loss improves the network's ability to restore local subtle structures. The specific formula is as follows: ; ; ; in, and Two hyperparameters to balance different losses.

[0012] The overall network loss function in S4 above is expressed as: ; in, , , , as well as is a hyperparameter that balances different losses.

[0013] The present invention provides a method for constructing an unsupervised rain removal network for physical perception, which has the following technical effects: 1) We proposed an unsupervised deraining network for physical perception, established a physical model in the feature space, guided the network to extract richer clear image information in the deep feature space, improved the interpretability of information in the feature space, and established a contrast constraint between the rain image generation network and the end-to-end image generation network through the rain model, thereby strengthening the network's constraint ability on reconstructed images. We achieved high-quality image deraining results, and proved that our proposed algorithms achieved the best performance on two public rain image datasets, Rain100L and Rain800.

[0014] 2) The proposed physical-aware unsupervised rain removal network fully exploits the modeling of physical models in the feature space, accurately extracts the features of potential clear images from the deep feature space, and achieves the effect of rain removal. This avoids the lack of interpretability of information in previous attention-based methods and ignores the problem of fully establishing physical models in the feature space. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The present invention will be further described below in conjunction with the accompanying drawings and embodiments: Figure 1 is an overall network structure diagram of an embodiment of the present invention; Figure 2 This is a structural diagram of the physical perception multi-branch unit proposed in the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solutions and advantages of the present invention clearer, the following content will systematically and completely describe the specific technical solutions of the present invention in combination with the drawings provided according to the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0017] A method for constructing a physical-aware unsupervised rain removal network includes the following steps: S1. Construct a rain image generation branch for generating the rain layer R, including two subnets. The first subnet is used to extract the density information of rain, and the second subnet is used to extract the location information of rain. S2, construct a clear image generation branch to generate a clear image B. The branch consists of an end-to-end image generation network, including a coding layer feature extraction module, a physical perception multi-branch unit, and a decoding layer image restoration module; S3. A clear image feature B' is obtained by using a physical model from the rain image O input by the network and the rain layer R obtained by the rain image generation network; the clear image B obtained by the end-to-end image generation network is added to the rain layer R to obtain the rain image O'; a clear consistency loss constraint is established using the clear image B and the reconstructed clear image feature B', and a discrimination loss constraint is established using the unpaired real image; an image content consistency loss constraint is established using the reconstructed rain image O' and the input rain image O; a bidirectional global contrast loss constraint and a bidirectional local contrast loss constraint are established using the input rain image, the clear image, the reconstructed clear image and the reconstructed rain image; S4. Establish the overall network loss function according to all the loss constraints established in S3, with the goal of minimizing the overall network loss function and optimizing network performance.

[0018] The rain image generation branch constructed by the above S1 is as follows: Subnet 1: first layer of feature extraction of coding layer → second layer of feature extraction of coding layer → third layer of feature extraction of coding layer → fourth layer of feature extraction of coding layer → first layer of recovery module of decoding layer → second layer of recovery module of decoding layer → third layer of recovery module of decoding layer → output raindrop density information D; Subnet 2: first layer of feature extraction of coding layer → second layer of feature extraction of coding layer → third layer of feature extraction of coding layer → fourth layer of feature extraction of coding layer → first layer of recovery module of decoding layer → second layer of recovery module of decoding layer → third layer of recovery module of decoding layer → output raindrop position information P; Synthesizing the final raindrop features: D P→get the final fine rainwater layer R.

[0019] The clear image generation branch constructed in the above S2 is as follows: Input image O→first layer of feature extraction of coding layer→second layer of feature extraction of coding layer→physical perception multi-branch unit→third layer of feature extraction of coding layer→physical perception multi-branch unit→fourth layer of feature extraction of coding layer→first layer of recovery module of decoding layer→second layer of recovery module of decoding layer→third layer of recovery module of decoding layer→output reconstructed image B→discriminator constraint ensures that B is a clear image; Constructing a physical-aware multi-branch unit: Input encoding feature F→3*3 Convolution→Activation function Relu→3*3 Convolution→Activation function sigmoid→Feature A with 1 channel→Feature A*Input feature F→Rainwater location information feature F P ; Input encoding feature F→3*3 Convolution→3*3 Convolution→Activation function Relu→3*3 Convolution→Activation function sigmoid→Feature B with 1 channel number→Feature B*Input feature F→Rainwater density information feature F D ; Feature F P , feature F D → Element-wise multiplication → Rain feature F R ; Input encoding feature F→3*3 Convolution→Activation function Relu→3*3 Convolution→Activation function sigmoid→Feature C with 1 channel→Feature C*Input feature F→Feature M→Global average pooling→1*1Convolution→Activation function Relu→1*1Convolution→Activation function sigmoid→ Feature D→Feature D*Feature M→Clear image feature F B ; Input encoding feature F, rain feature F R →Element-wise subtraction →Feature F B ’ Adaptive weight allocator assigns weights , F B →Multiply pixel by pixel; Adaptive weight allocator assigns weights , F B ’ →Multiply pixel by pixel; The two features after weighted multiplication → fusion addition → clear image perception feature J after fusion.

[0020] In S3 above, the loss constraint is constructed as follows: End-to-end image generation image B → adversarial loss → ensure that the generated B is a clear image → B, reconstructed clear image B' → clear consistency loss → improve image clarity → input image O, reconstructed rain image O' → content consistency constraint → ensure content detail consistency → end-to-end image generation B, reconstructed clear image B', input image O, reconstructed rain image O' → bidirectional global-local contrast loss.

[0021] The above five loss function constraints are: 1) Adversarial loss, G B represents the image generator, which mainly converts rainy images into clear images, and D is the discriminator, which mainly distinguishes the generated clear images from the real rain-free images. The specific adversarial loss is defined as follows: ; 2) Content consistency loss, ensuring that the image generator G B The generated clear image is consistent with the rain image input in the dataset in terms of content details. The definition of content consistency loss is as follows: ; Among them, G R represents the rainwater generator and RM represents the rainwater model.

[0022] 3) Clear consistency loss: In order to improve the clarity of the generated clear image, clear consistency loss is used to supervise the image generator G B The consistency between the generated clear image and the clear image reconstructed by the rain model RM. The definition of the clear consistency loss is as follows: ; 4) Bidirectional global contrast loss, which can effectively compensate for the constraints on the clarity and overall color of the generated image. In this way, the network's ability to learn the overall background content features can be significantly improved, and the deraining performance can be further improved. The specific formula is as follows: ; ; ; in and It is a hyperparameter used to balance different losses.

[0023] 5) Bidirectional local contrast loss: The bidirectional local contrast loss improves the network's ability to restore local subtle structures. The specific formula is as follows: ; ; ; in, and Two hyperparameters to balance different losses.

[0024] The overall network loss function in S4 above is expressed as: ; in, , , , as well as is a hyperparameter that balances different losses.

[0025] Embodiment 1: like Figures 1 to 2 As shown, a method for unsupervised rain removal network based on a bidirectional global-local contrast framework includes the following steps: S1, constructs a rain image generation branch, which consists of two subnetworks to comprehensively estimate the fine raindrop image information; S2, constructing a clear image generation branch, which consists of an end-to-end image generation network including a coding layer feature extraction module, a physical perception multi-branch unit, and a decoding layer image restoration module; S3, using the rain image in the data set and the rain layer obtained by the rain image generation branch to reconstruct a clear image feature again using the physical model; S4, adding the clear image obtained by the end-to-end image generation network to the rain layer to obtain a reconstructed rain image; S5. Use unpaired datasets and images reconstructed by two networks to establish multiple constraints to achieve better image reconstruction effects.

[0026] Step S1 specifically includes: The rain image generation branch is constructed, including two subnets. Subnet 1 and subnet 2 use a series of residual convolution blocks to extract rain features. Subnet 1 obtains the rain position information P after the encoding and decoding operation, and subnet 2 obtains the rain density information D after the encoding and decoding operation. Then the two features are multiplied to obtain the accurate raindrop information R.

[0027] Step S2 specifically includes: like Figure 1As shown in , an end-to-end image generation branch is constructed, which consists of a subnet. The coding layer feature extraction module consists of multiple ResBlock blocks. Then, the physical perception multi-branch unit physically models the features extracted from the coding layer to obtain a perceptual feature containing clear image information. Finally, the decoding stage uses a deconvolution reconstruction and upsampling network structure to restore the original image resolution and obtain a reconstructed image B. Then a discriminator is used to distinguish this reconstructed image from the unpaired clear image in the dataset.

[0028] Step S3 specifically includes: A clear image is reconstructed using the physical model, and the raindrop information R is obtained using the rain image generation network. According to the physical model, the input image O is subtracted from R to reconstruct a clear image B'.

[0029] Step S4 specifically includes: Construct a reconstructed rain image, the image B output by the end-to-end image generation network and the output R of the rain image generation network, and use the rain model to add B to R to get a reconstructed rain image O' again.

[0030] Step S5 specifically includes: The overall loss constraint of the network is constructed, and adversarial loss is used to constrain the output of the end-to-end image generation network to be a clear image. At the same time, the clear image B' reconstructed by the rain model is reversed and the loss is calculated with the clear image B generated by the end-to-end network to ensure image clarity. The rain image O' reconstructed by the rain model is calculated with the input rain image O for content consistency loss to ensure that the image content information is consistent. Finally, the generated clear image B and rain image O' are used as anchors, and the images generated in the framework and the rain images in the dataset are used as positive and negative samples to establish a bidirectional global-local contrast constraint to achieve better deraining effect. The specific formula of the loss constraint during training is as follows: The first is adversarial loss, which constrains the end-to-end image generation network to obtain a clear image. The specific formula is as follows: ; in, represents the loss of the discriminator D, y represents the real clear image in the unpaired dataset, Denotes the generator G B The loss is , x is a rainy image from the dataset.

[0031] The second is content consistency loss, which implements the content consistency constraint by reducing the L1 distance between the reconstructed O' and the input O pixel. The specific formula is as follows: ; Where RM represents the rainwater model, G Rrepresents the rain image generator, G B Represents a clear image generator.

[0032] The third is the clarity consistency loss, which implements the clarity consistency constraint by calculating the L1 distance between two clear images and improves the network's rain removal performance. The specific formula is as follows: ; The fourth is the bidirectional global contrast loss, which can effectively improve the clarity and overall color consistency of the generated image by constraining the overall content information of the clear image and the rainy image. The specific formula is as follows: ; ; ; in and It is a hyperparameter used to balance different losses.

[0033] The fifth type is the bidirectional local contrast loss. By constructing the local contrast loss for the clear image and the rainy image, each patch block in the image is constrained in the deep feature space, which improves the network's ability to restore local subtle structures. The specific formula is as follows: ; ; ; in, and Two hyperparameters to balance different losses.

[0034] The overall network loss function is expressed as: ; in, , , , as well as is a hyperparameter that balances different losses, = =1, =0.52, =0.001, = 0.008.

[0035] Embodiment 2: 1. Parameter settings The experiment was trained on an NVIDIA RTX3090 GPU, and the code was implemented based on Pytorch. The model was optimized using the Adam optimizer, and the learning rate and batch size were set to 0.0001 and 4, respectively. The momentum decay exponents were 𝛽1 = 0.9 and 𝛽2 = 0.999, respectively. At the same time, CosineAnnealingLR was used to dynamically adjust the learning rate between them. In addition, the effectiveness of the invention was evaluated on two public rain image datasets, Rain100L and Rain800. The Rain100L dataset contains 200 synthetic rain images as training sets and 100 synthetic rain images as test sets. The Rain800 dataset contains 700 synthetic image pairs as training sets and 100 synthetic rain images as test sets. In order to better evaluate the proposed method, the present invention uses peak signal-to-noise ratio PSNR, structural similarity index SSIM as evaluation indicators, and natural image quality evaluator NIQE as evaluation indicators. We compare our deraining network with 6 other recent excellent algorithms. The compared algorithms mainly include CUT, DerainCycleGAN, ANLCL, CycleGAN, DCD-GAN and DCLGAN.

[0036] 2. Experimental results Table 1 Comparison results of our method with each method on the Rain800 dataset Method PSNR SSIM NIQE CycleGAN (CVPR 2017) 21.52 0.78 3.5 CUT (ECCV 2020) 21.89 0.74 3.45 DerainCycleGAN (TIP 2021) 22.75 0.77 4.60 DCLGAN (CVPRW 2021) 21.73 0.73 5.42 DCD - GAN (CVPR 2022) 21.95 0.73 3.91 ANLCL (TPAMI 2024) 22.55 0.78 3.47 Ours 24.26 0.79 3.43 Table 2 Comparison results of our method with each method on the Rain100L dataset Method PSNR SSIM NIQE CycleGAN (CVPR 2017) 21.79 0.75 4.17 CUT (ECCV 2020) 21.58 0.77 4.08 DerainCycleGAN (TIP 2021) 30.59 0.92 3.32 DCLGAN (CVPRW 2021) 20.69 0.71 5.53 DCD - GAN (CVPR 2022) 22.17 0.79 4.88 ANLCL (TPAMI 2024) 21.84 0.76 3.77 Ours 30.97 0.93 3.07 Tables 1 and 2 show the deraining performance on the Rain800 dataset and the Rain100L dataset. It is worth noting that the present invention achieves PSNR of 30.97dB and 24.26dB, SSIM of 0.93 and 0.79, and NIQE of 3.07 and 3.43 on the Rain100L dataset and Rain800 dataset, respectively, which are the best performances among all unsupervised algorithms. Compared with the CycleGAN framework-based methods CycleGAN, DCD-GAN, DerainCycleGAN, and DCLGAN, the present invention achieves PSNR improvements of 9.18dB, 8.80dB, 0.38dB, and 10.28dB, SSIM improvements of 0.18, 0.14, 0.01, and 0.22, and NIQE reductions of 1.10, 1.81, 0.25, and 2.46 on the Rain00L dataset, respectively. On the Rain800 dataset, PSNR increased by 2.74dB, 2.31dB, 1.51dB and 2.53dB, SSIM increased by 0.01, 0.06, 0.02 and 0.06, and NIQE decreased by 0.07, 0.48, 1.17 and 1.99, respectively. These significant performance improvements are attributed to the fact that the proposed model can effectively separate clear images from rainy images, and builds consistency constraints based on the rain model to form an unsupervised framework, avoiding the problem of the generator in the CycleGAN framework learning two different data distributions. Compared with the unsupervised methods CUT and ANLCL based on contrastive learning, UBPDN achieves 9.39dB and 9.13dB PSNR improvement, 0.16 and 0.01 SSIM improvement, and 1.01 and 0.7 NIQE improvement on the Rain00L dataset, and 2.37dB and 1.71dB PSNR improvement, 0.05 and 0.01 SSIM improvement, and 0.02 and 0.04 NIQE reduction on the Rain800 dataset. This improvement is due to the bidirectional global-local contrast reconstruction constraint established in this paper, which removes rain from the global to the local, thereby improving the ability of the end-to-end image generator to generate clear images.

[0037] In summary, the rain removal network we invented has better rain removal performance in various rain scenes. Through our network, we can effectively reduce the impact of raindrops in the image, improve the clarity and details of the image, and restore a more realistic scene.

[0038] 3. Ablation experiment In order to verify the effectiveness of the proposed physical-aware unsupervised rain removal network, this paper conducted an ablation experiment on the Rain100L dataset and designed three experiments based on the innovative idea of ​​the physical-aware multi-branch unit: (1) Base: an unsupervised rain removal network without using a physical model. (2) Base+PDU: using a physical-aware dual-branch unit; (3) Base+PMU: introducing the physical-aware multi-branch unit proposed in this paper.

[0039] Table 3 Quantitative results of ablation experiments at each stage on the Rain100L dataset

[0040] In Table 3, the results of PSNR, SSIM and NIQE of three experiments are listed. First, the basic framework Base achieved 26.36, 0.84 and 3.75 in PSNR, SSIM and NIQE. Experiments (1) and (2) show that the performance of the network is significantly improved by establishing a physical model in the feature space. However, since the PDU in experiment (2) only used a single branch to estimate rain information, there is a deviation in the estimation of rain information and the enhancement effect is limited. Experiment (3) adds a physical perception multi-branch unit to the network and uses two branches to comprehensively estimate the rain information, which effectively solves the shortcomings of PDU in rain information estimation, improves the interpretability of information, and obtains perceptual features containing more clear image information, thereby improving the network's deraining performance. The experimental results show that the present invention has a certain effect on image deraining and verify the effectiveness of the present invention.

[0041] The present invention proposes an unsupervised rain removal network for physical perception, which mainly proposes a physical perception multi-branch unit and embeds it into the encoding stage of the end-to-end image generation network to extract potential clear image information in the deep feature space. It can avoid the problem that the previous rain removal network does not fully establish a physical model in the feature space, improve the quality of the network to restore the clear image, and remove rain while ensuring that the color and texture details of the image are not deviated.

Claims

1. A method for constructing a physical-aware unsupervised rain removal network, characterized in that: The following steps are involved: S1. Construct a rainwater layer generation branch for generating the rainwater layer R, including two subnets. The first subnet is used to extract the density information of rainwater, and the second subnet is used to extract the location information of rainwater. S2, build a clear image generation branch to generate a clear image B, and use a discriminator to distinguish the generated clear image from the unpaired real image. The branch consists of an end-to-end image generation network, including a coding layer feature extraction module, a physical perception multi-branch unit, and a decoding layer image restoration module; S3. A clear image feature B' is obtained by using a physical model from the rain image O input by the network and the rain layer R obtained by the rain image generation network; the clear image B obtained by the end-to-end image generation network is added to the rain layer R to obtain the rain image O'; a clear consistency loss constraint is established using the clear image B and the reconstructed clear image feature B', and a discrimination loss constraint is established using the unpaired real image; an image content consistency loss constraint is established using the reconstructed rain image O' and the input rain image O; a bidirectional global contrast loss constraint and a bidirectional local contrast loss constraint are established using the input rain image, the clear image, the reconstructed clear image and the reconstructed rain image; S4. Establish the overall network loss function according to all the loss constraints established in S3, with the goal of minimizing the overall network loss function and optimizing network performance.

2. The method for constructing a physical-aware unsupervised rain removal network according to claim 1, characterized in that: The rainwater layer generated by S1 is as follows: Subnet 1: first layer of feature extraction of coding layer → second layer of feature extraction of coding layer → third layer of feature extraction of coding layer → fourth layer of feature extraction of coding layer → first layer of recovery module of decoding layer → second layer of recovery module of decoding layer → third layer of recovery module of decoding layer → output raindrop density information D; Subnet 2: first layer of feature extraction of coding layer → second layer of feature extraction of coding layer → third layer of feature extraction of coding layer → fourth layer of feature extraction of coding layer → first layer of recovery module of decoding layer → second layer of recovery module of decoding layer → third layer of recovery module of decoding layer → output raindrop position information P; Synthesizing the final raindrop features: D P→get the final fine rainwater layer R.

3. The method for constructing a physical-aware unsupervised rain removal network according to claim 2, characterized in that: The clear image generation branch constructed in S2 is as follows: Input image O→first layer of feature extraction of coding layer→second layer of feature extraction of coding layer→physical perception multi-branch unit→third layer of feature extraction of coding layer→physical perception multi-branch unit→fourth layer of feature extraction of coding layer→first layer of recovery module of decoding layer→second layer of recovery module of decoding layer→third layer of recovery module of decoding layer→output reconstructed image B→discriminator constraint ensures that B is a clear image; Constructing a physical-aware multi-branch unit: Input encoding feature F→3*3 Convolution→Activation function Relu→3*3 Convolution→Activation function sigmoid→Feature A with 1 channel→Feature A*Input feature F→Rainwater location information feature F P ; Input encoding feature F→3*3 Convolution→3*3 Convolution→Activation function Relu→3*3 Convolution→Activation function sigmoid→Feature B with 1 channel number→Feature B*Input feature F→Rainwater density information feature F D ; Feature F P , feature F D → Element-wise multiplication → Rain feature F R ; Input encoding feature F→3*3 Convolution→Activation function Relu→3*3 Convolution→Activation function sigmoid→Feature C with 1 channel→Feature C*Input feature F→Feature M→Global average pooling→1*1Convolution→Activation function Relu→1*1Convolution→Activation function sigmoid→ Feature D→Feature D*Feature M→Clear image feature F B ; Input encoding feature F, rain feature F R →Element-wise subtraction →Feature F B ’ Adaptive weight allocator assigns weights , F B →Multiply pixel by pixel; Adaptive weight allocator assigns weights , F B ’ →Multiply pixel by pixel; The two features after weighted multiplication → fusion addition → clear image perception feature J after fusion.

4. The method for constructing a physical-aware unsupervised rain removal network according to claim 3, characterized in that: In S3, the loss constraint is constructed as follows: End-to-end image generation image B → adversarial loss → ensure that the generated B is a clear image → B, reconstructed clear image B' → clear consistency loss → improve image clarity → input image O, reconstructed rain image O' → content consistency constraint → ensure content detail consistency → end-to-end image generation B, reconstructed clear image B', input image O, reconstructed rain image O' → bidirectional global-local contrast loss.

5. The method for constructing a physical-aware unsupervised rain removal network according to claim 4, characterized in that: The five loss function constraints are: 1) Adversarial loss, G B represents the image generator, which mainly converts the rainy image into a clear image, and D is the discriminator, which mainly distinguishes the generated clear image from the real rain-free image. The specific adversarial loss is defined as follows: ； 2) Content consistency loss, ensuring that the image generator G B The generated clear image is consistent with the rain image input in the dataset in terms of content details. The definition of content consistency loss is as follows: ； Among them, G R represents the rainwater generator, RM represents the rainwater model; 3) Clear consistency loss: In order to improve the clarity of the generated clear image, clear consistency loss is used to supervise the image generator G B The consistency between the generated clear image and the clear image reconstructed by the rain model RM; the definition of the clear consistency loss is as follows: ； 4) Bidirectional global contrast loss, which can effectively compensate for the constraints on the clarity and overall color of the generated image; in this way, the network's ability to learn the overall background content features can be significantly improved, and the deraining performance can be further improved. The specific formula is as follows: ； ； ； in and It is a hyperparameter used to balance different losses; 5) Bidirectional local contrast loss: The bidirectional local contrast loss improves the network's ability to restore local subtle structures. The specific formula is as follows: ； ； ； in, and Two hyperparameters to balance different losses.

6. The method for constructing a physical-aware unsupervised rain removal network according to claim 5, characterized in that: The overall network loss function in S4 is expressed as: ； in, , , , as well as is a hyperparameter that balances different losses.

Images