A method for constructing a physical perception-oriented unsupervised rain removal network

Abstract

The application provides a physical perception-oriented unsupervised rain removal network construction method, which comprises the following steps: S1, constructing a rain image generation branch comprising two subnets; S2, constructing a clear image generation branch; S3, obtaining a clear image feature B' from input features O and rain features R by using a physical model, and calculating clear consistency loss of the clear image B' and B; S4, adding the clear image B and the rain layer R to obtain a rain image O', and calculating content consistency loss of the rain image O' and the input image O; and S5, constructing bidirectional global-local contrast loss by using an unpaired data set and images output by the two networks, so that better image reconstruction effect is achieved. The application solves the problems that most current unsupervised rain removal methods do not fully utilize a physical model for modeling in a feature space, and that physical interpretability of information in the feature space is insufficient.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method for constructing an unsupervised rain removal network oriented towards physical perception. Background Technology

[0002] In recent years, increased environmental pollution and frequent severe weather conditions have led to image quality issues such as color distortion, blurring, and low contrast in images captured by optical sensors under rainy conditions. These distortions not only affect the overall image quality but also complicate image analysis. With the rapid development of deep learning technology, advanced vision tasks such as object detection, object tracking, and image segmentation are playing an increasingly important role in fields such as autonomous driving, drone inspection, and security monitoring. Therefore, developing an efficient image deraining algorithm to obtain clear images as input for advanced vision tasks is particularly important. By solving the image deraining problem, the efficiency and accuracy of various practical applications can be improved, thereby enhancing people's quality of life.

[0003] Currently, image deraining methods can be broadly categorized into two types: methods based on artificial priors and methods based on deep learning. Early deraining research largely relied on prior information from the image itself to effectively remove rain streaks, such as layer priors, low-rank appearance models, and frequency domain decomposition models. In recent years, deep learning-based methods have been widely applied in various fields, including image deraining, image classification, and face recognition. In the field of image deraining, convolutional neural networks (CNNs) have been widely used for natural image deraining and have achieved significant progress. Learning-based deraining methods can be divided into two categories: supervised deraining and unsupervised deraining. Traditional supervised deraining algorithms rely on training with a large number of pairs of rainy and rainless images; however, obtaining these pairs of data is quite challenging in practical applications. To address this problem, researchers have begun exploring image deraining algorithms based on unsupervised learning. These methods can train models without pairwise data, achieving effective removal of rain streaks. Many researchers have proposed numerous unsupervised methods based on the CycleGAN framework. Jin et al.'s paper, "Unsupervised single image deraining with self-supervised constraints," proposed an unsupervised single-image deraining method utilizing self-supervised constraints. This method effectively removes rain streaks by leveraging the inherent statistical properties of unpaired data. Yu et al.'s paper, "Unsupervised image deraining: Optimization model driven deep CNN," utilizes statistical properties along raindrop directions to construct an optimization model that separates the rain layer and the image layer, improving the network's ability to reconstruct sharp images. Chen et al.'s paper, "Unpaired deep image deraining using dual contrastive learning," uses a bidirectional collaborative branch to encourage the generator to receive data distributions more consistent with expectations, avoiding the influence of different data distributions and better promoting rain removal and image restoration. While these methods have achieved some success in deraining natural images, they all neglect the interpretability of information in the feature space and fail to adequately model the physical model in the feature space. This results in the inability to obtain useful feature information for reconstructing sharp images, ultimately leading to a large amount of residual rainwater in the generated images. Summary of the Invention

[0004] This invention provides a method for constructing a physically-aware unsupervised rain removal network, which solves the problem that most current unsupervised rain removal methods do not fully utilize physical models in the feature space for modeling, resulting in insufficient physical interpretability of information in the feature space. This leads to a decrease in the network's ability to remove rainwater and deviations in the color and texture details of the generated image.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A method for constructing an unsupervised rain removal network for physical perception includes the following steps:

[0007] S1. Construct a rainwater layer generation branch for the generation of rainwater layer R, including two subnets. The first subnet is used to extract the density information of rainwater, and the second subnet is used to extract the location information of rainwater.

[0008] S2. Construct a clear image generation branch to generate clear image B, and use a discriminator to distinguish the generated clear image from the unpaired real image. The branch consists of an end-to-end image generation network, including a coding layer feature extraction module, a physical perception multi-branch unit, and a decoding layer image restoration module.

[0009] S3. A sharp image feature B' is obtained from the rainwater image O input to the network and the rainwater layer R obtained by the rainwater image generation network using a physical model; the sharp image B obtained by the end-to-end image generation network is added to the rainwater layer R to obtain the rainwater image O'; a sharpness consistency loss constraint is established using the sharp image B and the reconstructed sharp image feature B', and a discrimination loss constraint is established using the unpaired real image; an image content consistency loss constraint is established using the reconstructed rainwater image O' and the input rainwater image O; a bidirectional global contrast loss constraint and a bidirectional local contrast loss constraint are established using the input rainwater image, the sharp image, the reconstructed sharp image, and the reconstructed rainwater image.

[0010] S4. Based on all the loss constraints established in S3, establish the overall network loss function and optimize the network performance by minimizing the overall network loss function.

[0011] The rainwater image generation branch constructed by S1 above is as follows:

[0012] Subnet 1: Encoding layer feature extraction layer 1 → Encoding layer feature extraction layer 2 → Encoding layer feature extraction layer 3 → Encoding layer feature extraction layer 4 → Decoding layer recovery module layer 1 → Decoding layer recovery module layer 2 → Decoding layer recovery module layer 3 → Output raindrop density information D;

[0013] Subnet 2: Encoding layer feature extraction layer 1 → Encoding layer feature extraction layer 2 → Encoding layer feature extraction layer 3 → Encoding layer feature extraction layer 4 → Decoding layer recovery module layer 1 → Decoding layer recovery module layer 2 → Decoding layer recovery module layer 3 → Output raindrop position information P;

[0014] The final raindrop feature is synthesized: D P → yields the final, refined rainwater layer R.

[0015] The sharp image generation branch constructed in S2 above is as follows:

[0016] Input image O → Encoding layer feature extraction layer 1 → Encoding layer feature extraction layer 2 → Physical perception multi-branch unit → Encoding layer feature extraction layer 3 → Physical perception multi-branch unit → Encoding layer feature extraction layer 4 → Decoding layer recovery module layer 1 → Decoding layer recovery module layer 2 → Decoding layer recovery module layer 3 → Output reconstructed image B → Discriminator constraints ensure that B is a clear image;

[0017] Constructing a multi-branch physical sensing unit:

[0018] Input encoding feature F→3 3 Convolution → Activation function ReLU → 3 3. Convolution → Activation function sigmoid → Feature A with 1 channel → Feature A Input feature F → Location information of rainwater feature F P ;

[0019] Input encoding feature F→3 3 Convolution→3 3 Convolution → Activation function ReLU → 3 3. Convolution → Activation function sigmoid → Feature B with 1 channel → Feature B Input feature F → Rainwater density information feature F D ;

[0020] Feature F P Feature F D → Element-by-element multiplication → Rainwater characteristics F R ;

[0021] Input encoding feature F→3 3 Convolution → Activation function ReLU → 3 3. Convolution → Activation function sigmoid → Feature C with 1 channel → Feature C Input feature F → Feature M → Global average pooling → 1 1. Convolution → Activation function ReLU → 1 1. Convolution → Activation function sigmoid → Feature D → Feature D Feature M → Clear Image Feature F B ;

[0022] Input encoding feature F, rainwater feature FR →Element-by-element subtraction→Characteristic F B ’

[0023] Adaptive weight allocator assigns weights F B →Pixel-by-pixel multiplication;

[0024] Adaptive weight allocator assigns weights F B ’ →Pixel-by-pixel multiplication;

[0025] The two features after weighted multiplication are fused and added together to form a clear image perception feature J.

[0026] In S3 above, the loss constraint is constructed as follows:

[0027] End-to-end image generation: Image B → Adversarial loss → Ensure the generated B is a sharp image → B, reconstructed sharp image B' → Sharpness consistency loss → Improve image sharpness → Input image O, reconstructed rain image O' → Content consistency constraint → Ensure content detail consistency → End-to-end generated image B, reconstructed sharp image B', input image O, reconstructed rain image O' → Bidirectional global-local contrast loss.

[0028] The five loss function constraints mentioned above are:

[0029] 1) Combating losses, G B Let represent the image generator, which primarily transforms rainy images into clear images. D is the discriminator, which distinguishes between the generated clear images and real rainless images. The specific adversarial loss is defined as follows:

[0030] ;

[0031] 2) Loss of content consistency, ensuring image generator G B The generated clear image maintains consistency in content detail with the input rain image in the dataset. The definition of content consistency loss is as follows:

[0032] ;

[0033] Among them, G R RM represents the rain generator and the rain model.

[0034] 3) Sharpness Consistency Loss: To improve the sharpness of the generated sharp images, a sharpness consistency loss is used to supervise the image generator G. B The sharpness consistency loss is defined as the consistency between the generated sharp image and the sharp image reconstructed using the rain model RM.

[0035] ;

[0036] 4) Bidirectional global contrast loss, which effectively compensates for constraints on the sharpness and overall color of the generated image. This significantly improves the network's ability to learn overall background content features and further enhances deraining performance. The specific formula is as follows:

[0037] ;

[0038] ;

[0039] ;

[0040] in and It is a hyperparameter used to balance different losses.

[0041] 5) Bidirectional local contrast loss: This method improves the network's ability to recover local fine structures. The specific formula is as follows:

[0042] ;

[0043] ;

[0044] ;

[0045] in, and Two hyperparameters are used to balance different losses.

[0046] The overall network loss function in S4 above is expressed as:

[0047] ;

[0048] in, , , , as well as It is a hyperparameter that balances different losses.

[0049] This invention provides a method for constructing an unsupervised rain removal network for physical perception, which has the following technical effects:

[0050] 1) A physically-aware unsupervised deraining network is proposed. A physical model is established in the feature space to guide the network in extracting richer and clearer image information in the deeper feature space, improving the interpretability of information in the feature space. Simultaneously, a contrast constraint is established between the rain image generation network and the end-to-end image generation network through a rain model, strengthening the network's ability to constrain the reconstructed image. High-quality image deraining results are achieved, and the proposed algorithm demonstrates state-of-the-art performance on two publicly available rain image datasets, Rain100L and Rain800.

[0051] 2) The proposed unsupervised deraining network for physical perception fully leverages the modeling of physical models in the feature space, accurately extracting features of potentially clear images from the deep feature space to achieve deraining. This avoids the lack of interpretability of information and the neglect of fully establishing physical models in the feature space, which are problems inherent in previous attention-based methods. Attached Figure Description

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0053] Figure 1 This is an overall network structure diagram of an embodiment of the present invention;

[0054] Figure 2 This is a structural diagram of the physical sensing multi-branch unit proposed in this invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the following will describe the specific technical solutions of this invention systematically and completely in conjunction with the accompanying drawings provided by this invention. Obviously, the described embodiments are only some embodiments of this invention, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] A method for constructing an unsupervised rain removal network for physical perception includes the following steps:

[0057] S1. Construct a rainwater image generation branch for generating the rainwater layer R, which includes two subnets. The first subnet is used to extract the density information of the rainwater, and the second subnet is used to extract the location information of the rainwater.

[0058] S2. Construct a clear image generation branch to generate clear image B. The branch consists of an end-to-end image generation network, including a coding layer feature extraction module, a physical perception multi-branch unit, and a decoding layer image restoration module.

[0059] S3. A sharp image feature B' is obtained from the rainwater image O input to the network and the rainwater layer R obtained by the rainwater image generation network using a physical model; the sharp image B obtained by the end-to-end image generation network is added to the rainwater layer R to obtain the rainwater image O'; a sharpness consistency loss constraint is established using the sharp image B and the reconstructed sharp image feature B', and a discrimination loss constraint is established using the unpaired real image; an image content consistency loss constraint is established using the reconstructed rainwater image O' and the input rainwater image O; a bidirectional global contrast loss constraint and a bidirectional local contrast loss constraint are established using the input rainwater image, the sharp image, the reconstructed sharp image, and the reconstructed rainwater image.

[0060] S4. Based on all the loss constraints established in S3, establish the overall network loss function and optimize the network performance by minimizing the overall network loss function.

[0061] The rainwater image generation branch constructed by S1 above is as follows:

[0062] Subnet 1: Encoding layer feature extraction layer 1 → Encoding layer feature extraction layer 2 → Encoding layer feature extraction layer 3 → Encoding layer feature extraction layer 4 → Decoding layer recovery module layer 1 → Decoding layer recovery module layer 2 → Decoding layer recovery module layer 3 → Output raindrop density information D;

[0063] Subnet 2: Encoding layer feature extraction layer 1 → Encoding layer feature extraction layer 2 → Encoding layer feature extraction layer 3 → Encoding layer feature extraction layer 4 → Decoding layer recovery module layer 1 → Decoding layer recovery module layer 2 → Decoding layer recovery module layer 3 → Output raindrop position information P;

[0064] The final raindrop feature is synthesized: D P → The final, refined rainwater layer R is obtained.

[0065] The sharp image generation branch constructed in S2 above is as follows:

[0066] Input image O → Encoding layer feature extraction layer 1 → Encoding layer feature extraction layer 2 → Physical perception multi-branch unit → Encoding layer feature extraction layer 3 → Physical perception multi-branch unit → Encoding layer feature extraction layer 4 → Decoding layer recovery module layer 1 → Decoding layer recovery module layer 2 → Decoding layer recovery module layer 3 → Output reconstructed image B → Discriminator constraints ensure that B is a clear image;

[0067] Constructing a multi-branch physical sensing unit:

[0068] Input encoding feature F→3 3 Convolution → Activation function ReLU → 3 3. Convolution → Activation function sigmoid → Feature A with 1 channel → Feature A Input feature F → Location information of rainwater feature F P ;

[0069] Input encoding feature F→3 3 Convolution→3 3 Convolution → Activation function ReLU → 3 3. Convolution → Activation function sigmoid → Feature B with 1 channel → Feature B Input feature F → Rainwater density information feature F D ;

[0070] Feature F P Feature F D → Element-by-element multiplication → Rainwater characteristics F R ;

[0071] Input encoding feature F→3 3 Convolution → Activation function ReLU → 3 3. Convolution → Activation function sigmoid → Feature C with 1 channel → Feature C Input feature F → Feature M → Global average pooling → 1 1. Convolution → Activation function ReLU → 1 1. Convolution → Activation function sigmoid → Feature D → Feature D Feature M → Clear Image Feature F B ;

[0072] Input encoding features F, rainwater features F R →Element-by-element subtraction→Characteristic F B ’

[0073] Adaptive weight allocator assigns weights F B →Pixel-by-pixel multiplication;

[0074] Adaptive weight allocator assigns weights F B ’ →Pixel-by-pixel multiplication;

[0075] The two features after weighted multiplication are fused and added together to form a clear image perception feature J.

[0076] In S3 above, the loss constraint is constructed as follows:

[0077] End-to-end image generation: Image B → Adversarial loss → Ensure the generated B is a sharp image → B, reconstructed sharp image B' → Sharpness consistency loss → Improve image sharpness → Input image O, reconstructed rain image O' → Content consistency constraint → Ensure content detail consistency → End-to-end generated image B, reconstructed sharp image B', input image O, reconstructed rain image O' → Bidirectional global-local contrast loss.

[0078] The five loss function constraints mentioned above are:

[0079] 1) Combating losses, G B Let represent the image generator, which primarily transforms rainy images into clear images. D is the discriminator, which distinguishes between the generated clear images and real rainless images. The specific adversarial loss is defined as follows:

[0080] ;

[0081] 2) Loss of content consistency, ensuring image generator G B The generated clear image maintains consistency in content detail with the input rain image in the dataset. The definition of content consistency loss is as follows:

[0082] ;

[0083] Among them, G R RM represents the rain generator and the rain model.

[0084] 3) Sharpness Consistency Loss: To improve the sharpness of the generated sharp images, a sharpness consistency loss is used to supervise the image generator G. B The sharpness consistency loss is defined as the consistency between the generated sharp image and the sharp image reconstructed using the rain model RM.

[0085] ;

[0086] 4) Bidirectional global contrast loss, which effectively compensates for constraints on the sharpness and overall color of the generated image. This significantly improves the network's ability to learn overall background content features and further enhances deraining performance. The specific formula is as follows:

[0087] ;

[0088] ;

[0089] ;

[0090] in and It is a hyperparameter used to balance different losses.

[0091] 5) Bidirectional local contrast loss: This method improves the network's ability to recover local fine structures. The specific formula is as follows:

[0092] ;

[0093] ;

[0094] ;

[0095] in, and Two hyperparameters are used to balance different losses.

[0096] The overall network loss function in S4 above is expressed as:

[0097] ;

[0098] in, , , , as well as It is a hyperparameter that balances different losses.

[0099] Example 1:

[0100] like Figures 1 to 2 As shown, an unsupervised rain removal network based on a bidirectional global-local contrast framework includes the following steps:

[0101] S1. Construct a rainwater image generation branch, which is composed of two subnets that comprehensively estimate fine raindrop image information;

[0102] S2. Construct a clear image generation branch, which consists of an end-to-end image generation network containing a coding layer feature extraction module, a physical perception multi-branch unit, and a decoding layer image restoration module;

[0103] S3. Using the rainwater images and rainwater layer generated from the rainwater image generation branch in the dataset, a clear image feature is reconstructed again using a physical model;

[0104] S4. Add the clear image obtained by the end-to-end image generation network to the rainwater layer to obtain a reconstructed rainwater image;

[0105] S5. Use unpaired datasets and images reconstructed by two networks to establish multiple constraints to achieve better image reconstruction results.

[0106] Step S1 specifically includes:

[0107] A rainwater image generation branch is constructed, comprising two sub-networks. Sub-network 1 and sub-network 2 each utilize a series of residual convolutional blocks to extract rainwater features. After encoding and decoding operations, sub-network 1 obtains the location information P of the rainwater, and sub-network 2 obtains the density information D of the rainwater after encoding and decoding operations. These two features are then multiplied to obtain the precise raindrop information R.

[0108] Step S2 specifically includes:

[0109] like Figure 1 As shown, an end-to-end image generation branch is constructed, consisting of a subnetwork. The coding layer feature extraction module consists of multiple ResBlock blocks. Then, a physical perception multi-branch unit models the features extracted by the coding layer using a physical model, resulting in a perceptual feature containing clear image information. Finally, the decoding stage uses a network structure of deconvolution reconstruction and upsampling to restore the original image resolution and obtain a reconstructed image B. Then, a discriminator is used to distinguish this reconstructed image from unpaired clear images in the dataset.

[0110] Step S3 specifically includes:

[0111] A clear image is reconstructed using a physical model. Raindrop information R obtained from a rain image generation network is used to reconstruct a clear image B' by subtracting R from the input image O according to the physical model.

[0112] Step S4 specifically includes:

[0113] A reconstructed rainwater image is constructed. The image B output by the end-to-end image generation network and the output R of the rainwater image generation network are used to add R to B using the rainwater model to obtain a reconstructed rainwater image O'.

[0114] Step S5 specifically includes:

[0115] The overall network loss constraint is constructed by using adversarial loss to ensure that the output of the end-to-end image generation network is a clear image. Simultaneously, the clear image B' reconstructed from the rain model is compared with the clear image B generated by the end-to-end network to calculate the loss, ensuring image sharpness. The rain image O' reconstructed from the rain model is compared with the input rain image O to calculate content consistency loss, ensuring consistency of image content information. Finally, using the generated clear image B and rain image O' as anchor points, and the images generated in the framework and the rain images in the dataset as positive and negative samples, a bidirectional global-local contrast constraint is established to achieve better rain removal results. The specific formula for the loss constraint during training is as follows:

[0116] The first approach is adversarial loss, which constrains the end-to-end image generation network to produce a clear image. The specific formula is as follows:

[0117] ;

[0118] in, Let represent the loss of the discriminator D, and y represent the real, sharp images in the unpaired dataset. Represents generator G B The loss is x, which is a rain image from the dataset.

[0119] The second type is content consistency loss, which achieves content consistency constraints by reducing the L1 distance between the reconstructed O' and the input O pixel. The specific formula is as follows:

[0120] ;

[0121] Where RM represents the rainwater model, and G... R G represents a rain image generator. B This indicates a clear image generator.

[0122] The third method is sharpness consistency loss, which uses the L1 distance between two sharp images to constrain sharpness consistency and improve the network's rain removal performance. The specific formula is as follows:

[0123] ;

[0124] The fourth type is bidirectional global contrast loss. By constraining the overall content information of both the sharp image and the rain image, it can effectively improve the sharpness and overall color consistency of the generated image. The specific formula is as follows:

[0125] ;

[0126] ;

[0127] ;

[0128] in and It is a hyperparameter used to balance different losses.

[0129] The fifth type is bidirectional local contrast loss. By constructing local contrast loss for both clear and rainy images, it constrains each patch in the image within the deep feature space, improving the network's ability to recover local fine structures. The specific formula is as follows:

[0130] ;

[0131] ;

[0132] ;

[0133] in, and Two hyperparameters are used to balance different losses.

[0134] The overall network loss function is expressed as:

[0135] ;

[0136] in, , , , as well as It is a hyperparameter that balances different losses. = =1, =0.52, = 0.001, = 0.008.

[0137] Example 2:

[0138] 1. Parameter settings

[0139] The experiments were conducted on an NVIDIA RTX 3090 GPU, and the code was implemented using PyTorch. The model was optimized using the Adam optimizer, with a learning rate and batch size set to 0.0001 and 4, respectively. The momentum decay exponents were... 1 = 0.9 and 2 = 0.999. CosineAnnealingLR is used to dynamically adjust the learning rates between them. Furthermore, the effectiveness of this invention is evaluated on two publicly available rain map datasets, Rain100L and Rain800. The Rain100L dataset contains 200 synthetic rain maps as the training set and 100 synthetic rain maps as the test set. The Rain800 dataset contains 700 synthetic image pairs as the training set and 100 synthetic rain maps as the test set. To better evaluate the proposed method, this invention uses Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Natural Image Quality Evaluator (NIQE) as evaluation metrics. We compare our deraining network with six other recent state-of-the-art algorithms. The algorithms compared are mainly CUT, DerainCycleGAN, ANLCL, CycleGAN, DCD-GAN, and DCLGAN.

[0140] 2. Experimental Results

[0141] Table 1. Comparison results of our method with each other on the Rain800 dataset.

[0142] Method PSNR SSIM NIQE CycleGAN (CVPR 2017) 21.52 0.78 3.5 CUT (ECCV 2020) 21.89 0.74 3.45 DerainCycleGAN (TIP 2021) 22.75 0.77 4.60 DCLGAN (CVPRW 2021) 21.73 0.73 5.42 DCD - GAN (CVPR 2022) 21.95 0.73 3.91 ANLCL (TPAMI 2024) 22.55 0.78 3.47 Ours 24.26 0.79 3.43

[0143] Table 2. Comparison results of our method with each other on the Rain100L dataset.

[0144] Method PSNR SSIM NIQE CycleGAN (CVPR 2017) 21.79 0.75 4.17 CUT (ECCV 2020) 21.58 0.77 4.08 DerainCycleGAN (TIP 2021) 30.59 0.92 3.32 DCLGAN (CVPRW 2021) 20.69 0.71 5.53 DCD - GAN (CVPR 2022) 22.17 0.79 4.88 ANLCL (TPAMI 2024) 21.84 0.76 3.77 Ours 30.97 0.93 3.07

[0145] Tables 1 and 2 show the rain removal performance on the Rain800 and Rain100L datasets. Notably, this invention achieves PSNR of 30.97 dB and 24.26 dB, SSIM of 0.93 and 0.79, and NIQE of 3.07 and 3.43 on the Rain100L and Rain800 datasets, respectively, representing the best performance among all unsupervised algorithms. Compared to methods based on the CycleGAN framework, such as CycleGAN, DCD-GAN, DerainCycleGAN, and DCLGAN, this invention achieves PSNR improvements of 9.18 dB, 8.80 dB, 0.38 dB, and 10.28 dB, SSIM improvements of 0.18, 0.14, 0.01, and 0.22, and NIQE reductions of 1.10, 1.81, 0.25, and 2.46 on the Rain00L dataset, respectively. On the Rain800 dataset, the PSNR was improved by 2.74 dB, 2.31 dB, 1.51 dB, and 2.53 dB, respectively; the SSIM was improved by 0.01, 0.06, 0.02, and 0.06; and the NIQE was reduced by 0.07, 0.48, 1.17, and 1.99, respectively. These significant performance improvements are attributed to the proposed model's ability to effectively separate clear images from rain images and the construction of consistency constraints based on the rain model, forming an unsupervised framework that avoids the problem of the generator learning two different data distributions in the CycleGAN framework. Compared to the contrastive learning-based unsupervised methods CUT and ANLCL, UBPDN achieves PSNR improvements of 9.39 dB and 9.13 dB, SSIM improvements of 0.16 and 0.01, and NIQE improvements of 1.01 and 0.7 on the Rain00L dataset, respectively. On the Rain800 dataset, it achieves PSNR improvements of 2.37 dB and 1.71 dB, SSIM improvements of 0.05 and 0.01, and NIQE reductions of 0.02 and 0.04, respectively. This improvement is attributed to the bidirectional global-local contrastive reconstruction constraint established in this paper, which removes rainwater from a global to a local perspective, thereby enhancing the end-to-end image generator's ability to generate sharp images.

[0146] In summary, our invented rain removal network exhibits superior rain removal performance across various rainfall scenarios. Through our network, the impact of raindrops in images can be effectively reduced, enhancing image clarity and detail, and restoring a more realistic scene.

[0147] 3. Ablation test

[0148] To verify the effectiveness of the proposed unsupervised rain removal network for physical perception, ablation experiments were conducted on the Rain100L dataset. Three experiments were designed based on the innovative idea of ​​physical perception multi-branch units: (1) Base: Unsupervised rain removal network without physical model. (2) Base+PDU: Using physical perception dual-branch units; (3) Base+PMU: Introducing the physical perception multi-branch unit proposed in this invention.

[0149] Table 3. Quantitative results of ablation experiments at various stages on the Rain100L dataset.

[0150]

[0151] Table 3 lists the PSNR, SSIM, and NIQE results for the three experiments. First, the Base framework achieved PSNR, SSIM, and NIQE scores of 26.36, 0.84, and 3.75, respectively. Experiments (1) and (2) show that establishing a physical model in the feature space significantly improves network performance. However, in Experiment (2), the PDU only used a single branch to estimate rainwater information, leading to a bias in the estimation of rainwater information and limited enhancement. Experiment (3) incorporates a physical perception multi-branch unit into the network, using two branches to comprehensively estimate rainwater information. This effectively addresses the shortcomings of the PDU in rainwater information estimation, improves the interpretability of information, and yields perceptual features containing more clear image information, thereby improving the network's rain removal performance. The experimental results demonstrate that this invention has a certain effect on image rain removal and verify the effectiveness of this invention.

[0152] This invention proposes an unsupervised deraining network oriented towards physical perception. It mainly proposes a physical perception multi-branch unit and embeds it into the encoding stage of an end-to-end image generation network. It extracts potential clear image information in the deep feature space, which can avoid the problem that previous deraining networks did not fully establish a physical model in the feature space, improve the quality of the network to recover clear images, and remove rainwater while ensuring that the color and texture details of the image are not deviated.

Claims

1. A method for constructing an unsupervised rain removal network for physical perception, characterized in that, Includes the following steps: S1. Construct a rainwater layer generation branch for the generation of rainwater layer R, including two subnets. The first subnet is used to extract the density information of rainwater, and the second subnet is used to extract the location information of rainwater. S2. Construct a clear image generation branch to generate clear image B, and use a discriminator to distinguish the generated clear image from the unpaired real image. The branch consists of an end-to-end image generation network, including a coding layer feature extraction module, a physical perception multi-branch unit, and a decoding layer image restoration module. S3. A sharp image feature B' is obtained from the rainwater image O input to the network and the rainwater layer R obtained by the rainwater image generation network using a physical model; the sharp image B obtained by the end-to-end image generation network is added to the rainwater layer R to obtain the rainwater image O'; a sharpness consistency loss constraint is established using the sharp image B and the reconstructed sharp image feature B', and a discrimination loss constraint is established using the unpaired real image; an image content consistency loss constraint is established using the reconstructed rainwater image O' and the input rainwater image O; a bidirectional global contrast loss constraint and a bidirectional local contrast loss constraint are established using the input rainwater image, the sharp image, the reconstructed sharp image, and the reconstructed rainwater image. S4. Based on all the loss constraints established in S3, establish the overall network loss function and optimize the network performance by minimizing the overall network loss function. The rainwater layer constructed by S1 generates the following branches: Subnet 1: Encoding layer feature extraction layer 1 → Encoding layer feature extraction layer 2 → Encoding layer feature extraction layer 3 → Encoding layer feature extraction layer 4 → Decoding layer recovery module layer 1 → Decoding layer recovery module layer 2 → Decoding layer recovery module layer 3 → Output raindrop density information D; Subnet 2: Encoding layer feature extraction layer 1 → Encoding layer feature extraction layer 2 → Encoding layer feature extraction layer 3 → Encoding layer feature extraction layer 4 → Decoding layer recovery module layer 1 → Decoding layer recovery module layer 2 → Decoding layer recovery module layer 3 → Output raindrop position information P; The final raindrop feature is synthesized: D P → yields the final, finer rainwater layer R; The clear image generation branch constructed in S2 is as follows: Input image O → Encoding layer feature extraction layer 1 → Encoding layer feature extraction layer 2 → Physical perception multi-branch unit → Encoding layer feature extraction layer 3 → Physical perception multi-branch unit → Encoding layer feature extraction layer 4 → Decoding layer recovery module layer 1 → Decoding layer recovery module layer 2 → Decoding layer recovery module layer 3 → Output reconstructed image B → Discriminator constraints ensure that B is a clear image; Constructing a multi-branch physical sensing unit: Input encoding feature F→3 3 Convolution → Activation function ReLU → 3 3. Convolution → Activation function sigmoid → Feature A with 1 channel → Feature A Input feature F → Location information of rainwater feature F P ; Input encoding feature F→3 3 Convolution→3 3 Convolution → Activation function ReLU → 3 3. Convolution → Activation function sigmoid → Feature B with 1 channel → Feature B Input feature F → Rainwater density information feature F D ; Feature F P Feature F D → Element-by-element multiplication → Rainwater characteristics F R ; Input encoding feature F→3 3 Convolution → Activation function ReLU → 3 3. Convolution → Activation function sigmoid → Feature C with 1 channel → Feature C Input feature F → Feature M → Global average pooling → 1 1. Convolution → Activation function ReLU → 1 1. Convolution → Activation function sigmoid → Feature D → Feature D Feature M → Clear Image Feature F B ; Input encoding features F, rainwater features F R →Element-by-element subtraction→Characteristic F B ’ Adaptive weight allocator assigns weights F B →Pixel-by-pixel multiplication; Adaptive weight allocator assigns weights F B ’ →Pixel-by-pixel multiplication; The two features after weighted multiplication are fused and added together to form a clear image perception feature J.

2. The method for constructing an unsupervised rain removal network for physical perception as described in claim 1, characterized in that, In S3, the loss constraint is constructed as follows: End-to-end image generation: Image B → Adversarial loss → Ensure the generated B is a sharp image → B, reconstructed sharp image B' → Sharpness consistency loss → Improve image sharpness → Input image O, reconstructed rain image O' → Content consistency constraint → Ensure content detail consistency → End-to-end generated image B, reconstructed sharp image B', input image O, reconstructed rain image O' → Bidirectional global-local contrast loss.

3. The method for constructing an unsupervised rain removal network for physical perception as described in claim 2, characterized in that, The constraints of the five loss functions—adversarial loss, content consistency loss, clarity consistency loss, bidirectional global contrast loss, and bidirectional local contrast loss—are as follows: 1) Combating losses, G B The sharp image generator, denoted by , primarily transforms rainy images into sharp ones. D is the discriminator, responsible for distinguishing between the generated sharp images and real, rain-free images. The specific adversarial loss is defined as follows: ； in, Let represent the loss of the discriminator D, and y represent the real, sharp images in the unpaired dataset. Represents generator G B The loss is denoted by x, which is a rain image from the dataset; 2) Loss of content consistency, ensuring image generator G B The generated clear image maintains consistency in content detail with the input rain image in the dataset. The definition of content consistency loss is as follows: ； Among them, G R RM represents the rain generator; 3) Sharpness Consistency Loss: To improve the sharpness of the generated sharp images, a sharpness consistency loss is used to supervise the image generator G. B The consistency between the generated sharp image and the sharp image reconstructed using the rain model RM; the definition of sharpness consistency loss is as follows: ； 4) Bidirectional global contrast loss, which effectively compensates for constraints on the sharpness and overall color of the generated image; in this way, the network's ability to learn overall background content features can be significantly improved, and the deraining performance can be further enhanced. The specific formula is as follows: ； ； ； in and These are hyperparameters used to balance different losses; 5) Bidirectional local contrast loss: This method improves the network's ability to recover local fine structures. The specific formula is as follows: ； ； ； in, and Two hyperparameters are used to balance different losses.

4. The method for constructing an unsupervised rain removal network for physical perception as described in claim 3, characterized in that, The overall network loss function in S4 is expressed as follows: ； in, , , , as well as It is a hyperparameter that balances different losses.

Images