Low-light rain scene graph restoration method based on double-branch characterization learning network
By constructing a dual-branch representation learning network, utilizing the learning characteristics of rain and illumination maps and combining RGB channel subtraction, the problem of image restoration under the combined degradation of low light and heavy rain was solved, achieving better image restoration results.
Patent Information
- Application Number
- CN202410799397.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-20
- Publication Date
- 2025-12-23
AI Technical Summary
Existing single-image rain removal and low-light enhancement methods have limited effectiveness in handling combined low-light and rain degradation tasks, and existing methods suffer from overexposure and residual rain streaks during the restoration process.
A dual-branch representation learning network is constructed to learn rain patterns and low-light characteristics using rain and illumination maps respectively. Prior detail maps are obtained by subtracting RGB channels, and the network is trained using a loss function to achieve image restoration of low-light rain scenes.
It effectively removes the combined degradation caused by low light and rain, improves image restoration, reduces rain streaks, and enhances image clarity and detail retention.
Smart Images
Figure BDA0004902909090000041 
Figure BDA0004902909090000061 
Figure BDA0004902909090000062
Abstract
Description
Technical Field
[0001] This invention relates to low-light rain image restoration technology, specifically to a method for restoring low-light rain scene images based on a dual-branch representation learning network, belonging to the field of image processing. Background Technology
[0002] Visual image information, as a crucial medium for human communication, is widely used in modern society, including in fields such as autonomous driving, video surveillance, medical image processing, and artificial intelligence. However, various real-world environments can degrade the quality of acquired image information, such as camera film noise and severe weather. This not only affects the accuracy of the information we obtain but also severely impacts downstream tasks, such as detection and reconstruction. Therefore, the development of image restoration technology is essential. Rainy nights are common severe weather conditions. Although existing single-image de-raining methods and low-light enhancement methods have made significant progress in addressing specific degradation problems, their effectiveness in solving combined low-light and rain degradation tasks remains limited because rain patterns are hidden in darkness and difficult to capture. Furthermore, cascading restoration methods still face challenges such as overexposure and residual rain streaks. Summary of the Invention
[0003] To address the aforementioned problems, this invention provides a method for restoring low-light rain scene images based on a dual-branch representation learning network. Specifically, this invention constructs an LLR dataset containing both real and synthetic images based on the characteristics of real low-light rain images, and proposes a dual-branch representation learning network and a detail map prior guidance module based on RGB channel subtraction, enabling the network to effectively restore real-world low-light rain scene images.
[0004] The low-light rain scene image restoration method based on a dual-branch representation learning network proposed in this invention mainly includes the following steps:
[0005] (1) Based on the physical characteristics of real-world low-light rain maps, namely, the area near the light source is mainly degraded by rain streaks, and the area far from the light source is mainly degraded by low light, a large number of paired datasets were constructed to train the network.
[0006] (2) Input the constructed low-light rain map dataset into the proposed dual-branch representation learning network, use the rain map to make one branch learn the rain pattern characteristics, use the illumination map to make the other branch learn the low-light characteristics, and then fuse the two features to obtain the degraded representation required by the subsequent restoration network.
[0007] (3) The low-light rain image is fed into the restoration network, and the prior detail image that eliminates the rain streaks hidden in the dark area is obtained by subtracting the RGB channels of the image as a prior hint for the restoration network.
[0008] (4) Construct the loss functions of the dual-branch representation learning network and the restoration network in the first stage. Both are learnable. Train the representation network model by minimizing the loss function.
[0009] (5) Fix the parameters of the dual-branch representation learning network trained in the first stage, and use the obtained representation to guide the training of the restoration network.
[0010] (6) Construct the loss function in the second-stage restoration network and train the restoration network model by minimizing the loss function;
[0011] (7) After the current restoration network has fully converged, complete the network training and obtain the final model, and test the performance of the trained network model in restoring low-light rain maps. Attached Figure Description
[0012] Figure 1 This is a schematic diagram illustrating the principle of the low-light rain scene image restoration method based on a dual-branch representation learning network according to the present invention.
[0013] Figure 2 It is the loss function for training the dual-branch representation network in the network of this invention.
[0014] Figure 3 This is a block diagram of the detail graph prior guidance in the network of this invention.
[0015] Figure 4 This is a comparison of the restoration effects of the present invention and different methods on the LLR real image test set for low-light rain images. (a) is a low-light rain image, (b) to (d) are the restoration results of other methods, and (e) is the restoration result of the present invention.
[0016] Figure 5 This is a comparison of the restoration effects of the present invention and different methods on the LLR composite image test set for low-light rain images. (a) is a low-light rain image, (b) is a clean image, (c) to (i) are the reconstruction results of other methods, and (j) is the denoising result of the present invention. Detailed Implementation
[0017] The present invention will be further described below with reference to the accompanying drawings:
[0018] Figure 1 In this paper, the unsupervised real image denoising method based on self-cooperative parallel generative adversarial networks can be divided into the following seven steps:
[0019] (1) Based on the physical characteristics of real-world low-light rain maps, namely, the area near the light source is mainly degraded by rain streaks, and the area far from the light source is mainly degraded by low light, a large number of paired datasets were constructed to train the network.
[0020] (2) Input the constructed low-light rain map dataset into the proposed dual-branch representation learning network, use the rain map to make one branch learn the rain pattern characteristics, use the illumination map to make the other branch learn the low-light characteristics, and then fuse the two features to obtain the degraded representation required by the subsequent restoration network.
[0021] (3) The low-light rain image is fed into the restoration network, and the prior detail image that eliminates the rain streaks hidden in the dark area is obtained by subtracting the RGB channels of the image as a prior hint for the restoration network.
[0022] (4) Construct the loss functions of the dual-branch representation learning network and the restoration network in the first stage. Both are learnable. Train the representation network model by minimizing the loss function.
[0023] (5) Fix the parameters of the dual-branch representation learning network trained in the first stage, and use the obtained representation to guide the training of the restoration network.
[0024] (6) Construct the loss function in the second-stage restoration network and train the restoration network model by minimizing the loss function;
[0025] (7) After the current restoration network has fully converged, complete the network training and obtain the final model, and test the performance of the trained network model in restoring low-light rain maps.
[0026] Specifically, in step (1), this invention analyzes the physical characteristics of real-world low-light rain images and concludes that the degradation in areas near the light source is mainly due to rain streaks, while the degradation in areas far from the light source is mainly due to low light. Based on this, this invention selects existing DID image deraining datasets as a benchmark, selects images with rain streak intensities of "large" and "medium", and performs illumination reduction on them to construct low-light rain images, totaling 8800 images, of which 8000 are used for training and 800 for testing. In addition, this invention also collects a large number of real-world low-light rain images, of which 368 are used for training and 42 are used for testing.
[0027] In step (2), to input the low-light rain map into the dual-branch representation learning network to obtain a hybrid representation, the present invention first inputs the low-light rain map, rain attention map, and light attention map into the dual-branch representation learning network. Figure 1 The inputs are fed into a dual-branch representation learning network and concatenated on each channel:
[0028] Att R =Concat[Input,Map R ]
[0029] Att L =Concat[Iput,Map L ]
[0030] Where Iput represents a low-light rain map, Map R and Map L These represent rain attention and light attention, respectively, and Concat represents splicing on the channel.
[0031] The resulting stitched images are then fed into two encoders to learn features of different properties related to rain and low light, respectively, and then fused together before being sent to the decoder.
[0032] θ R =∈(Att) R )
[0033] θ L =∈(Att) L )
[0034] θ F =Concat[θ R ,θ L ]
[0035] θ Latent =D(θ) F )
[0036] Where ∈(·) and D(·) represent the encoder and decoder of the dual-branch representation learning network, respectively, and θ R θ L θ F θ Latent These represent rain characterization, low light characterization, fusion characterization, and a series of degradation characterizations, respectively.
[0037] In step (3), the input low-light, rainy image is first segmented into RGB (red, green, blue) channels. Then, the absolute difference between each pair of channels is calculated, resulting in three single-channel images: |RG|, |RB|, and |GB|. Finally, these images are stitched together to create a prior detail map, which captures the unique features of each single-layer image. The prior detail map is passed to parallel frequency / spatial domain branches: the spatial domain uses regular convolution, and the frequency domain operates in the Fourier domain space, fusing the features from both.
[0038] In step (4), the present invention designs a degradation representation loss to constrain the training of the dual-branch representation learning network. First, the present invention focuses on enabling the network to autonomously acquire degradation representations of rain and low light. Next, the present invention fuses these representations. Then, the input image is augmented (rotated by 90, 180, and 270 degrees and horizontally flipped) to obtain an augmented image. The original low-light rain image, the augmented image, and the paired clean image are fed into the dual-branch representation learning network encoder. The resulting fused degradation is input into a two-layer multilayer perceptron (MLP) projection head to generate three vectors. The learned degradation representation enables the anchor blocks to better approach "positive" pairs and move away from "negative" pairs in a specific metric space. The dual degradation loss (DDLoss) loss function is defined as follows:
[0039]
[0040] Among them, DDR E (·) represents the encoder of the dual-branch representation learning network, x, x aug x tar These represent the input low-light rain map, positive sample, and negative sample, respectively.
[0041] In step (5), the present invention fixes the parameters of the dual-branch representation network trained in step (4) and uses its output representation to guide the training of the restoration network. The specific steps are as follows:
[0042] Output = Restoration(Input, θ) Latent )
[0043] Where Input and Output represent the input low-light rain map and the output restored map, Restoration is the restoration network, and θ Latent This is a degraded representation obtained from a dual-branch representation learning network.
[0044] In step (6), the present invention uses L1 loss and perceptual loss to constrain the training of the restoration network. The loss function is defined as follows:
[0045] L R =|R(x)-x clean |+λ p |Φ(R(x))-Φ(x clean )|
[0046] Where R(·) represents the restoration network, λ p For the hyperparameters of the perceptual loss, Φ(·) is the pre-trained VGG19 network.
[0047] In step (7), after the current network has fully converged, the network training is completed and the final model is obtained. The performance of the denoiser in the trained network model is then tested.
[0048] To better illustrate the effectiveness of this invention, comparative experiments were conducted on both the synthetic test dataset and the real test dataset of LLR. Eight methods were used for comparison, including two low-light enhancement methods, three image deraining methods, and three image restoration methods.
[0049] The selected algorithm is:
[0050] Method 1: The method proposed by Wang et al., see reference "Rcdnet: An interpretable rainconvolutional dictionary network for single image deraining. IEEE Transactions on Neural Networks and Learning Systems, 2023, pp. 8668-8682".
[0051] Method 2: The method proposed by Zamir et al., see reference "Learning enriched features for fast image restoration and enhancement, IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1934–1948, 2022."
[0052] Method 3: The method proposed by Zamir et al., see reference "Restormer: Efficient transformer for high resolution image restoration, in IEEE International Conference on Computer Vision and Pattern Recognization, 2022, pp. 5728–5739."
[0053] Method 4: The method proposed by Chen et al., see reference "Simple baselines for imagerestoration," in European Conference on ComputerVision, 2022, pp. 17–33.
[0054] Method 5: The method proposed by Chen et al., see reference "Learning a sparse transformer network for effective image deraining", in IEEE International Conference on Computer Vision and Pattern Recognization, 2023, pp. 5896–5905.
[0055] Method 6: The method proposed by Zhang et al., see reference "Learning rain location prior for nighttime deraining", in IEEE International Conference on ComputerVision, 2023, pp. 13148–13157.
[0056] Method 7: The method proposed by Li et al., see reference "Embedding fourier for ultra-high-definition low-light image enhancement," in IEEE International Conference on Learning Representations, 2023.
[0057] Method 8: The method proposed by Fu et al., see reference "Learning a simple low-light image enhancer from paired low-light instances," in IEEE International Conference on Computer Vision and Pattern Recognization, 2023, pp. 22252–22261.
[0058] The comparative experiment is as follows:
[0059] Experiment 1 was conducted using the synthetic image validation set in LLR. Methods 1, 2, 3, 4, 5, 6, 7, 8, and the method of this invention were used to restore low-light rain images. The objective evaluation results of the reconstruction are shown in Table 1. PSNR (Peak Signal to Noise Ratio, in dB) and SSIM (Structure Similarity Index) were used to evaluate the reconstruction effect. A higher PSNR / SSIM value indicates a better reconstruction effect. This invention also visualizes several of the best-performing networks, such as... Figure 4 As shown.
[0060] Experiment 2 uses a real-world image validation set from LLR to perform reconstruction processing on low-light rain images using methods 1, 2, 3, 4, 5, 6, 7, 8, and the method of this invention. The objective evaluation results of the reconstruction are shown in Table 2. Since real-world images lack paired clean image labels, PSNR and SSIM cannot be used. Therefore, this invention selects NIQE and CEIQ to evaluate the reconstruction effect. A lower NIQE value indicates a better reconstruction effect, while a higher CEIQ value indicates a better reconstruction effect. This invention also visualizes several of the best-performing networks, such as... Figure 5 As shown.
[0061] As can be seen from Tables 1 and 2, this invention achieves better PSNR, SSIM, NIQE, and CEIQ results. Figure 4 , 5 As can be seen, the restoration result of this invention has clear and natural edges, revealing more details, while the restoration result of the contrast algorithm exhibits certain artifacts and blurred edges in terms of subjective visual effect. In summary, compared with the contrast method, the reconstruction result of this invention achieves significant advantages in both subjective and objective evaluation. Therefore, this invention is an effective image denoising method.
[0062] Table 1. Test results of different algorithms on the LLR synthesis graph validation set.
[0063]
[0064] Table 2. Test results of different algorithms on the LLR real graph validation set.
[0065]
Claims
1. A method for restoring low-light rain scene images based on a dual-branch representation learning network, characterized in that... Includes the following steps: Step 1: Based on the physical characteristics of real-world low-light rain images, namely that the area near the light source is mainly characterized by rain streaks and the area far from the light source is mainly characterized by low light and degraded quality, a large number of paired datasets are constructed to train the network. Step 2: Input the constructed low-light rain map dataset into the proposed dual-branch representation learning network. Use the rain map to teach one branch the rain pattern characteristics, and use the illumination map to teach the other branch the low-light characteristics. Then, fuse the two features to obtain the degraded representation required by the subsequent restoration network. Step 3: Feed the low-light rain image into the restoration network, and use the subtraction of the RGB channels of the image to obtain a prior detail image that eliminates the rain streaks hidden in the dark areas as a hint for the restoration network. Step 4: Construct the loss functions for the dual-branch representation learning network and the restoration network in the first stage. Both are learnable. Train the representation network model by minimizing the loss functions. Step 5: Fix the parameters of the dual-branch representation learning network trained in the first stage, and use the obtained representations to guide the training of the restoration network; Step 6: Construct the loss function in the second-stage restoration network, and train the restoration network model by minimizing the loss function; Step 7: After the current restoration network has fully converged, complete the network training and obtain the final model. Test the performance of the trained network model in restoring low-light rain maps.
2. The low-light rain scene image restoration method based on a dual-branch representation learning network according to claim 1, characterized in that... In step two, a dual-branch network designed using the physical characteristics of real-world low-light rain images learns the representations of rain streaks and low light separately, and then merges the two to guide the restoration task. The specific formula is as follows: Att R =Concat[Input,Map R ] Att L =Concat[Input,Map L ] i R =ε(Att R ) θ L =∈(To L ) i F =Concat[θ R ,i L ] i Latent =D(θ F ) Where Input represents a low-light rain map, and Map... R and Map L Representing rain attention and light attention respectively, Concat represents concatenation on the channel, ∈(·) and D(·) represent the encoder and decoder of the dual-branch representation learning network respectively, θ R θ L θ F θ Latent These represent rain characterization, low light characterization, fusion characterization, and a series of degradation characterizations, respectively.
3. The low-light rain scene image restoration method based on a dual-branch representation learning network according to claim 1, characterized in that... In step three, the RGB channels of the low-light rain map are separated, subtracted from each other, and their absolute values are taken to obtain |RG|, |RB|, and |GB|. Finally, they are spliced together to generate a prior detail map. The prior detail map is passed to the parallel frequency domain / spatial domain branches: the spatial domain uses regular convolution, the frequency domain operates in the Fourier domain space, and the features of the two are fused.
4. The low-light rain scene image restoration method based on a dual-branch representation learning network according to claim 1, characterized in that... In step four, the dual-branch degradation representation loss function is constructed by augmenting the input image (rotating by 90, 180, and 270 degrees and horizontally flipping) to obtain an augmented image. The original low-light rain image, the augmented image, and the paired clean image are then fed into the dual-branch representation learning network encoder. The resulting fused degradation input is then fed into a two-layer multilayer perceptron (MLP) projector to generate three vectors. The learned degradation representation enables anchor blocks to better approach "positive" pairs and move away from "negative" pairs in a specific metric space. The dual degradation loss (DDLoss) function is defined as follows: Among them, DDR E (·) represents the encoder of the dual-branch representation learning network, x, x aug x tar These represent the input low-light rain map, positive sample, and negative sample, respectively.