A low-illumination and rain superimposed image enhancement method based on TSDFF-Net

Abstract

The application discloses a low-illumination and rain superposition image enhancement method based on a TSDFF-Net, and comprises the following steps: 1) constructing LLR-Train and LLR-Test data sets; 2) training a rain prior feature extraction module; 3) training a low-illumination enhancement module; and 4) testing the TSDFF-Net. According to the method, the model enhancement quality can be improved according to rain mark information, and the method has good effects on rain removal and low-illumination enhancement of a weak-light rainy-day image.

Description

Technical Field

[0001] This invention relates to low-light image enhancement, rain streaks removal, and conditional variational autoencoders. Specifically, it is a low-light and rain-overlay image enhancement method based on the Two-Stage Degraded Feature Flow Net (TSDFF-Net). Background Technology

[0002] Existing low-light image enhancement algorithms primarily improve details in dark areas and suppress overexposure in bright areas through image denoising and contrast enhancement. However, these algorithms experience a significant performance degradation when rain obscures the image under low-light conditions. Currently, mainstream algorithms in low-light image enhancement focus on image denoising and contrast enhancement techniques to improve the detail and visibility of dark areas and balance overall contrast by suppressing overexposure in bright areas. Representative methods include multi-scale Retinex algorithms based on Retinex theory, contrast stretching based on curve transforms, image denoising based on wavelet transforms, and data-driven deep network methods. These methods can enhance the quality of low-light images and recover more detailed information to a certain extent.

[0003] However, when low-light conditions are compounded by occlusion factors such as rain, the aforementioned algorithms show a significant decline in performance. This is because rainwater generates a large amount of high-frequency noise, severely impacting the visual quality of the image. Simple image denoising and contrast adjustment algorithms struggle to effectively remove the degradation caused by rainwater, revealing a clear blind spot in traditional methods. Data-driven deep network methods are also affected by rainwater degradation information, resulting in loss of detail and interference with the model's feature extraction process, thus leading to a deterioration in model enhancement.

[0004] Rain removal is an ill-posed problem, so traditional methods often employ various image priors to provide additional constraints. However, these manually set priors tend to rely on empirical observation and cannot construct a clear understanding of the image's intrinsic properties. To address this issue, numerous CNN-based methods have been proposed in the last decade, with common rain removal methods including texture analysis, attention mechanisms, and dictionary learning. Rain removal is followed by other enhancement operations. These methods can achieve good image restoration results. To better represent rain distribution, some researchers have attempted to utilize various rain characteristics, such as rain direction and density. These existing algorithms are more suitable for rain removal under normal illumination conditions, but their effectiveness diminishes significantly under poor illumination.

[0005] When an image suffers from both low light and rain occlusion degradation, existing algorithms struggle to effectively handle both effects simultaneously. Directly concatenating existing low-light enhancement and rain removal algorithms fails to achieve good synergistic enhancement because neither algorithm considers the image characteristics under dual degradation conditions. Therefore, direct concatenation has limited effectiveness. This necessitates the development of novel image enhancement algorithms specifically designed for low-light and rainy conditions to address the shortcomings of existing technologies. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by providing a low-light and rain overlay image enhancement method based on TSDFF-Net. This method can improve the enhancement quality of the model based on rain streaks, and has good effects on rain removal and low-light enhancement of images on rainy days.

[0007] The technical solution to achieve the objective of this invention is:

[0008] A low-light and rain-overlay image enhancement method based on TSDFF-Net includes the following steps:

[0009] 1) Construct the LLR-Train and LLR-Test datasets: including:

[0010] 1-1) Collect 13,712 image pairs from the existing dataset Rain14000 and 29,500 image pairs from RealRain as the training set;

[0011] 1-2) Collect 100 clean-rain image pairs, one for each of Rain1001 and Rain100h, containing light and dense rain streaks, as test sets;

[0012] 1-3) Based on the real-world lighting distribution, the collected dataset is used to simulate low-light environments in the images from steps 1-1) and 1-2), enhancing the high-brightness pixels in these image blocks while reducing the brightness of other high-brightness areas, resulting in the processed natural low-light and rain overlay image dataset LLR.

[0013] 1-4) The image dataset LLR processed in step 1-3) is divided into groups, that is, the dataset is divided into 80% as the training set and 20% as the test set, resulting in 34,597 images as the training set LLR-Train, of which 200 images are used as the validation set and 8,615 images are used as the test set LLR-Test.

[0014] 2) Training Rain Prior Feature Extraction Module: Includes:

[0015] 2-1) Pair the images from the LLR-Train dataset in steps 1-4, i.e., the low-light and rain overlay image X and the clean image y. gtPreprocessing involves randomly cropping each image to 512×512 pixels, and simultaneously performing histogram equalization on the low-light and rain overlay image X, as shown in formula (1):

[0016] f(r k )=(L-1)s k (1),

[0017] in s k It is a low-light and rain overlay image X grayscale level r k All previous grayscale r j pixel probability p(r) j The sum of f(r), where L represents the total number of gray levels in the image, f(r) k ) is the input grayscale value r k The function mapped to the output grayscale yields the histogram-equalized image X. he And calculate the histogram equalized image X he The corresponding attention image X attn The attention image is calculated as shown in formula (2):

[0018]

[0019] in The pixel value in the i-th row and j-th column is... For image X he The values ​​corresponding to the three channels in the i-th row and j-th column are used to compare the low-light and rain overlay image X with the histogram-equalized image X. he Normalization involves linearly mapping image pixel values ​​from [0, 255] to [0, 1].

[0020] 2-2) Histogram equalization of image X from step 2-1) he and attention image X attn The input is fed into the feature encoding module, which consists of three rain channel-independent encoding blocks. The rain channel-independent encoding block structure first uses a concatenation operator to input X... he With X attn The channels are concatenated along the channel direction, then a convolutional kernel is used to adjust the channels, and then a depthwise convolutional kernel is used to extract features, retaining the feature as F. c7 Then the feature F c7 The input is fed into a convolutional kernel to adjust the channels, then passed through a GeLU activation function, and then the feature F is applied. c7 The input is fed into a convolutional kernel to adjust the channel, and the obtained features are then compared with F. c7 The elements are added one by one, and then fed into the downsampling module, which consists of a convolutional kernel and a ReLU activation function, to finally extract the rain line density and intensity encoding features.

[0021] 2-3) The rain line density and intensity encoded features from step 2-2) are fed into the saliency feature extraction module. This module consists of three parallel saliency multi-head self-attention (SMSA) modules, one of which is a branch structure saliency multi-head self-attention module. Self-Attention (SMSA) consists of three branches. The first, second, and third branches have the same structure. Each branch first adds channels to the features by passing them through a 1×1 convolutional kernel, then through a 3×3 depthwise separable convolution, and then through a flatten function (·) to change the feature size from C×W×H to C×HW. The output of the first branch is multiplied by the transpose of the output of the second branch to obtain a channel attention weight map of size C×C, which is then fed into the saliency feature extraction module. The saliency features are then fed into softmax (·) to obtain a normalized channel attention weight map. The matrix multiplication of this weight map with the output of the third branch is calculated, and finally fed into the deflatten function to change the weighted features of size C×HW to size C×H×W. Finally, the outputs of the three SMSA branches are concatenated along the channel direction to obtain the saliency features of size 3C×HxW.

[0022] 2-4) The salient features from step 2-3) are fed into the feature decoding module. The feature decoding module consists of three rain-channel-independent decoding blocks. The rain-channel-independent decoding block structure first adjusts the channels through a convolutional kernel, then extracts features through a depthwise convolutional kernel, and retains the feature as F. c7 Then, the feature is fed into a convolutional kernel with adjusted channels, and then through a GeLU activation function. The resulting feature is then compared with F... c7 Element-wise addition is performed, and then the result is fed into an upsampling module. The upsampling module consists of a convolutional kernel, a deconvolutional kernel, and a ReLU activation function. This module decodes the image to obtain the corresponding rain-removed residual image and the multi-scale rain streak prior features F. s F m F l ;

[0023] 2-5) Add the rain-removed residual image obtained in step 2-4) to the low-light and rain superimposed image pixel by pixel, and then perform a quantization operation quant(·) on the superimposed image, as shown in formula (3):

[0024]

[0025] This involves linearly mapping the original pixel value x to the range [0, 255], where This represents a rounding operation, where the maximum value of the resulting image element is M. A quantization operation is then applied pixel-by-pixel to the resulting image to obtain the rain-removed image. Image of rain removal Compared with the clean image y in step 2-1) gt By comparing the results, the prior feature extraction module for training rain is completed;

[0026] 3) Training the low-light enhancement module: including:

[0027] 3-1) The multi-scale rain streak prior features F from steps 2-4) s F m F l The data is fed into the Multi-scale Features Fusion Module (MSFFM), and first... s F m F l The data is processed separately using a global average pooling function, then concatenated along the C direction to obtain a concatenated feature. This feature is then fed into a convolutional kernel with ReLU activation. The feature is then fed into three separate convolutional kernels to exchange channel information, resulting in three features. These three features are then concatenated along the H direction and fed into a Softmax function for calculation along the H direction, splitting the data into three weighted features. These weighted features are then fed into three separate convolutional kernels with adjusted channels, resulting in three weighted operators. Finally, these three weighted operators are respectively coupled with F... s F m F l Calculate pixel-by-pixel multiplication by H along the C direction, and perform a broadcast operation (F) along the W direction. s F m F l For different pixels in the same channel, the corresponding weight operator is used to calculate the pixel value of that channel to obtain refined multi-scale rain streak features;

[0028] 3-2) The low-light and rain overlay image X from step 2-1) and the refined multi-scale rain streaks from step 3-1) are fed into the variational prior feature encoding module. The variational prior feature encoding module consists of three cascaded prior convolutional encoding blocks (PCEBs), and two prior statistical calculation operators connected in parallel after the last PCEB. The structure of the prior convolutional encoding block is composed of a channel direction concatenation operator, a 1×1 convolutional kernel, a 7×7 convolutional kernel, a 1×1 convolutional kernel, a GeLU activation function, a 1×1 convolutional kernel, and a downsampling module cascaded together. The downsampling module consists of a 2-stride convolutional kernel and a ReLU activation function. The prior statistical calculation operator flattens the input features to map them into a one-dimensional vector, and then uses two 1×1 convolutional kernels to perform a convolution operation on the flattened one-dimensional vector to obtain the prior feature latent space statistics μ. p ,logσ p :

[0029] 3-3) Overlay the low-light and rain image X with the clean image y from step 2-1). gt The sequence is connected to the variational coding module in step 3-1) which refines the multi-scale rain streak features. The variational coding module consists of three prior convolutional encoder blocks (PCEB) and two statistical operators. The statistical operators are the same as those in step 2-1). The variational coding module is structurally the same as the variational prior feature coding module in step 3-2), but the weight parameters are shared. Finally, the feature latent space statistics μ are obtained. e ,logσ e ;

[0030] 3-4) The prior feature latent space statistics μ from step 3-2) p ,logσ p The sample is sent to the latent space sampling module, i.e., z ijk =μ p ·S(0,1)+σ p , where z ijk μ is the value of the i-th row, j-th column, and k-th channel of a sample sampled from the sample space. p , σ p Let S(0,1) be the mean and standard deviation. S(0,1) is a common distribution sampling operation with a mean of 0 and a variance of 1. It samples from the sample space of the latent space sampling module to generate a sampling feature sample z of size 43×128×128.

[0031] 3-5) Combine the feature sample z from step 3-4) with the clean image y from step 2-1). gtAfter being concatenated with the refined multi-scale rain streaks in step 3-1), the image is fed into the variational prior feature decoding module. The variational prior feature decoding module consists of three prior convolutional decoding blocks (PCDBs). Each PCDB is composed of a cascaded 3×3 convolutional kernel, ReLU activation function, 3×3 convolutional kernel, 3×3 deconvolutional kernel, and ReLU activation function to obtain the rain-reduced and enhanced residual image.

[0032] 3-6) Add the rain-reinforced residual image extracted in step 3-5) to the low-light and rain overlay image X in step 2-1 pixel by pixel, and perform quantization on the resulting image. Refer to formula (3) to obtain the rain-reinforced image.

[0033] 3-7) Enhance the rain-removing image from step 3-6). Compared with the clean image y in step 2-1) gt Compare the results to complete the training of the low-light enhancement module;

[0034] 4) Testing TSDFF-Net: including:

[0035] 4-1) Take low-light and rain superimposed images from the LLR-Test dataset in steps 1-4) and feed them into the rain prior feature extraction module trained in step 2-5). This module is composed of the feature encoding module in step 2-2), the saliency feature extraction module in step 2-3), and the feature decoding module in step 2-4) cascaded together to obtain multi-scale rain streak features.

[0036] 4-2) The multi-scale rain streaks obtained in step 4-1) and the low-light and rain superimposed image X in step 2-1) are fed into the low-light enhancement module trained in step 3-7). This module is composed of the multi-scale feature refinement module in step 3-1), the variational prior feature encoding module in step 3-2), the latent space sampling module in step 3-4), and the variational prior feature decoding module in step 3-5. The resulting image is then quantized and, according to formula (3), the rain-removed enhanced image is obtained.

[0037] This technical solution addresses the characteristics of images under low-light and rain-occluded conditions, achieving better image enhancement while simultaneously removing rain. Recognizing the significant domain differences between rain removal and enhancement tasks, this solution employs a feature stream approach instead of an image stream to transmit prior rain information. The feature stream consists of multi-layered rain streak feature outputs from the decoder at different scales during the rain removal process. Transmitting rain streak information via a feature stream approach preserves more crucial rain streak information compared to an image stream. Furthermore, the solution utilizes an MSFFM (Multi-Scale Features Fusion Module) module, which complements the key rain streak information at multiple scales semantically and in detail, further enhancing the representational capability of rain streak features. Addressing the prior knowledge that rain streaks are more susceptible to illumination effects in low-light images, this solution employs an RCIEM (Rain Channel Independent Encoder Block) and RCIDM (Rain Channel Independent Decoder Block) to address the increased susceptibility of rain information to light sources in unevenly lit low-light environments. Block); and a low-light enhancement module employing a rain streak feature condition, which effectively improves the uneven illumination enhancement of low-light and rain-overlaid images and accurately removes rain streak degradation information from the images.

[0038] This method can simultaneously handle the combined effects of low-light degradation and rain streaks. It processes the superimposed degradation information through a two-stage module. First, it obtains the prior rain degradation information of the image. Then, the multi-scale rain streak prior features of the feature decoding module in the rain prior feature extraction process are used as reference features for the low-light enhancement module. This allows the model to use rain streak information more accurately, improves the model enhancement quality, and has good results in the joint task of rain removal and low-light enhancement of low-light rainy images. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the TSDFF-Net network structure in an embodiment;

[0040] Figure 2 This is a schematic diagram of the RCIEB and RCIDB structures in the embodiment;

[0041] Figure 3 This is a schematic diagram of the SMSA structure in the embodiment;

[0042] Figure 4 This is a schematic diagram of the MSFFM structure in the embodiment;

[0043] Figure 5This is a schematic diagram of the PCEB and PCDB structures in the embodiment. Detailed Implementation

[0044] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this is not intended to limit the scope of the invention.

[0045] Example:

[0046] Reference Figure 1 A low-light and rain-overlay image enhancement method based on TSDFF-Net includes the following steps:

[0047] 1) Construct the LLR-Train and LLR-Test datasets: including:

[0048] 1-1) Collect 13,712 image pairs from the existing dataset Rain14000 and 29,500 image pairs from RealRain as the training set;

[0049] 1-2) Collect 100 clean-rain image pairs, one for each of Rain1001 and Rain100h, containing light and dense rain streaks, as test sets;

[0050] 1-3) Based on the real-world lighting distribution, the collected dataset is used to simulate low-light environments in the images from steps 1-1) and 1-2), enhancing the high-brightness pixels in these image blocks while reducing the brightness of other high-brightness areas, resulting in the processed natural low-light and rain overlay image dataset LLR.

[0051] 1-4) The image dataset LLR processed in steps 1-3) is divided into groups, with 80% used as the training set and 20% as the test set, resulting in 34,597 images as the training set LLR-Train, 200 images as the validation set, and 8,615 images as the test set LLR-Test. Table 1 shows the sources of the publicly available datasets.

[0052] Table 1:

[0053]

[0054] 2) Training Rain Prior Feature Extraction Module: Includes:

[0055] Reference Figure 1First, the low-light and rain overlay image X from the LLR-Train dataset in steps 1-4) is fed into the feature encoding module to obtain rain line density and intensity encoding features. Second, the obtained rain line density and intensity encoding features are fed into the saliency feature extraction module to obtain saliency features. Third, the extracted saliency features are fed into the feature decoding module to extract the rain-removed residual image. Finally, the rain-removed residual image is added pixel-by-pixel to the low-light and rain overlay image X to obtain the rain-removed image, specifically:

[0056] 2-1) Pair the images from the LLR-Train dataset in steps 1-4, i.e., the low-light and rain overlay image X and the clean image y. gt Preprocessing involves randomly cropping each image to 512×512 pixels, and simultaneously performing histogram equalization on the low-light and rain overlay image X, as shown in formula (1):

[0057] f(r k )=(L-1)s k (1),

[0058] in s k It is a low-light and rain overlay image X grayscale level r k All previous grayscale r j pixel probability p(r) j The sum of f(r), where L represents the total number of gray levels in the image, f(r) k ) is the input grayscale value r k The function mapped to the output grayscale yields the histogram-equalized image X. he And calculate the histogram equalized image X he Corresponding attention image X a ttn The attention image is calculated as shown in formula (2):

[0059]

[0060] in The pixel value in the j-th row and j-th column. For image X he The values ​​corresponding to the three channels in the i-th row and j-th column are used to compare the low-light and rain overlay image X with the histogram-equalized image X. he Normalization involves linearly mapping image pixel values ​​from [0, 255] to [0, 1].

[0061] 2-2) Reference Figure 1 Step 2-1) Histogram equalization of image X he and attention image X attn Send in Figure 1The feature encoding module consists of three rain channel-independent encoding blocks. Within each rain channel-independent encoding block, X is first encoded using a concatenation operator. he With X attn The concatenation is performed along the channel direction, then the channels are adjusted using a convolutional kernel with 3 kernels, a size of 1×1, and a stride of 1. Features are then extracted using a depthwise convolution with 3 kernels and a size of 7×7, and this feature is retained as F. c7 Then, the feature is fed into a convolutional channel with 6 kernels, a size of 1×1, and a stride of 1. After passing through a GeLU activation function, the feature is then fed into a convolutional channel with 3 kernels, a size of 1×1, and a stride of 1. The resulting feature is then compared with F... c7 Element-wise addition is performed, and then the data is fed into a downsampling module. The downsampling module consists of a 3×3 convolutional kernel with 6 kernels and a stride of 2, and a ReLU activation function. Finally, it extracts rain line density and intensity encoded features with a size of 12×64×64. Table 2 shows the feature encoding module.

[0062] Table 2:

[0063]

[0064]

[0065] 2-3) Input the rain line density and intensity encoding features from step 2-2) into... Figure 1 The salient feature extraction module shown is composed of three salient multi-head self-attention (SMSA) modules connected in parallel. The SMSA structure is shown below. Figure 3One of the branch structures, the Salient Multi-head Self-Attention (SMSA) module, contains three branches. The first, second, and third branches have the same structure. Each branch first adds channels to the features by passing them through a 1×1 convolutional kernel, then through a 3×3 depthwise separable convolution with 12 kernels, and then through a flatten function (·) to change the feature size from 4×64×64 to 4×4096. The first branch is then matrix-multiplied with the transpose of the second branch's output to obtain a 4×4 channel attention weight map, which is then fed into the salient features. The extraction module obtains salient features, which are then fed into a softmax function to obtain a normalized channel attention weight map. The matrix multiplication of this weight map with the output of branch 3 is calculated, and finally fed into a deflating function to transform the 4×4096 weighted features into 4×64×64 features. Finally, the outputs of the three SMSA branches are concatenated along the channel direction to obtain salient features of size 12×64×64. 2-4) The salient features extracted in step 2-3) are fed into the feature decoding module, which consists of three rain-channel independent decoding blocks, as described above. Figure 2 As shown in Table 4, in this rain channel-independent decoding block, the channels are first adjusted using a 1×1 convolutional kernel with a stride of 1, and then features are extracted using a 7×7 depthwise convolution. These features are then retained as F. c7 Then, the feature is fed into a convolutional channel with a kernel size of 1×1 and a stride of 1, and then passed through a GeLU activation function. The resulting feature is then fed into another convolutional channel with a kernel size of 1×1 and a stride of 1. c7 Element-wise summation is performed, and then the result is fed into an upsampling module. The upsampling module consists of a 3×3 deconvolution kernel with a stride of 2 and a ReLU activation function to obtain the corresponding rain-removed residual image with a size of 3×512×512 and multi-scale rain streak prior features F. s F nv F l The dimensions are 12×128×128, 6×256×256, and 3×512×512 respectively. Table 3 shows the SMSA modules:

[0066] Table 3:

[0067]

[0068]

[0069] 2-4) Input the salient features from step 2-3) into Figure 1The feature decoding module consists of three rain-channel-independent decoding blocks. The structure of the rain-channel-independent decoding blocks is referenced below. Figure 2 First, the channels are adjusted using a 1×1 convolutional kernel with a stride of 1. Then, features are extracted using a 7×7 depthwise convolution, and these features are retained as F. c7 Then, the feature is fed into a convolutional channel with a kernel size of 1×1 and a stride of 1, and then passed through a GeLU activation function. The resulting feature is then fed into another convolutional channel with a kernel size of 1×1 and a stride of 1. c7 Element-wise addition is performed, and then the result is fed into an upsampling module. The upsampling module consists of a 3×3 deconvolution kernel with a stride of 2 and a ReLU activation function. This decodes the image to obtain the corresponding rain-removed residual image and the multi-scale rain streak prior features F. s F m ,F l Table 4 shows the feature decoding module:

[0070] Table 4:

[0071]

[0072]

[0073] 2-5) Add the rain-removed residual image obtained in step 2-4) to the low-light and rain superimposed image pixel by pixel, and then perform a quantization operation quant(·) on the superimposed image, as shown in formula (3):

[0074]

[0075] This involves linearly mapping the original pixel value x to the range [0, 255], where This represents a rounding operation, where the maximum value of the resulting image element is M. A quantization operation is then applied pixel-by-pixel to the resulting image to obtain the rain-removed image. To enable the rain prior feature extraction module to effectively extract rain streak information, a perceptual loss is designed to judge the quality of rain streak extraction, as shown in formula (4):

[0076]

[0077] VGG16(·) utilizes the feature extraction capabilities of the pre-trained VGG16 network to evaluate the quality of the generated images. Using this loss, the quality of rain removal can be differentiated even under illumination-insensitive conditions, enabling the model to learn to generate high-quality, detailed images and enhancing its ability to reconstruct rain streaks. gt For a clean image, For the rain-removed image, ||·||1 represents the L1 distance. In order to constrain the model at the image level, a brightness-independent loss is designed, as shown in Equation (5):

[0078]

[0079] Where HE(.) is the histogram equalization function, referring to formula (1), the final total loss of this module is... As shown in formula (6):

[0080]

[0081] Finally, the prior feature extraction module for training rain was completed;

[0082] 3) Training the low-light enhancement module: including:

[0083] Reference Figure 1 First, the multi-scale rain streak prior features from steps 2-4) of the feature decoding module in the rain prior feature extraction module are fed into the multi-scale feature refinement module to obtain refined multi-scale rain streak features. Second, the refined multi-scale rain streak features and the low-light and rain overlay image X from step 2-1) are fed into the variational prior feature encoding module to obtain latent space statistics. Third, the refined multi-scale rain streak features, the clean image, and the low-light and rain overlay image X from step 2-1) are fed into the variational encoding module to obtain reference latent space statistics. Fourth, the statistics obtained from the variational prior feature encoding module are fed into the latent space sampling module, and the sampled feature sample z is obtained after sampling. Fifth, the sampled feature sample z, the clean image, and the refined multi-scale rain streak features are concatenated and fed into the variational prior feature decoding module to obtain the rain-reduced enhanced residual image. Finally, this image is added pixel-by-pixel to the low-light and rain overlay image X from step 2-1) to obtain the rain-reduced enhanced image.

[0084] 3-1) The multi-scale rain streak prior features F from steps 2-4) s F m F l Send in Figure 1 The Multi-Scale Feature Refinement Module (MSFFM) is described in the following section. Figure 4 First F s F m F lThe features are obtained by applying global average pooling to the convolutional layers, with corresponding sizes of 12×1×1, 6×1×1, and 3×1×1. These features are then concatenated along the C direction to obtain a feature of size 21×1×1. This feature is then fed into a 50-kernel, 1×1-sized convolutional layer with ReLU activation. This feature is then fed into three 50-kernel, 1×1-sized convolutional layers to exchange channel information, resulting in three features. These three features are then concatenated along the H direction to obtain a feature of size 50×3×1. This feature is then fed into a Softmax function along the H direction for calculation, splitting it into three weighted features, each with a size of 50×1×1. These weighted features are then fed into three convolutional layers with kernels of sizes 12, 6, and 3, each with a size of 1×1, resulting in three weighted operators with sizes of 12×1×1, 6×1×1, and 3×1×1, respectively. Finally, these three weighted operators are respectively connected to F... s F m ,F l Calculate pixel-by-pixel multiplication along the C direction, and perform a broadcast operation (F) along the W direction. s F m ,F l For different pixels in the same channel, the corresponding weight operator is used to calculate the pixel value of that channel, resulting in the refined multi-scale rain streak feature F. s ′,F m ′, F l The corresponding dimensions are 12×128×128, 6×256×256, and 3×512×512, respectively, resulting in refined multi-scale rain streak features. Table 5 shows the multi-scale feature refinement module.

[0085] Table 5:

[0086]

[0087]

[0088] 3-2) Input the low-illuminance and rain overlay image X from step 2-1) and the refined multi-scale rain streak features from step 3-1) into... Figure 1 The variational prior feature encoding module consists of three prior convolutional encoding blocks (PCEB) and two prior statistical computation operators. The structure of the prior convolutional encoding blocks is as follows: Figure 5 The prior statistics calculation operator is referenced in Table 6. First, a concatenation operator is used to combine the low-light and rain overlay map X with one of the refined multi-scale rain streak features F′∈{F s ′,F m ′,F l By connecting the '} along the channel direction, the cascade feature F is obtained. concatThen, the input is fed into a 3×3 convolutional kernel with a stride of 1, along with a ReLU activation function. The resulting features are then compared with F... concat The pixels are added one by one, and then fed into a downsampling module. The downsampling module has a 3×3 convolutional kernel with a stride of 2, and a ReLU activation function. The resulting features are then fed into a prior statistics operator, specifically into a flattening function Flatten(·), and then into two 1×1 convolutional kernels to obtain the prior feature latent space statistics μ. p ,logσ p Table 6 shows the variational prior feature encoding module:

[0089] Table 6:

[0090]

[0091]

[0092] 3-3) Reference Figure 1 In step 2-1), the low-light and rain overlay image X is compared with the clean image y. gt The concatenation and step 3-1) refine the multi-scale rain streak features and input them. Figure 1 The variational coding module consists of three prior convolutional coding blocks (PCEB) and two statistical calculation operators, as shown in Table 7. Structurally, the variational coding module is identical to the variational prior feature coding module in step 3-2), but shares the weight parameters, ultimately yielding the feature latent space statistics μ. e ,logσ e Table 7 shows the variational coding modules:

[0093] Table 7:

[0094]

[0095]

[0096] 3-4) The prior feature latent space statistics μ from step 3-2) p ,logσ p Send in Figure 1 The mid-depth space sampling module is based on the statistical number μ p ,logσ p A sampling feature sample z of size 43×128×128 is randomly generated from the sample space, as shown in formula (7):

[0097]

[0098] Where z ijkμ is the value of the i-th row, j-th column, and k-th channel of a sample sampled from the sample space. p ,σ p The mean and standard deviation are given. A Gaussian distribution sampling operator with a mean of 0 and a variance of 1 is used to sample from the sample space of the latent space sampling module, generating a sampling feature sample z of size 43×128×128;

[0099] 3-5) Combine the feature sample z from step 3-4) with the clean image y from step 2-1). gt After being concatenated with the refined multi-scale rain streak features in step 3-1), the data is fed into... Figure 1 The variational prior feature decoding module consists of three prior convolutional decoding blocks (PCDBs). The PCDB reference... Figure 5 As shown in Table 8, the feature is first passed through a 3×3 convolutional kernel with a stride of 1, then through a ReLU activation function, followed by another 3×3 convolutional kernel with a stride of 1. This feature is then added pixel-by-pixel to the input feature, and finally fed into a 3×3 deconvolutional kernel with a stride of 2. The final result is the rain-reduced and enhanced residual image. Table 8 shows the variational prior feature decoding module.

[0100] Table 8:

[0101]

[0102] 3-6) Add the rain-reinforced residual image extracted in step 3-5) to the low-light and rain overlay image X in step 2-1 pixel by pixel, and perform quantization on the resulting image. Refer to formula (3) to obtain the rain-reinforced image. In order for the low-light enhancement module to effectively enhance the low-light and rain-overlaid image, a reconstruction loss is designed to judge the quality of the rain removal and image enhancement effect, as shown in formula (8):

[0103]

[0104] Where y gt For a clean image, For region-enhanced images, ||·||1 is the L1 distance operator. Simultaneously, to make the latent space distribution of the latent space sampling module closer to the distribution space of the clean image, the latent space distribution of the variational coding module is used to constrain the latent space distribution of the variational prior feature coding module. The KL divergence between the latent space distribution of the variational coding module and the latent space distribution of the prior feature coding module is then calculated. As shown in formula (9):

[0105]

[0106] Wherein, KL divergence is used to describe the degree of similarity between two Gaussian distribution spaces, μe ,σ e μ is the characteristic latent space statistic. p , σ p For the prior feature latent space statistics, The mean is μ and the variance is σ. 2 The Gaussian distribution ultimately results in the total loss of the low-light enhancement module. As shown in formula (10):

[0107]

[0108] 3-7) Enhance the rain-removing image from step 3-6). Compared with the clean image y in step 2-1) gt By comparing the loss functions, the total training loss of the TSDFF-Net model is shown in Equation (11):

[0109]

[0110] Complete the training of the low-light enhancement module;

[0111] 4) Testing TSDFF-Net: including:

[0112] 4-1) Take low-light and rain superimposed images from the LLR-Test dataset in steps 1-4) and feed them into the rain prior feature extraction module trained in step 2-5). This module is composed of the feature encoding module in step 2-2), the saliency feature extraction module in step 2-3), and the feature decoding module in step 2-4) cascaded together to obtain multi-scale rain streak features.

[0113] 4-2) The multi-scale rain streaks obtained in step 4-1) and the low-light and rain superimposed image X in step 2-1) are fed into the low-light enhancement module trained in step 3-7). This module is composed of the multi-scale feature refinement module in step 3-1), the variational prior feature encoding module in step 3-2), the latent space sampling module in step 3-4), and the variational prior feature decoding module in step 3-5. The resulting image is then quantized and the rain-removed enhanced image is obtained by referring to formula (3).

Claims

1. A low-light and rain-overlay image enhancement method based on TSDFF-Net, characterized in that, Includes the following steps: 1) Construct the LLR-Train and LLR-Test datasets: including: 1-1) Collect 13,712 image pairs from the existing dataset Rain14000 and 29,500 image pairs from RealRain as the training set; 1-2) Collect 100 clean-rain image pairs, each containing light and dense rain streaks, for Rain100l and Rain100h as test sets; 1-3) Based on the real-world illumination distribution, the collected dataset is used to simulate low-light environments in the images from steps 1-1) and 1-2), enhancing the high-brightness pixels in these image blocks while reducing the brightness of other high-brightness areas, resulting in the processed natural low-light and rain overlay image dataset LLR. 1-4) The image dataset LLR processed in step 1-3) is divided into groups, that is, the dataset is divided into 80% as the training set and 20% as the test set, resulting in 34597 images as the training set LLR-Train, of which 200 images are used as the validation set and 8615 images are used as the test set LLR-Test. 2) Training Rain Prior Feature Extraction Module: Includes: 2-1) Combine the image pairs from the LLR-Train dataset in steps 1-4, i.e., the low-light and rain overlay images. With clean images Preprocessing involves randomly cropping each image. Pixels, simultaneously overlaying low-light and rain images Perform histogram equalization as shown in formula (1): in , It is a low-light and rain overlay image. grayscale All previous grayscale levels pixel probability The sum of Represents the total number of gray levels in the image. Input grayscale The function mapped to the output grayscale yields the histogram-equalized image. And calculate histogram equalization of the image. Corresponding attention image The attention image is calculated as shown in formula (2): in For the first Line 1 Column pixel values, For image No. Line 1 List the corresponding values ​​for the three channels, and plot the low-light and rain overlay. and histogram equalization image Normalization involves linearly mapping image pixel values ​​from [0, 255] to [0, 1]. 2-2) Perform histogram equalization on the image from step 2-1). and attention images The input is fed into the feature encoding module, which consists of three rain channel-independent encoding blocks. The rain channel-independent encoding block structure first uses a concatenation operator to... and The channels are concatenated along the channel direction, then a convolutional kernel is used to adjust the channels, and then a depthwise convolutional kernel is used to extract features, which are then retained as... Then the feature The feature is fed into a convolutional kernel to adjust the channels, then passed through a GeLU activation function, and then... The input is fed into a convolutional kernel to adjust the channel, and the obtained features are then compared with... The elements are added one by one, and then fed into the downsampling module, which consists of a convolutional kernel and a ReLU activation function, to finally extract the rain line density and intensity encoding features. 2-3) The rain line density and intensity encoded features from step 2-2) are fed into the saliency feature extraction module. This module consists of three parallel saliency multi-head self-attention (SMSA) modules. Each SMSA branch contains three sub-branches; the first, second, and third sub-branches have identical structures. The features of each sub-branch are first processed by a convolutional kernel with a kernel size of [missing information]. The convolution kernel increases the number of channels, and then passes through another convolution kernel with a size of [missing information]. The depthwise separable convolution is then passed through a flattening function. , change the size of the feature from Become Multiply the first branch by the transpose of the second branch's output to obtain a matrix of size [value missing]. The channel attention weight map is then fed into the saliency feature extraction module to obtain saliency features. The normalized channel attention weight map is obtained, and the matrix multiplication between this weight map and the output of branch 3 is calculated. Finally, it is fed into the deflating function to reduce the size to [value missing]. The weighted features become size Finally, the outputs of the three SMSA branches are concatenated along the channel direction to obtain the salient features, with a size of [missing value]. ; 2-4) The salient features from step 2-3) are fed into the feature decoding module. The feature decoding module consists of three rain channel-independent decoding blocks. The structure of the rain channel-independent decoding block is as follows: first, the channels are adjusted using a convolutional kernel, and then the features are extracted using a depthwise convolutional kernel. The retained features are... Then, the feature is fed into a convolutional kernel with adjusted channels, followed by a GeLU activation function, and then fed into another convolutional kernel with adjusted channels. The resulting feature is then compared with... The elements are added one by one, and then fed into the upsampling module, which consists of a convolutional kernel, a deconvolutional kernel, and a ReLU activation function. This module decodes the image to obtain the corresponding rain-removed residual image and multi-scale rain streaks prior features. ; 2-5) Add the rain-removed residual image obtained in step 2-4) to the low-light and rain superimposed image pixel by pixel, and then perform a quantization operation on the superimposed image. As shown in formula (3): Original pixel values A linear mapping is applied to the range [0, 255], where This represents a rounding operation, where the maximum value of the resulting image element is... Perform quantization operation pixel by pixel on the summed image. Obtain rain-removed image The image will go to rain. Compared with the clean image in step 2-1) By comparing the results, the prior feature extraction module for training rain is completed; 3) Training the low-light enhancement module: including: 3-1) The multi-scale rain streak prior features from steps 2-4) The data is fed into the multi-scale feature refinement module MSFFM, firstly... Each is processed through a global average pooling function, and then the result is passed along... The concatenated features are obtained by concatenating the directions, and then the features are fed into convolutional kernels to adjust the channels. The activation function is ReLU. Then, the features are fed into three convolutional kernels to exchange channel information, resulting in three features. Finally, the three features are concatenated along the direction. The directions are concatenated, and then fed into the Softmax function along the direction. Calculate the direction and along The direction is split into 3 weighted features, which are then fed into 3 convolutional kernels to adjust the channels, resulting in 3 weight operators. Finally, the 3 weight operators are respectively compared with... Perform pixel-wise multiplication in the channel dimension C, and in the spatial dimension and Broadcasting is performed on the same channel so that different pixels in the same channel share the corresponding weight operator, thereby obtaining the refined multi-scale rain streak features; 3-2) Overlay the low-light and rain images from step 2-1). The refined multi-scale rain streak features from step 3-1) are fed into the variational prior feature encoding module. This module consists of three concatenated prior convolutional encoding blocks (PCEBs) and two prior statistical calculation operators connected in parallel after the last PCEB. The structure of the prior convolutional encoding blocks is a series of operators along the channel direction. convolution kernel, convolution kernel, Convolution kernel, GeLU activation function, The convolutional kernel and downsampling module are cascaded together. The downsampling module consists of a convolutional kernel with a stride of 2 and a ReLU activation function. The prior statistics calculation operator flattens the input features into a one-dimensional vector, and then uses two... The convolution kernel performs a convolution operation on the flattened one-dimensional vector to obtain the latent space statistics of the prior features. ; 3-3) Overlay the low-light and rain images from step 2-1). With clean images The sequence is connected to the variational coding module in step 3-1) where the refined multi-scale rain streak features are fed in. The variational coding module consists of three prior convolutional coding blocks (PCEB) and two statistical calculation operators. The structure and calculation method of the statistical calculation operators are the same as those of the prior statistical calculation operators in step 3-2). The variational coding module is structurally the same as the variational prior feature coding module in step 3-2), but the weight parameters are shared. Finally, the feature latent space statistics are obtained. ; 3-4) Calculate the latent space statistics of the prior features from step 3-2). Send into the latent space sampling module ,in For the sample number sampled from the sample space Line number Liede The value of the channel, The mean and standard deviation are given. For common distribution sampling operations with a mean of 0 and variance of 1, sampling is performed from the sample space of the latent space sampling module to generate a sample of size . Sampling feature samples ; 3-5) Sample the feature samples from step 3-4). Clean image in step 2-1) After being concatenated with the refined multi-scale rain streak features in step 3-1), the features are fed into the variational prior feature decoding module. The variational prior feature decoding module consists of three prior convolutional decoding blocks (PCDBs), each with a specific size. Convolution kernel, ReLU activation function convolution kernel, A deconvolution kernel and a ReLU activation function are cascaded together to obtain a rain-reduced and enhanced residual image. 3-6) Overlay the rain-reduced and enhanced residual image extracted in step 3-5) with the low-light and rain overlay image from step 2-1). Add pixels one by one and quantize the resulting image. Obtain rain-reducing enhanced images ; 3-7) Enhance the image by removing rain in step 3-6). Compared with the clean image in step 2-1) Compare the results to complete the training of the low-light enhancement module; 4) Testing TSDFF-Net: including: 4-1) Take low-light and rain superimposed images from the LLR-Test dataset in steps 1-4) and feed them into the rain prior feature extraction module trained in step 2-5). This module is composed of the feature encoding module in step 2-2), the saliency feature extraction module in step 2-3), and the feature decoding module in step 2-4) cascaded together to obtain multi-scale rain streak features. 4-2) Combine the multi-scale rain streak features obtained in step 4-1) with the low-light and rain overlay image obtained in step 2-1). The low-light enhancement module trained in steps 3-7) is fed into the image. This module is composed of a cascaded multi-scale feature refinement module from step 3-1), a variational prior feature encoding module from step 3-2), a latent space sampling module from step 3-4), and a variational prior feature decoding module from step 3-5. Quantization is then performed pixel-by-pixel on the resulting image. The resulting image was enhanced by de-raining.

Images