A fast low-light image enhancement method based on global frequency domain filtering

Abstract

The application discloses a kind of fast low-light image enhancement methods based on global frequency domain filtering, it includes the following steps: S1, the image training set of known label is constructed and low-light image enhancement network model;S2, using image training set is trained to low-light image enhancement network model, obtains the low-light image enhancement network model after training;S3, using the low-light image enhancement network model after training is enhanced to low-light image.The application provides a kind of efficient, light level low-light image enhancement network model, can give consideration to the enhancement speed and enhancement effect of low-light image.

Description

Technical Field

[0001] A fast low-light image enhancement method based on global frequency domain filtering. Background Technology

[0002] Low-light image enhancement (LLIE) is a key problem in advanced vision tasks such as object detection and image classification in dark environments. It is widely used in tasks such as robotic emergency rescue, especially in dark environments where direct human operation is not possible, such as underwater or nighttime conditions.

[0003] Currently, low-light image enhancement methods mainly include traditional histogram equalization (HE), Retinex-based low-light enhancement methods, and deep learning-based low-light image enhancement, which has also received widespread attention. Early low-light enhancement methods used histogram equalization (HE) to change the image's histogram distribution or gamma correction to adjust image contrast. These methods only change the image's grayscale values, without considering the influence of semantic information or the relationships between adjacent pixels. Retinex-based methods often increase computational complexity to improve enhancement results. The most advanced LLIE methods are either based on highly complex networks, making them too slow for real-time inference, or they have real-time inference speed but poor enhancement results. Summary of the Invention

[0004] To address the aforementioned shortcomings in existing technologies, this invention provides a fast low-light image enhancement method based on global frequency domain filtering, which solves the problem that existing low-light image enhancement methods struggle to balance speed and effectiveness.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0006] A fast low-light image enhancement method based on global frequency domain filtering is provided, which includes the following steps:

[0007] S1. Construct a training set of images with known labels and a low-light image enhancement network model;

[0008] S2. Train the low-light image enhancement network model using the image training set to obtain the trained low-light image enhancement network model.

[0009] S3. Use the trained low-light image enhancement network model to enhance the low-light image.

[0010] Furthermore, images with known labels were selected from the LOLdataset dataset and the synthetic dataset.

[0011] Furthermore, the construction process of the low-light image enhancement network model in step S1 includes the following sub-steps:

[0012] S1-1. Construct an encoder-decoder network based on depthwise separable convolution and residual downsampling blocks to extract features from the input image;

[0013] S1-2. Construct a fully convolutional neural network to reconstruct details of the extracted image features and obtain the enhanced image.

[0014] Furthermore, the encoder-decoder network based on depthwise separable convolution and residual downsampling blocks includes one depthwise separable convolutional layer, three residual downsampling blocks, three transposed convolutional layers, and two skip connection modules;

[0015] A depthwise separable convolutional layer is connected in series with three residual downsampling blocks; the output of the third residual downsampling block is connected to the input of the first transposed convolutional layer; the outputs of the second residual downsampling block and the first transposed convolutional layer are respectively connected to the input of the second skip connection module; the output of the first skip connection module is connected to the input of the second transposed convolutional layer; the outputs of the first residual downsampling block and the second transposed convolutional layer are respectively connected to the input of the first skip connection module; the output of the second skip connection module is the output of the encoder-decoder network based on the depthwise separable convolution and residual downsampling blocks.

[0016] Furthermore, the depthwise separable convolutional layers are activated by the ReLU function; each residual downsampling block includes two cascaded depthwise separable convolutional layers and a convolutional layer connected in parallel with the two cascaded depthwise separable convolutional layers; the final outputs of the two cascaded depthwise separable convolutional layers are added to the output of the convolutional layer to obtain the output of the residual downsampling block.

[0017] The output scale of the first residual downsampling block is the same as the output scale of the second transposed convolutional layer; the output scale of the second residual downsampling block is the same as the output scale of the first transposed convolutional layer.

[0018] Furthermore, each skip connection module includes a reshaping unit, a fast Fourier transform unit, a low-pass filter, an inverse fast Fourier transform unit, a conversion unit, and a splicing unit connected in sequence; wherein:

[0019] The reshaping unit is used to reshape the output of the residual downsampling block into a global 2D feature matrix;

[0020] The Fast Fourier Transform unit is used to convert the global 2D feature matrix into amplitude and phase spectra.

[0021] A low-pass filter is used to filter out high-frequency redundant information in the amplitude spectrum to obtain a filtered amplitude spectrum.

[0022] The inverse fast Fourier transform unit is used to transform the filtered amplitude spectrum and phase spectrum into a global 2D feature matrix in the time domain;

[0023] The transformation unit is used to convert the temporal domain global 2D feature matrix into a feature map of the same size as the input of the reshaping unit;

[0024] The concatenation unit is used to concatenate the feature map output by the transformation unit with the output of the transposed convolutional layer.

[0025] Furthermore, the fully convolutional neural network includes three convolutional layers connected in series, each with a kernel size of 3×3. The first and second convolutional layers of the fully convolutional neural network are activated using the LeakReLU function. The third convolutional layer of the fully convolutional neural network is activated using the sigmoid function to control the pixel values ​​of the output enhanced image between 0 and 1.

[0026] Furthermore, the specific method for step S2 is as follows:

[0027] S2-1. Input the low-light images from the image training set into the low-light image enhancement network model to obtain the output image of the low-light image enhancement network model.

[0028] S2-2, According to the formula:

[0029]

[0030] Obtain the brightness loss L during training. brightness Where N is the total number of low-light images input into the low-light image enhancement network model during the current training process, and i represents the i-th low-light image input into the low-light image enhancement network model during training; λ j R represents the weight of the j-th channel, where r, g, b correspond to the red, green, and blue color channels in the RGB color space; i,j (x,y) represents the output image of the j-th channel for the i-th low-light image; Y i,j (x,y) represents the normal illumination label image corresponding to the i-th low-light image input;

[0031] S2-3, According to the formula:

[0032]

[0033] Obtain the structural similarity loss L during the training process ssim Where SSIM(·) represents the structural similarity calculation function;

[0034] S2-4. According to the formula:

[0035]

[0036] The structure-aware loss L during training is obtained through the VGG network. perceptual C represents the proportion of the feature image in the VGG network, and H and W represent the dimensions of the feature image in the VGG network; φ λ,k (R(x,y)) represents the feature map of the k-th channel of the low-light image enhancement network model output by the λ-th convolutional layer in the VGG network; φ λ,k (Y(x,y)) represents the feature map of the k-th channel of the normal illumination label image obtained by the λ-th convolutional layer in the VGG network;

[0037] S2-5. According to the formula:

[0038]

[0039] Obtain the smooth loss L during training. smooth ;where R(i * ,j * -1) represents the i-th element in the enhanced image output by the low-light image enhancement network model. * Line j * -1 column pixel values; R(i * +1,j * ) represents the i-th element in the enhanced image output by the low-light image enhancement network model. * +1 row, j * The pixel values ​​of the column; R(i * ,j * ) represents the i-th element in the enhanced image output by the low-light image enhancement network model. * Line j * The pixel values ​​of the column;

[0040] S2-6. According to the formula:

[0041] L=α1(L brightness +L ssim +L perceptual )+α2L smooth

[0042] Construct the final loss function and obtain the loss value L; where α1 and α2 are both weight parameters;

[0043] S2-7. Backpropagation is performed based on the loss value L to update the parameters of the low-light image enhancement network model;

[0044] S2-8. Determine whether the enhancement accuracy of the current low-light image enhancement network model has reached the set threshold. If so, use the current low-light image enhancement network model as the trained low-light image enhancement network model; otherwise, return to step S2-1.

[0045] The beneficial effects of this invention are as follows: This invention provides an efficient and lightweight low-light image enhancement network model that can balance the enhancement speed and enhancement effect of low-light images. Attached Figure Description

[0046] Figure 1 This is a flowchart illustrating the method.

[0047] Figure 2 A schematic diagram of the low-light image enhancement network model in this method;

[0048] Figure 3 This is a structural diagram of the Residual Downsampling Block (RDB).

[0049] Figure 4 This is a low-light image from the embodiment;

[0050] Figure 5 This is the enhanced image (in grayscale format) obtained by this method in the embodiment. Detailed Implementation

[0051] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0052] like Figure 1 As shown, this fast low-light image enhancement method based on global frequency domain filtering includes the following steps:

[0053] S1. Construct a training set of images with known labels and a low-light image enhancement network model;

[0054] S2. Train the low-light image enhancement network model using the image training set to obtain the trained low-light image enhancement network model.

[0055] S3. Use the trained low-light image enhancement network model to enhance the low-light image.

[0056] The construction process of the low-light image enhancement network model in step S1 includes the following sub-steps:

[0057] S1-1. Construct an encoder-decoder network based on depthwise separable convolution and residual downsampling blocks to extract features from the input image;

[0058] S1-2. Construct a fully convolutional neural network to reconstruct details of the extracted image features and obtain the enhanced image.

[0059] like Figure 2 As shown, the encoder-decoder network based on depthwise separable convolution and residual downsampling blocks includes one depthwise separable convolutional layer, three residual downsampling blocks, three transposed convolutional layers, and two skip connection modules.

[0060] A depthwise separable convolutional layer is connected in series with three residual downsampling blocks; the output of the third residual downsampling block is connected to the input of the first transposed convolutional layer; the outputs of the second residual downsampling block and the first transposed convolutional layer are respectively connected to the input of the second skip connection module; the output of the first skip connection module is connected to the input of the second transposed convolutional layer; the outputs of the first residual downsampling block and the second transposed convolutional layer are respectively connected to the input of the first skip connection module; the output of the second skip connection module is the output of the encoder-decoder network based on the depthwise separable convolution and residual downsampling blocks.

[0061] Depthwise separable convolutional layers are activated using the ReLU function; such as Figure 3 As shown, each residual downsampling block includes two cascaded depthwise separable convolutional layers and a convolutional layer connected in parallel with the two cascaded depthwise separable convolutional layers; the final outputs of the two cascaded depthwise separable convolutional layers are added to the output of the convolutional layer to obtain the output of the residual downsampling block; the kernel size of the two cascaded depthwise separable convolutional layers is 3×3, and the kernel size of the parallel convolutional layer is 1×1; the kernel size of the three transposed convolutional layers is 4×4.

[0062] The output scale of the first residual downsampling block is the same as the output scale of the second transposed convolutional layer; the output scale of the second residual downsampling block is the same as the output scale of the first transposed convolutional layer.

[0063] Each jump connection module includes a reshaping unit, a fast Fourier transform unit, a low-pass filter, an inverse fast Fourier transform unit, a conversion unit, and a splicing unit connected in sequence; wherein:

[0064] The reshaping unit is used to reshape the output ω of the residual downsampling block. k ∈R b×c×h×w Reshape into a global 2D feature matrix ω' k ∈R ch×wb b and c represent batch size, and h and w represent feature map size;

[0065] The Fast Fourier Transform unit is used to transform the global 2D feature matrix ω' k ∈R ch×wb Convert to amplitude spectrum and phase spectrum;

[0066] A low-pass filter is used to filter out high-frequency redundant information in the amplitude spectrum to obtain a filtered amplitude spectrum.

[0067] The inverse fast Fourier transform unit is used to transform the filtered amplitude spectrum and phase spectrum into a global 2D feature matrix in the time domain;

[0068] The transformation unit is used to convert the temporal domain global 2D feature matrix into a feature map of the same size as the input of the reshaping unit;

[0069] The concatenation unit is used to concatenate the feature map output by the transformation unit with the output of the transposed convolutional layer.

[0070] The fully convolutional neural network consists of three concatenated convolutional layers with a kernel size of 3×3. The first and second convolutional layers of the fully convolutional neural network are activated by the LeakReLU function. The third convolutional layer of the fully convolutional neural network is activated by the sigmoid function to control the pixel values ​​of the output enhanced image between 0 and 1.

[0071] In skip connections, symmetric encoder and decoder layers with the same feature map scale are called mirror layers. Typically, in a mirror layer, the encoder layer aggregates features from the input low-light image, while the decoder layer reconstructs those features. The encoder layer contains more feature information, helping to enhance the recovery of more details and textures from the low-light image. Skip connections achieve this by fusing features, thus aggregating more features in the decoder layer. Therefore, to achieve better feature fusion results, this invention proposes a specific skip connection method: fusing feature map addition and feature map channel concatenation operations. This increases both the number of features describing the output data and the effective information within each feature during the feature fusion process.

[0072] However, skipping connections introduces certain problems while enhancing feature fusion, causing noise from low-light images to accumulate during forward propagation in the network, affecting the quality of the enhanced image. Therefore, the detail reconstruction network proposes a fully convolutional network consisting of three conventional convolutional layers, which suppresses noise in low-light images while restoring details and texture features.

[0073] The specific method for step S2 is as follows:

[0074] S2-1. Input the low-light images from the image training set into the low-light image enhancement network model to obtain the output image of the low-light image enhancement network model.

[0075] S2-2, According to the formula:

[0076]

[0077] Obtain the brightness loss L during training. brightness Where N is the total number of low-light images input into the low-light image enhancement network model during the current training process, and i represents the i-th low-light image input into the low-light image enhancement network model during training; λ j R represents the weight of the j-th channel, where r, g, b correspond to the red, green, and blue color channels in the RGB color space; i,j (x,y) represents the output image of the j-th channel for the i-th low-light image; Y i,j (x,y) represents the normal illumination label image corresponding to the i-th low-light image input;

[0078] S2-3, According to the formula:

[0079]

[0080] Obtain the structural similarity loss L during the training process ssim Where SSIM(·) represents the structural similarity calculation function;

[0081] S2-4. According to the formula:

[0082]

[0083] The structure-aware loss L during training is obtained through the VGG network. perceptual C represents the proportion of the feature image in the VGG network, and H and W represent the dimensions of the feature image in the VGG network; φ λ,k (R(x,y)) represents the feature map of the k-th channel of the low-light image enhancement network model output by the λ-th convolutional layer in the VGG network; φ λ,k (Y(x,y)) represents the feature map of the k-th channel of the normal illumination label image obtained by the λ-th convolutional layer in the VGG network;

[0084] S2-5. According to the formula:

[0085]

[0086] Obtain the smooth loss L during training. smooth ;where R(i * ,j * -1) represents the i-th element in the enhanced image output by the low-light image enhancement network model. * Line j * -1 column pixel values; R(i * +1,j * ) represents the i-th element in the enhanced image output by the low-light image enhancement network model. * +1 row, j *The pixel values ​​of the column; R(i * ,j * ) represents the i-th element in the enhanced image output by the low-light image enhancement network model. * Line j * The pixel values ​​of the column;

[0087] S2-6. According to the formula:

[0088] L=α1(L brightness +L ssim +L perceptual )+α2L smooth

[0089] Construct the final loss function and obtain the loss value L; where α1 and α2 are both weight parameters;

[0090] S2-7. Backpropagation is performed based on the loss value L to update the parameters of the low-light image enhancement network model;

[0091] S2-8. Determine whether the enhancement accuracy of the current low-light image enhancement network model has reached the set threshold. If so, use the current low-light image enhancement network model as the trained low-light image enhancement network model; otherwise, return to step S2-1.

[0092] In practice, the low-light image and the normal illumination label image are taken from the same location, the difference being the illumination level. The purpose of enhancing the low-light image is to obtain the specific content in the scene, which can be clearly seen under normal illumination. Therefore, the image under normal illumination can be used as the label image.

[0093] SSIM, as a full-reference evaluation metric, measures the similarity between an enhanced image and a reference image based on three components: brightness, contrast, and structure. A higher SSIM value indicates greater structural similarity between the two images, resulting in better enhancement of the low-light image. The calculation process for SSIM is as follows:

[0094] SSIM(R,Y)=l α (R,Y)·cs β (R,Y)

[0095]

[0096]

[0097] Where l(R,Y) represents the brightness comparison formula in structural similarity, cs(R,Y) represents the contrast and structural comparison formula, and μ R μ represents the mean value of all pixel values ​​in the enhanced image output by the low-light image enhancement network model. Y σ represents the mean value of all pixel values ​​in the normal illumination label image.R σ represents the variance of all pixel values ​​in the enhanced image output by the low-light image enhancement network model. Y σ represents the variance of all pixel values ​​in a normal illumination label image. R.Y This represents the covariance between the model output image R and the label image Y, where α and β are weighting coefficients and can take a value of 1.

[0098] In one embodiment of the invention, images with known labels are selected from the LOLdataset dataset and the synthetic dataset, which together contain 2000 pairs of low-light / normal-light images. r =0.29891, λ g =0.58661, λ b =0.11448, α1=1, α2=0.1. A fast low-light image enhancement network N based on the global frequency domain was built on an RTX 3060 graphics card using the PyTorch32 framework. The model was updated via the Adam optimizer with an initial learning rate of 0.1, and trained for 200 epochs using a multi-step learning rate strategy. Simultaneously, the learning rate was set to γ=0.1 at epochs 60 and 150.

[0099] To compare the performance of this method, models N1 (without skip connections), N2 (with skip connections but without an encoder-decoder structure), and N3 (without skip connections and an encoder-decoder structure) were constructed. During training, the dataset was randomly cropped to 384×384 pixels, horizontally or vertically flipped, and the batch size was set to 16. The final image output by the model is the final augmented image. Figure 4 For low-light images that need enhancement, Figure 5 This is the enhanced image (in grayscale format) obtained by this method.

[0100] Quantitative analysis was conducted on the inference speed of models N, N1, N2, and N3 and the performance of their corresponding enhanced images on several image evaluation metrics, including Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM), Learned Perceptual Patch Similarity (LPIPS), and Natural Image Quality Evaluator (NIQE). PSNR and SSIM are supervised metrics, with larger values ​​indicating better enhancement results; conversely, LPIPS and NIQE are unsupervised metrics, with smaller values ​​indicating better enhancement results. This embodiment statistically analyzed the values ​​of each metric under four conditions: using global frequency domain skip connections / lightweight encoder-decoder and not using lightweight encoder-decoder / global frequency domain skip connections, as shown in Table 1. Considering the inference speed in conjunction with the five metrics (PSNR, SSIM, LPIPS, and NIQE), it is evident that this method simultaneously guarantees high-quality image enhancement results.

[0101] Table 1

[0102] N N1 N2 N3 speed 98 96 128 125 PSNR 20.02 20.68 19.86 20.60 SSIM 0.70 0.81 0.70 0.79 NIQE 3.3260 3.1342 3.3242 3.1741 LPIPS 0.1730 0.1868 0.1833 0.1738

[0103] In summary, this invention provides an efficient and lightweight low-light image enhancement network model that balances enhancement speed and effectiveness for low-light images. The skip connection module proposed in this method further balances enhancement speed and enhancement results.

Claims

1. A fast low-light image enhancement method based on global frequency domain filtering, characterized in that, The method comprises the following steps: S1, constructing an image training set of known labels and a low-light image enhancement network model; S2, training the low-light image enhancement network model using the image training set to obtain a trained low-light image enhancement network model; S3, enhancing a low-light image using the trained low-light image enhancement network model; The specific method of step S2 is: S2-1, inputting the low-light image in the image training set into the low-light image enhancement network model to obtain an output image of the low-light image enhancement network model; S2-2, according to the formula: Obtaining brightness loss during training Where N is the total number of low-light images input into the low-light image enhancement network model during the current training process. i This represents the first input to the low-light image enhancement network model during training. i A low-light image; Indicates the first j The weight of each channel, Corresponding to the red, green, and blue color channels in the RGB color space; Indicates the first j The first channel is for the first i Output image of a low-light image; Indicates the input number of the first... i A normal illumination label image corresponding to a low-light image; S2-3, according to the formula: Obtaining structural similarity loss in training process ; wherein denotes a structural similarity computation function; S2-4, according to the formula: Obtain the structure-aware loss during training using the VGG network. C represents the proportion of the feature image in the VGG network, and H and W represent the dimensions of the feature image in the VGG network. Indicates the first in the VGG network The low-light image enhancement network model obtained from the convolutional layer outputs the enhanced image corresponding to the first... k Feature maps of each channel; Indicates the first in the VGG network The normal illumination label image obtained from the convolutional layer corresponds to the first... k Feature maps of each channel; S2-5, according to the formula: Obtain the smooth loss during training ;in This represents the enhanced image output by the low-light image enhancement network model, specifically the first... Line number -1 column pixel values; This represents the enhanced image output by the low-light image enhancement network model, specifically the first... +1 line The pixel values ​​of the column; This represents the enhanced image output by the low-light image enhancement network model, specifically the first... Line number The pixel values ​​of the column; S2-6, according to the formula: Constructing the final loss function, obtaining a loss value L; wherein and are weight parameters; S2-7, based on the loss value L, the low-light image enhancement network model parameters are updated through back propagation; S2-8, determining whether the enhancement accuracy of the current low-light image enhancement network model reaches a set threshold, if yes, the current low-light image enhancement network model is taken as the trained low-light image enhancement network model; otherwise, returning to step S2-1.

2. The method of claim 1, wherein, The known label images are selected from LOLdataset and synthetic datasets.

3. The method of claim 1, wherein, The construction process of the low-light image enhancement network model in step S1 comprises the following sub-steps: S1-1, constructing an encoder-decoder network based on a deep separable convolution and a residual down-sampling block to extract features of an input image; S1-2, constructing a fully convolutional neural network to reconstruct details of the extracted image features to obtain an enhanced image.

4. The method of claim 3, wherein, The encoder-decoder network based on the deep separable convolution and the residual down-sampling block comprises a deep separable convolution layer, three residual down-sampling blocks, three transposed convolution layers and two jump connection modules; The deep separable convolution layer and the three residual down-sampling blocks are connected in series; the output end of the third residual down-sampling block is connected to the input end of the first transposed convolution layer; the output ends of the second residual down-sampling block and the first transposed convolution layer are connected to the input end of the second jump connection module; the output end of the first jump connection module is connected to the input end of the second transposed convolution layer; the output ends of the first residual down-sampling block and the second transposed convolution layer are connected to the input end of the first jump connection module; and the output end of the second jump connection module is the output end of the encoder-decoder network based on the deep separable convolution and the residual down-sampling block.

5. The method of claim 4, wherein, The deep separable convolution layer is activated by a ReLU function; each residual down-sampling block comprises two deep separable convolution layers connected in series and a convolution layer connected in parallel to the two deep separable convolution layers; the final output of the two deep separable convolution layers is added to the output of the convolution layer to obtain the output of the residual down-sampling block; The output scale of the first residual down-sampling block is the same as the output scale of the second transposed convolution layer; and the output scale of the second residual down-sampling block is the same as the output scale of the first transposed convolution layer.

6. The method of claim 4, wherein, Each jump connection module comprises a reshaping unit, a fast Fourier transform unit, a low-pass filter, a fast inverse Fourier transform unit, a conversion unit and a splicing unit connected in series; wherein: a reshaping unit, configured to reshape the output of the residual down-sampling block into a global 2-dimensional feature matrix; a fast Fourier transform unit, configured to convert the global 2-dimensional feature matrix into a magnitude spectrum and a phase spectrum; a low-pass filter, configured to filter high-frequency redundant information in the magnitude spectrum to obtain a filtered magnitude spectrum; a fast inverse Fourier transform unit, configured to convert the filtered magnitude spectrum and the phase spectrum into a time-domain global 2-dimensional feature matrix; a conversion unit, configured to convert the time-domain global 2-dimensional feature matrix into a feature map with the same size as the input of the reshaping unit; a splicing unit, configured to splice the feature map output by the conversion unit and the output of the transposed convolution layer.

7. The method of claim 3, wherein the method is characterized by, The full convolutional neural network comprises three convolutional layers with a convolution kernel size of 3*3 connected in sequence; the first convolutional layer and the second convolutional layer of the full convolutional neural network are activated by a LeakReLU function; the third convolutional layer of the full convolutional neural network is activated by a sigmod function to control the pixel value of the output enhanced image between 0 and 1.

Images