Frequency selection and structure guided attention double-domain low-illumination image enhancement method

Abstract

The invention provides a frequency selection and structure guidance attention double-domain low-illumination image enhancement method, which comprises the following steps: converting an input low-illumination image from a spatial domain to a frequency domain, and carrying out brightness improvement on the low-illumination image to obtain a low-illumination adjustment image; extracting structural features of the low-illumination image by using a deep convolutional network; enhancing the whole frequency spectrum of the low-illumination image by using the global interaction capability of Transform, and recovering the frequency to a normal level; and a double-domain illumination restorer is adopted to enhance the performance of the low-illumination image.

Description

Technical Field

[0001] The present invention relates to an image processing technology, in particular to a dual-domain low-light image enhancement method of frequency selection and structure-guided attention. Background Art

[0002] In recent years, a variety of methods based on convolutional neural networks (CNNs) have been developed for various low-light image enhancement tasks, which combine advanced modules from other fields. However, these methods are limited by the limited receptive field of convolutional units and cannot effectively capture long-range dependencies. Many researchers have also applied Transformer to low-light enhancement tasks, but existing methods mainly perform restoration in the spatial domain and do not fully utilize the frequency difference between clear images and low-light degraded images. To solve this problem, some studies have used tools such as wavelet transform or Fourier transform to decompose features into different frequency components, and then process these components separately to reconstruct the corresponding features. However, wavelet transform decomposes feature maps into fixed sub-bands and cannot effectively distinguish the most useful and least useful frequency components for enhancement or suppression. In addition, these methods also require inverse transforms of Fourier or wavelet transforms, resulting in additional computational overhead. Summary of the invention

[0003] The object of the present invention is to provide a dual-domain low-light image enhancement method with frequency selection and structure-guided attention, comprising:

[0004] Step S100, converting the input low-light image from the spatial domain to the frequency domain, and performing brightness enhancement on the low-light image to obtain a low-light adjusted image I; using a deep convolutional network to extract structural features F of the low-light image; using the global interaction capability of the Transformer to enhance the entire spectrum of the low-light image and restore the frequency to a normal level;

[0005] Step S200: using a dual-domain illumination restorer to enhance the performance of low-light images.

[0006] Furthermore, in step S100, the given low-light image is converted from the spatial domain to the frequency domain by a modulator.

[0007] (1)

[0008] in, represents the image in the frequency domain, D(*) represents the operation of the decomposition module, Represents the low-frequency component of the image, represents the intermediate frequency component, represents the high-frequency component, and H, W, and C are the height, width, and number of channels of the input image, respectively.

[0009] Furthermore, the global interaction capability of Transformer is used to enhance the entire spectrum of low-light images and restore the frequency to normal levels, as shown in formula (3):

[0010] (3)

[0011] in, represents structural features extracted by a deep convolutional network; SDSGA represents the global frequency enhancement operation in the spatial domain; Represents the image after brightness enhancement of the low-light image in the frequency domain.

[0012] Furthermore, in step S200, a dual-domain illumination restorer is used to enhance the performance of low-light images, and a three-scale U-shaped structure is constructed by integrating the dual-domain illumination restorer into the backbone network; step S200 includes:

[0013] Step S201, inputting the low-light adjusted image X into the convolution layer to obtain image features;

[0014] Step S202, down-sampling after inputting into the first dual-domain illumination restorer, and combining operation with the low-light adjustment image;

[0015] Step S203, down-sampling is performed after passing through the second dual-domain illumination restorer, and combined operation is performed with the low illumination adjustment image;

[0016] Step S204, generating a first enhanced image after passing through the third dual-domain illumination restoration device, and performing a combined operation with the output of the second dual-domain illumination restoration device after upsampling;

[0017] Step S205, after convolution, the second enhanced image is input into the fourth dual-domain illumination restorer, and at the same time, the second enhanced image is combined with the output of the first dual-domain illumination restorer after upsampling;

[0018] Step S206: Output the residual image I after passing through the fifth dual-domain illumination restorer re , through the convolution layer and add it to the low-light adjusted image X to get the enhanced image I en ;

[0019] Further, in step S202 and step S203, the low-light adjusted image is integrated into the main path through the shallow layer and the hierarchical layer respectively.

[0020] Further, in step S204 and step S205, the second dual-domain illumination restorer and the first dual-domain illumination restorer are connected to the third dual-domain illumination restorer and the fourth dual-domain illumination restorer through a jump.

[0021] Furthermore, each dual-domain illumination restorer includes a frequency selective enhancement module FSE and a spatial domain structure guided attention module SDSGA; FSE selectively enhances the input image in the frequency domain; SDSGA uses the structural features captured by SFG to guide the calculation of self-attention.

[0022] Furthermore, the method in which the frequency selective enhancement module selectively enhances the input image in the frequency domain includes:

[0023] Generate a low-frequency image of the input image through a low-pass filter and its corresponding mid-pass filter and IF , by subtracting the intermediate frequency map from the input image and low frequency graph Get high frequency graph ;

[0024] The three frequency graphs are obtained by formula (6): , , The fusion feature Z

[0025] (6)

[0026] in, is the parameter of the fully connected layer, GAP(*) is the global average pooling layer operation;

[0027] The fused features are separated into two paths, one of which is transmitted to the GAP network for global average pooling to expand the global receptive field; the other is transmitted to the LAP network for local pooling to expand the local receptive field;

[0028] The data after global average pooling and local pooling are fused.

[0029] Furthermore, the spatial domain structure guides the attention module to process the fused data F after global average pooling and local pooling. in Do the following:

[0030] (1) Input feature F in The query Q, key K and value V components are linearly projected through an unbiased fully connected layer.

[0031]

[0032] in, , is the learnable parameter of the fully connected layer, and T represents the matrix transpose;

[0033] (2) Reshape the projected features into tokens X and divide them into k heads, ,in , , i=1,2,...,k;

[0034] (3) Reshape the structural feature F into token Y, which is also divided into k heads, ;

[0035] (4) Combine Y and value V to form a new value V new , ;

[0036] (5) Obtaining a self-attention mechanism ;

[0037] (6) The self-attention mechanism is applied to each head, and the output token X is obtained by concatenating and combining the position encoding out , and then reshaped to form the output feature F out ;

[0038] (7) Output feature F out Transmitted to the feed-forward neural network FFN for processing.

[0039] This paper proposes a dual-domain low-light image enhancement network DDLIE-Net based on frequency selection and structure-guided attention. The FSE module dynamically constructs a set of filters to decompose the feature map into frequency components, and combines channel attention to effectively select key frequencies for restoration. The SDSGA module is used to guide the non-local interaction modeling of the area processed by the FSE module, thereby overcoming the inherent limitations of frequency domain operations. The DDIR module, which integrates the FSE and SDSGA modules, can effectively restore the frequency of low-light images.

[0040] The present invention will be further described below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the method flow of the present invention.

[0042] Figure 2 Schematic diagram of the architecture of the Decoupler submodule.

[0043] Figure 3 This is a schematic diagram of the architecture of the Modulator submodule.

[0044] Figure 4 Schematic diagram of the architecture of the SG-MSA submodule.

[0045] Figure 5 It is a schematic diagram of the visualization results of different data sets, among which the figure shown by ours is a schematic diagram of the results obtained by the method adopted by the present invention.

[0046] Figure 6Schematic diagram for visual comparison. DETAILED DESCRIPTION

[0047] Combination Figure 1 , a dual-domain low-light image enhancement method with frequency selection and structure-guided attention, including:

[0048] Step S100, converting the input low-light image from the spatial domain to the frequency domain, and performing brightness enhancement on the low-light image to obtain a low-light adjusted image I; and extracting structural features F of the low-light image using a deep convolutional network;

[0049] Step S200: using a dual-domain illumination restorer to enhance the performance of low-light images.

[0050] Each image in the spatial domain can be regarded as a combination of various frequency waves in the frequency domain. Therefore, it is necessary to transform the low-light image from the spatial domain to the frequency domain. Figure 1 In step S100, the given low-light image is converted from the spatial domain to the frequency domain through a modulator.

[0051] (1)

[0052] in, represents the image in the frequency domain, including all the frequency components of the image; D(*) represents the operation of the decomposition module; Represents the low-frequency components of the image, corresponding to smooth changes and general shapes; Represents mid-frequency components, corresponding to medium details and textures; Represents high-frequency components, corresponding to fine details and sharp edges; H, W, and C are the height, width, and number of channels of the input image, respectively.

[0053] The low-frequency, medium-frequency and high-frequency components are enhanced in the frequency domain to restore the brightness and structural information of the image. The enhancement method is shown in formula (2):

[0054] (2)

[0055] in, represents the enhanced image in the frequency domain; M(*) represents the modulation module operation.

[0056] Although frequency domain enhancement can restore the brightness and structural information of the image, frequency domain operations usually only enhance fixed frequency subbands, which may amplify noise and artifacts, resulting in underexposure / overexposure and color distortion. To overcome the limitations of frequency subbands, the global interaction capability of Transformer is used to enhance the entire spectrum of low-light images and restore the frequency to normal levels, as shown in formula (3):

[0057] (3)

[0058] in, Represents structural features (Structural Feature), extracted by deep convolutional network; SDSGA represents the global frequency enhancement operation in the spatial domain; Represents the enhanced clear image, i.e., the low-light adjusted image.

[0059] In step S200, this embodiment uses a dual-domain illumination restorer (DDIRBlock) to enhance the performance of low-light images, and constructs a three-scale U-shaped structure by integrating DDIRBlock into the backbone network. DDIRBlock can enhance the performance of low-light images by taking advantage of the advantages of frequency domain and spatial domain methods.

[0060] Combination Figure 1-3 ,Each DDIR Block includes a frequency selective enhancement module (FSE) and a spatial domain structure guided attention module (SDSGA); FSE selectively enhances the image in the frequency domain, and SDSGA uses the structural features captured by SFG to guide the calculation of self-attention.

[0061] Combination Figure 2 , Figure 3 ,FSE mainly includes a frequency decomposition submodule (Decoupler) and a modulation submodule (Modulator). The Decoupler dynamically decomposes the features into different frequency parts according to the learned filters, and the Modulator emphasizes useful frequencies through channel attention.

[0062] Combination Figure 2 In the Decoupler, a learnable and theoretically proven low-pass filter and its corresponding mid-pass filter are used to generate low-frequency and mid-frequency maps, and the high-frequency map is obtained by subtracting the mid-frequency and low-frequency maps from the input map. These learned filters are shared in the group dimension to balance complexity and feature diversity. For the low-light adjusted image I, a filter generation layer is first used to generate a low-pass filter for each group of the input, as shown in formula (4)

[0063] (4)

[0064] Where BN, W, and GAP represent batch normalization, convolution parameters, and global average pooling, respectively. The Softmax function is applied to each (k×k) filter, where (k×k) is the low-pass filter kernel size. The absolute value of the low-pass filter is subtracted from a matrix of all 1s to obtain the mid-pass filter. For each group feature , where i is the group index, , c represents the channel index, g represents the number of groups, and its low-frequency components and intermediate frequency components The corresponding filter can be used and The remaining is high frequency components , the expressions are:

[0065] (5)

[0066] Among them, (h, w) represents the spatial coordinates of the pixel; (p, q) represents the local offset coordinates of (h, w), that is, the convolution window range of the filter, , given a (k×k) filter, for each pixel (h, w), it will be calculated based on the weights of the surrounding neighboring pixels (for example, (h-1, w-1) to (h+1, w+1)) and the corresponding position of the filter. When p=-1, it corresponds to the pixel above the filter; when p=0, it corresponds to the pixel in the middle of the filter; when p=1, it corresponds to the pixel below the filter. Similarly, q represents the horizontal offset. For example, for each position (h, w), the filter F L [i, p,q] will be multiplied with the corresponding neighborhood pixel X[i,h+p,w+q,c] and accumulated to generate a low-frequency component.

[0067] The low-frequency components, mid-frequency components, and high-frequency components of each channel are added separately to form three frequency graphs. , , .

[0068] Combination Figure 3 , Modulator includes Sum layer, Gap layer, Fc layer, Concat layer, Softmax layer, and Split layer. The Sum layer sums the pixel values ​​of the low-frequency, medium-frequency, and high-frequency components of each channel of the image to obtain a high-dimensional vector composed of multiple channels; the Gap layer performs global average pooling to map the high-dimensional vector to a one-dimensional vector; the Fc layer integrates the one-dimensional vector, as shown in formula (6)

[0069] (6)

[0070] in, is the parameter of the fully connected layer, and three frequency maps are obtained through formula (6) , , The fusion feature Z.

[0071] Then, the Concat layer fuses the features; the Softmax layer calculates the probability distribution; the Split layer separates the data, and the separated data are respectively , , Multiply and then add.

[0072] The specific process in FSE is:

[0073] (1-1) After the low-light adjusted image I is input into the convolution layer, it is split into two paths of information and passed to the corresponding Decoupler respectively. The convolution kernel parameters k in the two Decouplers are different;

[0074] (1-2) Each Decoupler transmits the low-frequency, medium-frequency and high-frequency image information to the corresponding Modulator for feature fusion;

[0075] (1-3) The fused features are transmitted to the Concat layer for feature fusion and then separated. One path of the data is transmitted to the GAP network for global average pooling to expand the global receptive field; the other path is transmitted to the LAP network for local pooling to expand the local receptive field;

[0076] (1-4) The processed data is input into the next level Concat layer for fusion.

[0077] SDSGA includes a structure-guided multi-head self-attention submodule (SG-MSA), which can effectively use structural features to guide self-attention calculations, thereby enhancing the interaction between regions at different frequency levels. The main operation process is:

[0078] (2-1) The output of step (1-4) is used as the input feature F of SG-MSA in ; Input feature F in Linear projection into query (Q), key (K), and value (V) components through an unbiased fully connected layer

[0079]

[0080] in, , is the learnable parameter of the fully connected layer, and T represents the matrix transpose;

[0081] (2-2) Reshape the projected features into tokens X and divide them into k heads, ,in , , i=1,2,...,k;

[0082] (2-3) Reshape the structural feature F into token Y, which is also divided into k heads, ;

[0083] (2-4) Combine Y and value (V) to form a new value V new , ;

[0084] (2-5) Obtaining the self-attention mechanism ;

[0085] (2-6) The self-attention mechanism is applied to each head, and the output token X is obtained by concatenating and combining the position encoding out , and then reshaped to form the output feature F out .

[0086] Output feature F in SDSGA out Transmitted to the feed-forward neural network FFN for processing.

[0087] Based on the DDIRBlock of FSE and SDSGA, the data transmission process of step S200 in this embodiment includes:

[0088] Step S201, inputting the low-light adjusted image X into the convolution layer to obtain image features;

[0089] Step S202, down-sampling is performed after inputting the first DDIR Block, and concatenation is performed with the low-resolution degraded image (i.e., the low-light adjusted image);

[0090] Step S203, down-sampling after the second DDIR Block, and concatenation with the low illumination adjustment image;

[0091] Step S204, generating a first enhanced image after the third DDIR Block, and performing concatenation with the output of the second DDIR Block after upsampling;

[0092] Step S205, after convolution into the fourth DDIR Block, a second enhanced image is generated, and at the same time, a combination operation (concatenation) is performed with the output of the first DDIR Block after upsampling;

[0093] Step S206: Output the residual image I after the fifth DDIR block. re , through the convolution layer and the low-light adjustment image I to obtain the enhanced image I en .

[0094] In step S202 and step S203, the low-resolution degraded image is integrated into the main path through the shallow layer and the layer joint respectively.

[0095] In step S204 and step S205, the second DDIR Block and the first DDIR Block are connected to the third DDIR Block and the fourth DDIR Block through a jump, so as to reduce the information loss caused by the down-sampling branch.

[0096] In step S204 and step S205, the first enhanced image and the second enhanced image have lower resolutions and are used for training.

[0097] In order to optimize the frequency selection process, we introduce the L1 loss function in both the spatial domain and the frequency domain. The spatial domain loss is defined as follows:

[0098]

[0099] Where s represents the index of the input / output image at different scales; represents the number of elements used for normalization, and Corresponding to the output image and the target image respectively. The frequency domain loss is similarly defined as:

[0100]

[0101] in, Represents a Fast Fourier Transform operation.

[0102] The final loss function is expressed as , where the parameters It is set to 0.1 as a factor to balance the loss in the spatial and frequency domains.

[0103] Example

[0104] Multiple datasets are used for evaluation, including LOLv1, LOLv2-real, LOLv2-synthetic, SID, SMID, and SDSD.

[0105] The LOLv1 dataset contains 485 pairs of images for training and 15 pairs of images for testing, each pair of images includes a low-light input image and a corresponding well-exposed reference image. The LOLv2-real dataset contains 689 pairs of training images and 100 pairs of test images, which are from various real environments such as residential areas, campuses, clubs, and streets. The LOLv2-synthetic dataset provides 900 pairs of training images and 100 pairs of test images, which are synthesized by analyzing the illumination distribution of RAW images. This embodiment evaluates a subset of the SID dataset, using the same in-camera signal processing to convert RAW images into RGB format under low-light and good-light conditions. The subset includes 2099 pairs of training image pairs and 598 pairs of test image pairs. The SMID dataset converts RAW data into RGB image pairs under low-light and normal lighting conditions. The benchmark uses 20,809 pairs of short-exposure and long-exposure RAW image pairs, of which 15,763 pairs are used for training and the rest are used for testing. This embodiment also uses a static version of the SDSD dataset for evaluation, which includes indoor and outdoor subsets, each of which provides image pairs under low-light and normal lighting conditions.

[0106] This embodiment compares the method DDLIE-Net proposed in the present invention with LIME, NPE, MEF, DICM and VV.

[0107] This example uses PyTorch to implement DDLIE-Net, and trains and tests it on a Linux system equipped with a 2080Ti GPU. The Adam optimizer is used with the parameters set to and The training samples are generated by randomly cropping 256×256 image patches from low-light / normal-light image pairs. During testing, the input images are padded to multiples of 8×8 using reflection padding. After inference, the padded images are cropped back to their original size. Peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as evaluation metrics.

[0108] The results are as follows:

[0109] (1) Quantitative results. DDLIE-Net is quantitatively compared with various state-of-the-art (SOTA) methods. As shown in Table 1, DDLIE-Net outperforms all baseline methods on the LOL-v1, LOLv2-real, and LOLv2-synthetic datasets. Compared with the current SOTA method Retinexformer, DDLIE-Net achieves 0.18, 0.37, and 1.72 dB improvements in PSNR on the LOL-v1, LOLv2-real, and LOLv2-synthetic datasets, respectively, and 1.7%, 3.1%, and 2.2% improvements in SSIM, respectively. Under the GT average condition, our DDLIE-Net obtains 0.45, 1.03, and 3.39 dB improvements in PSNR, and 2.6%, 3.7%, and 3.3% improvements in SSIM, respectively. Table 2 shows the comparison results on the SID, SMID, SDSD-indoor, and SDSD-outdoor datasets. DDLIE-Net also outperforms all other methods on these datasets. Specifically, compared with the Retinexformer model, DDLIE-Net improves the PSNR scores by 0.24, 0.60, 1.22, and 0.38 dB on the SID, SMID, SDSD-indoor, and SDSD-outdoor datasets, and the SSIM scores by 0.8%, 1.1%, 1.5%, and 1.0%, respectively. These results show that the DDLIE-Net approach achieves new SOTA performance on all evaluated datasets.

[0110] Table 1

[0111]

[0112] Table 2

[0113]

[0114] (2) Qualitative results. Visual comparison of DDLIE-Net and strong baseline methods Figure 5 and Figure 6 shown. Figure 5 It includes visualization results of LIME, NPE, MEF, DICM and VV datasets; Figure 6From top to bottom, they are visual comparisons of LOLv1, LOL-v2-real, LOL-v2-synthetic, and SID. It can be seen that previous methods either cause color distortion, or contain over / underexposed areas and fail to suppress noise, or generate blurry images with black spots and unnatural artifacts. In contrast, DDLIE-Net can effectively enhance areas with poor visibility and low contrast or low light, reliably remove noise without introducing spots and artifacts, and can robustly maintain color.

Claims

1. A dual-domain low-light image enhancement method with frequency selection and structure-guided attention, characterized in that: include: Step S100, converting the input low-light image from the spatial domain to the frequency domain, and performing brightness enhancement on the low-light image to obtain a low-light adjusted image I; using a deep convolutional network to extract structural features F of the low-light image; using the global interaction capability of the Transformer to enhance the entire spectrum of the low-light image and restore the frequency to a normal level; Step S200: using a dual-domain illumination restorer to enhance the performance of low-light images.

2. The method according to claim 1, characterized in that In step S100, a given low-light image is converted from the spatial domain to the frequency domain through a modulator. （1）， in, represents the image in the frequency domain, D(*) represents the operation of the decomposition module, Represents the low-frequency component of the image, represents the intermediate frequency component, represents the high-frequency component, and H, W, and C are the height, width, and number of channels of the input image, respectively.

3. The method according to claim 2, characterized in that The global interaction capability of Transformer is used to enhance the entire spectrum of low-light images and restore the frequency to normal levels, as shown in formula (3): （3）， in, represents structural features extracted by a deep convolutional network; SDSGA represents the global frequency enhancement operation in the spatial domain; Represents the image after brightness enhancement of the low-light image in the frequency domain.

4. The method according to claim 3, characterized in that In step S200, a dual-domain illumination restorer is used to enhance the performance of low-light images, and a three-scale U-shaped structure is constructed by integrating the dual-domain illumination restorer into the backbone network; Step S200 includes: Step S201, inputting the low-light adjusted image X into the convolution layer to obtain image features; Step S202, down-sampling after inputting into the first dual-domain illumination restorer, and combining operation with the low-light adjustment image; Step S203, down-sampling is performed after passing through the second dual-domain illumination restorer, and combined operation is performed with the low illumination adjustment image; Step S204, generating a first enhanced image after passing through the third dual-domain illumination restoration device, and performing a combined operation with the output of the second dual-domain illumination restoration device after upsampling; Step S205, after convolution, the second enhanced image is input into the fourth dual-domain illumination restorer, and at the same time, the second enhanced image is combined with the output of the first dual-domain illumination restorer after upsampling; Step S206: Output the residual image I after passing through the fifth dual-domain illumination restorer re , through the convolution layer and add it to the low-light adjusted image X to get the enhanced image I en。 5. The method according to claim 4, characterized in that In step S202 and step S203, the low-light adjusted image is merged into the main path through the shallow layer and the hierarchical layer respectively.

6. The method according to claim 4, characterized in that In step S204 and step S205, the second dual-domain illumination restorer and the first dual-domain illumination restorer are connected to the third dual-domain illumination restorer and the fourth dual-domain illumination restorer through jump connection.

7. The method according to claim 5 or 6, characterized in that: Each dual-domain illumination restorer includes a frequency selective enhancement module FSE and a spatial domain structure guided attention module SDSGA; The frequency selective enhancement module FSE selectively enhances the input image in the frequency domain; the spatial domain structure guided attention module SDSGA uses structural features to guide the calculation of self-attention.

8. The method according to claim 7, characterized in that The method for the frequency selective enhancement module to selectively enhance the input image in the frequency domain includes: Generate a low-frequency image of the input image through a low-pass filter and its corresponding mid-pass filter and IF , by subtracting the intermediate frequency map from the input image and low frequency graph Get high frequency graph ; The three frequency graphs are obtained by formula (6): , , The fusion feature Z (6), in, is the parameter of the fully connected layer, GAP(*) is the global average pooling layer operation; The fused features are separated into two paths, one of which is transmitted to the GAP network for global average pooling to expand the global receptive field; the other is transmitted to the LAP network for local pooling to expand the local receptive field; The data after global average pooling and local pooling are fused.

9. The method according to claim 8, characterized in that The spatial domain structure guides the attention module to process the fused data F after global average pooling and local pooling. in Do the following: (1) Input feature F in The query Q, key K and value V components are linearly projected through an unbiased fully connected layer. in, , is the learnable parameter of the fully connected layer, and T represents the matrix transpose; (2) Reshape the projected features into tokens X and divide them into k heads, ,in , , i=1,2,...,k; (3) Reshape the structural feature F into token Y, which is also divided into k heads, ; (4) Combine Y and value V to form a new value V new , ; (5) Obtaining a self-attention mechanism ; (6) The self-attention mechanism is applied to each head, and the output token X is obtained by concatenating and combining the position encoding out , and then reshaped to form the output feature F out ; (7) Output feature F out Transmitted to the feed-forward neural network FFN for processing.

Images