Low-light image enhancement method and device based on Fourier transform Transformer network

Abstract

The present invention discloses a low-light image enhancement method and device based on a Fourier transform Transformer network. The present invention first obtains a number of low-light images and corresponding reference standard images, and preprocesses the low-light images, then constructs a Fourier transform Transformer network, and then uses the preprocessed low-light images and corresponding normal light images as samples to train the Fourier transform Transformer network; finally, the low-light image to be enhanced is input into the trained Fourier transform Transformer network to obtain an enhanced image. The present invention can enhance low-light images and even extremely low-light images, and the enhanced image effect is better than the prior art.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a low-light image enhancement method and device based on a Fourier transform Transformer network. Background Art

[0002] Low-light image enhancement is an important and challenging task in computer vision. Images taken in low-light environments usually contain a large proportion of dark areas. These images have problems such as low contrast, high noise, and color deviation, which mask most of the useful information and reduce the performance of many computer vision algorithms, such as object detection and semantic segmentation. Low-light image enhancement provides an effective visual solution for this purpose, which aims to improve the low visibility and low contrast of low-light images and solve a series of image quality degradation problems (such as noise, artifacts, color distortion, etc.). Low-light image enhancement has important applications in many fields, including visual surveillance, autonomous driving, and computational photography. In the fields of visual surveillance and autonomous driving, low-light enhancement technology can enhance low-light or dark-light images in visual surveillance, realize the capture of effective targets in the scene, and ensure their effective analysis; in the field of computational photography, low-light enhancement technology can improve the imaging quality of images taken under low light or backlight conditions, and obtain images that satisfy users.

[0003] Low-light image enhancement has been continuously improved in recent years. With the development of deep learning technology, from the original traditional methods, such as histogram equalization and Retinex theory, to the use of convolutional neural network training to infer low-light image enhancement pictures. However, convolutional neural networks with only local modeling capabilities cannot perceive the overall lighting information of the image, lack of perception of details in different brightness areas, resulting in network performance degradation. Most of the existing methods using Transformer with global modeling capabilities face problems such as large number of parameters and high computational complexity. In addition, most of the loss functions in existing work directly calculate the difference between the network output and the reference image, which affects the convergence ability of the network and cannot produce the optimal enhanced image. Summary of the invention

[0004] Purpose of the invention: In view of the problems existing in the prior art, the present invention provides a low-light image enhancement method and device based on Fourier transform Transformer network with better restoration effect.

[0005] Technical solution: The low-light image enhancement method and device based on Fourier transform Transformer network described in the present invention include:

[0006] S1: Acquire several low-light images and corresponding reference standard images;

[0007] S2: Data preprocessing for low-light images;

[0008] S3: Construct a Fourier transform Transformer network, which is a U-shaped structure network, including a Transformer layer, a downsampling layer, and an upsampling layer. Each Transformer layer includes a multi-head attention module based on Fourier transform and a feedforward network with local feature enhancement. The multi-head attention module based on Fourier transform is used to extract the global information of the low-light image to initially restore the brightness, and the feedforward network with local feature enhancement extracts the local information of the low-light image to further restore the texture features of the image;

[0009] S4: using the low-light image preprocessed by S2 and the corresponding normal-light image as samples to train the Fourier transform Transformer network;

[0010] S5: Input the low-light image to be enhanced into the trained Fourier transform Transformer network to obtain the enhanced image.

[0011] Furthermore, the Fourier transform-based multi-head attention module in S3 specifically includes:

[0012] The Fourier transform layer is used to perform Fourier transform on the low-light image features and convert them from the spatial domain to the frequency domain;

[0013] The frequency domain feature extractor consists of two convolutional layers, an activation function, and a convolutional layer for fusion features, which is used to extract frequency domain features;

[0014] A multi-head attention unit is used to use the frequency domain features of the frequency domain feature extractor as queries and keywords, calculate the frequency domain feature attention map of the low-light image, and perform matrix multiplication on the features in the spatial domain and the frequency domain feature attention map to extract the global features of the low-light image;

[0015] The position encoding layer, which consists of two convolutional layers and an activation function, is used to record the relative position information of the features.

[0016] Furthermore, the Fourier transform layer is implemented by the following formula:

[0017]

[0018] X(u,v)=R(X(u,v))+jI(X(u,v))

[0019] Where F(·) represents the Fourier transform function that transforms the input feature x into the frequency domain X, H and W are the length and width of the input feature x, h and w are the coordinates in the spatial domain, u and v are the coordinates in the frequency domain, j is the imaginary unit, and R(·) and I(·) are the real and imaginary parts of X(u,v), respectively.

[0020] Furthermore, the frequency domain feature extractor is implemented by the following formula:

[0021] R'=φ 1 (R), I'=φ 2 (I)

[0022] F freq =conv(cat(R',I'))

[0023] In the formula, φ 1 (·) and φ 2 (·) represents two convolutional layers with an activation function for processing the real part and the imaginary part respectively, R' represents the real part after processing, I' represents the imaginary part after processing, cat(·) is the feature aggregation operation along the channel dimension, conv is a convolutional layer to complete the feature fusion operation, F freq Represents the frequency domain features extracted by the frequency domain feature extractor.

[0024] Furthermore, the multi-head attention unit is implemented by the following formula:

[0025]

[0026]

[0027] Where W Q ,W K ,W V represents the learnable parameter of the fully connected layer, Τ represents the matrix transpose, Q is the query, K is the keyword, V is the value, α is a learnable parameter, and softmax represents the SoftMax function.

[0028] Furthermore, the local feature enhanced feedforward network is implemented by the following formula:

[0029]

[0030] In the formula, x in represents the input features, They represent the operations of increasing and decreasing the dimension of features along the channel dimension. Conv is a 3*3 convolution. FFN It is the output feature of the local feature enhancement network.

[0031] Furthermore, the S4 specifically includes:

[0032] S4-1: Divide the preprocessed low-light images and the corresponding reference standard images into a training set and a validation set;

[0033] S4-2: Initialize learning rate;

[0034] S4-3: Input the training set into the Fourier transform Transformer network, calculate the network loss based on the network output and the corresponding reference standard image, and optimize the network parameters using the Adam optimizer based on the network loss;

[0035] S4-4: The network parameters that minimize the loss of the validation set are taken as the optimal network parameters;

[0036] S4-5: Initialize the Fourier transform Transformer network using the optimal network parameters to complete network training.

[0037] Furthermore, the method for calculating the network loss in S4-3 specifically includes:

[0038] S4-3-1: Calculate L1 loss according to the following formula:

[0039]

[0040] In the formula, I out is the output image of the network, I gt is the corresponding reference standard image;

[0041] S4-3-2: Calculate the progressive structural loss according to the following formula:

[0042]

[0043] Where SSIM(·) is the structural similarity loss;

[0044] S4-3-3: Calculate network loss according to the following formula:

[0045]

[0046] In the formula, is the network loss, and λ1 is the hyperparameter.

[0047] The low-light image enhancement device based on the Fourier transform Transformer network described in the present invention includes a processor and a computer program stored in a memory and executable on the processor, and the above method is implemented when the processor executes the program.

[0048] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0049] (1) The present invention proposes a multi-head attention module in the Fourier transform Transformer network, which uses a combination of frequency domain information and a multi-head attention mechanism to capture rich global information, and can effectively extract global information to restore image illumination and hidden content;

[0050] (2) The present invention proposes a feedforward network for local feature enhancement, which can restore the edge texture of the image and further improve the enhancement result;

[0051] (3) The present invention proposes a progressive structural loss, which can enable the network to gradually perceive changes in the network structure during the training process, thereby reducing the difficulty of network learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of a low-light image enhancement method and device based on a Fourier transform Transformer network provided by the present invention;

[0053] Figure 2 It is a structural diagram of the Fourier transform Transformer network of the present invention. DETAILED DESCRIPTION

[0054] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] This embodiment specifically provides a low-light image enhancement method and device based on a Fourier transform Transformer network. Figure 1 and Figure 2 As shown, the following steps are included:

[0056] S1: Acquire several low-light images and corresponding reference standard images;

[0057] S2: Data preprocessing for low-light images;

[0058] The preprocessing method specifically includes:

[0059] (1) Performing data augmentation on the low-light image. The data augmentation method is as follows: 1) Randomly flipping the low-light image horizontally with a preset probability; 2) Randomly rotating the low-light image with a preset probability. In this embodiment, the preset probability is specifically 0.5;

[0060] (2) Randomly crop the low-light image to a preset value, which is 128*128 in this embodiment;

[0061] S3: Construct a Fourier transform Transformer network, which is a U-shaped structure network, including a Transformer layer, a downsampling layer, and an upsampling layer. Each Transformer layer includes a multi-head attention module based on Fourier transform and a feedforward network with local feature enhancement. The multi-head attention module based on Fourier transform is used to extract the global information of the low-light image to initially restore the brightness, and the feedforward network with local feature enhancement extracts the local information of the low-light image to further restore the texture features of the image;

[0062] The Fourier transform-based multi-head attention module specifically includes:

[0063] The Fourier transform layer is used to perform Fourier transform on the low-light image features and convert them from the spatial domain to the frequency domain;

[0064] The Fourier transform layer is implemented by the following formula:

[0065]

[0066] X(u,v)=R(X(u,v))+jI(X(u,v))

[0067] Where F(·) represents the Fourier transform function that transforms the input feature x into the frequency domain X, H and W are the length and width of the input feature x, h and w are the coordinates in the spatial domain, u and v are the coordinates in the frequency domain, j is the imaginary unit, and R(·) and I(·) are the real and imaginary parts of X(u,v), respectively.

[0068] The frequency domain feature extractor consists of two convolutional layers, an activation function, and a convolutional layer for fusion features, which is used to extract frequency domain features;

[0069] The frequency domain feature extractor is implemented by the following formula:

[0070] R'=φ 1 (R), I'=φ 2 (I)

[0071] F freq =conv(cat(R',I'))

[0072] In the formula, φ 1 (·) and φ 2 (·) represents two convolutional layers with an activation function for processing the real part and the imaginary part respectively, R' represents the real part after processing, I' represents the imaginary part after processing, cat(·) is the feature aggregation operation along the channel dimension, conv is a convolutional layer to complete the feature fusion operation, F freqRepresents the frequency domain features extracted by the frequency domain feature extractor.

[0073] A multi-head attention unit is used to use the frequency domain features of the frequency domain feature extractor as queries and keywords, calculate the frequency domain feature attention map of the low-light image, and perform matrix multiplication on the features in the spatial domain and the frequency domain feature attention map to extract the global features of the low-light image;

[0074] The multi-head attention unit is implemented by the following formula:

[0075]

[0076]

[0077] Where W Q ,W K ,W V represents the learnable parameter of the fully connected layer, Τ represents the matrix transpose, Q is the query, K is the keyword, V is the value, α is a learnable parameter, and softmax represents the SoftMax function.

[0078] The position encoding layer, which consists of two convolutional layers and an activation function, is used to record the relative position information of the features.

[0079] The local feature enhanced feedforward network is implemented by the following formula:

[0080]

[0081] In the formula, x in represents the input features, They represent the operations of increasing and decreasing the dimension of features along the channel dimension. Conv is a 3*3 convolution. FFN It is the output feature of the local feature enhancement network.

[0082] S4: using the low-light image preprocessed by S2 and the corresponding normal-light image as samples to train the Fourier transform Transformer network;

[0083] Wherein, the S4 specifically includes:

[0084] S4-1: Divide the preprocessed low-light images and the corresponding reference standard images into a training set and a validation set;

[0085] S4-2: Initialize learning rate;

[0086] S4-3: Input the training set into the Fourier transform Transformer network, calculate the network loss based on the network output and the corresponding reference standard image, and optimize the network parameters using the Adam optimizer based on the network loss;

[0087] The calculation method of network loss specifically includes:

[0088] S4-3-1: Calculate L1 loss according to the following formula:

[0089]

[0090] In the formula, I out is the output image of the network, I gt is the corresponding reference standard image;

[0091] S4-3-2: Calculate the progressive structural loss according to the following formula:

[0092]

[0093] Where SSIM(·) is the structural similarity loss;

[0094] S4-3-3: Calculate network loss according to the following formula:

[0095]

[0096] In the formula, is the network loss, and λ1 is the hyperparameter.

[0097] S4-4: The network parameters that minimize the loss of the validation set are taken as the optimal network parameters;

[0098] S4-5: Initialize the Fourier transform Transformer network using the optimal network parameters to complete network training.

[0099] After the training is completed, the test data set is used for testing to obtain the enhanced normal light image, and the PSNR (peak signal-to-noise ratio) and SSIM (structural similarity) of the corresponding reference image are calculated.

[0100] S5: Input the low-light image to be enhanced into the trained Fourier transform Transformer network to obtain the enhanced image.

[0101] This embodiment also provides a low-light image enhancement device based on a Fourier transform Transformer network, including a processor and a computer program stored in a memory and executable on the processor, wherein the processor implements the above method when executing the program.

[0102] The above disclosure is only a preferred embodiment of the present invention, which cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A low-light image enhancement method based on Fourier transform Transformer network, characterized in that: The method includes: S1: Acquire several low-light images and corresponding reference standard images; S2: Data preprocessing for low-light images; S3: Construct a Fourier transform Transformer network, which is a U-shaped structure network, including a Transformer layer, a downsampling layer, and an upsampling layer. Each Transformer layer includes a multi-head attention module based on Fourier transform and a feedforward network with local feature enhancement. The multi-head attention module based on Fourier transform is used to extract global information of low-light images to preliminarily restore brightness, and the feedforward network with local feature enhancement extracts local information of low-light images to further restore texture features of the image. S4: using the low-light image preprocessed by S2 and the corresponding normal-light image as samples to train the Fourier transform Transformer network; S5: Input the low-light image to be enhanced into the trained Fourier transform Transformer network to obtain the enhanced image; The Fourier transform-based multi-head attention module in S3 specifically includes: The Fourier transform layer is used to perform Fourier transform on the low-light image features and convert them from the spatial domain to the frequency domain; The frequency domain feature extractor consists of two convolutional layers, an activation function, and a convolutional layer for fusion features, which is used to extract frequency domain features; A multi-head attention unit is used to use the frequency domain features of the frequency domain feature extractor as queries and keywords, calculate the frequency domain feature attention map of the low-light image, and perform matrix multiplication on the features in the spatial domain and the frequency domain feature attention map to extract the global features of the low-light image; The position encoding layer consists of two convolutional layers and an activation function, which is used to record the relative position information of the features; The Fourier transform layer is implemented by the following formula: X(u,v)=R(X(u,v))+jI(X(u,v)) Where F(·) represents the Fourier transform function that transforms the input feature x into the frequency domain X, H and W are the length and width of the input feature x, h, w are the coordinates in the spatial domain, u, v are the coordinates in the frequency domain, j is the imaginary unit, R(·) and I(·) are the real and imaginary parts of X(u, v), respectively. The frequency domain feature extractor is implemented by the following formula: R'=φ 1 (R),I'=φ 2 (I) F freq =conv(cat(R',I')) In the formula, φ 1 (·) and φ 2 (·) represents two convolutional layers with an activation function for processing the real part and the imaginary part respectively, R' represents the real part after processing, I' represents the imaginary part after processing, cat(·) is the feature aggregation operation along the channel dimension, conv is a convolutional layer to complete the feature fusion operation, F freq represents the frequency domain features extracted by the frequency domain feature extractor; The multi-head attention unit is implemented by the following formula: Where W Q ,W K ,W V represents the learnable parameters of the fully connected layer, Τ represents the matrix transpose, Q is the query, K is the keyword, V is the value, α is a learnable parameter, softmax represents the SoftMax function, x is the input feature, F freq Represents the frequency domain features extracted by the frequency domain feature extractor.

2. The low-light image enhancement method based on Fourier transform Transformer network according to claim 1, characterized in that: The local feature enhanced feedforward network is implemented by the following formula: In the formula, x in Indicates input, They represent the operations of increasing and decreasing the dimension of features along the channel dimension, conv is a convolution, and F FFN It is the output feature of the local feature enhancement network.

3. The low-light image enhancement method based on Fourier transform Transformer network according to claim 1, characterized in that: The S4 specifically includes: S4-1: Divide the preprocessed low-light images and the corresponding reference standard images into a training set and a validation set; S4-2: Initialize learning rate; S4-3: Input the training set into the Fourier transform Transformer network, calculate the network loss based on the network output and the corresponding reference standard image, and optimize the network parameters using the Adam optimizer based on the network loss; S4-4: The network parameters that minimize the loss of the validation set are taken as the optimal network parameters; S4-5: Initialize the Fourier transform Transformer network using the optimal network parameters to complete network training.

4. The low-light image enhancement method based on Fourier transform Transformer network according to claim 3, characterized in that: The method for calculating the network loss in S4-3 specifically includes: S4-3-1: Calculate L1 loss according to the following formula: In the formula, I out is the output image of the network, I gt is the corresponding reference standard image; S4-3-2: Calculate the progressive structural loss according to the following formula: Where SSIM(·) is the structural similarity loss; S4-3-3: Calculate network loss according to the following formula: In the formula, is the network loss, and λ1 is the hyperparameter.

5. A low-light image enhancement device based on a Fourier transform Transformer network, comprising a processor and a computer program stored in a memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 4 is implemented.

Images