Low-light image enhancement method based on high-frequency texture reconstruction

Abstract

This invention discloses a low-light image enhancement method using high-frequency texture reconstruction. It uses an SGHFP integrated CNN module and a Transformer module to enable the network to simultaneously focus on both local and global information about the image's high-frequency information, ultimately fusing the network's output features into a high-frequency image reconstruction. By dynamically adjusting the weights of the loss function to avoid manual assignment, the network is constrained to achieve the optimal fusion result. The final enhanced image is obtained from a visible light image in a low-light environment, and the enhanced image has richer detail information and clearer structural contours. This invention solves the problem that existing enhancement algorithms in the art do not easily produce a clear image with distinct edge contour structures and excellent denoising results.

Description

Technical Field

[0001] The invention belongs to the technical field of computer digital image processing, and in particular relates to a low-light image enhancement method for high-frequency texture reconstruction. Background Art

[0002] Currently, an increasing number of photos are captured in low-light conditions, resulting in low brightness, known as low-light images. Low-light imaging is finding increasingly widespread applications, such as nighttime surveillance and low-light landscape imaging. However, low-light images suffer from poor visual quality, high levels of noise, blurred image details, and reduced human perception of visibility. Directly processing these images for visual tasks, such as object recognition and classification, can also be affected. Therefore, enhancement of these low-light images is necessary. Various enhancement methods are available, including deep learning-based enhancement methods, which can learn feature representations for low-light images from large amounts of data. By training networks to understand patterns in low-light images, higher-level enhancement can be achieved while mitigating the impact of noise. Therefore, deep learning-based image enhancement algorithms are a crucial research area, playing an irreplaceable role in diverse fields such as security surveillance, object detection, and video detection.

[0003] Existing low-light image enhancement methods can be roughly divided into four categories: autoencoder-based enhancement, GAN-based enhancement, traditional CNN-based enhancement, and Transformer-based enhancement. Because existing low-light images have a low signal-to-noise ratio (SNR), containing a large amount of noise and low image signal, and the complex and diverse texture details of the images are intertwined with the noise, existing image enhancement algorithms struggle to achieve clearer images with distinct edge structures and excellent denoising results. Summary of the Invention

[0004] The purpose of the present invention is to provide a low-light image enhancement method for high-frequency texture reconstruction, which solves the problem that the enhancement algorithm in the prior art is not easy to obtain an image with clear edge contour structure and excellent denoising results.

[0005] The technical solution adopted by the present invention is a low-light image enhancement method based on high-frequency texture reconstruction. It adopts SGHFP integrated CNN module and Transformer module, so that the network can focus on both local and global information of the image's high-frequency information, and finally fuse the network output features into high-frequency information to reconstruct the image. By dynamically adjusting the weights of the loss function, manual assignment is avoided to constrain the network to achieve the optimal fusion result, and the final enhanced image is obtained from the visible light image in a low-light environment. The enhanced image has richer detail information and clearer structural contours.

[0006] The present invention is also characterized in that:

[0007] Please follow the steps below to implement it:

[0008] Step 1: Design an image frequency decomposition module IFD to decompose the high-frequency information Y of the image by performing frequency decomposition on the input low-light image X. H and low-frequency information Y L Two parts, respectively performing enhancement processing operations on these two parts of frequency information;

[0009] Step 2: Design a signal-to-noise ratio perception module SNRP based on the Unet network. The signal-to-noise ratio perception module processes the input low-frequency image information to obtain the denoised image low-frequency information Y d and generate a signal-to-noise ratio map M of the low-frequency image;

[0010] Step 3: Design a signal-to-noise ratio-guided high-frequency information reconstruction network (SGHFR) based on CNN and Transformer. Through this network, the local features Y1 and global features Y2 of the high-frequency information of the input image are obtained, and the features of the high-frequency image information are deeply extracted.

[0011] Step 4: Design a high-frequency feature fusion module FCT to selectively fuse the local features Y1 and global features Y2 of the image to obtain the enhanced high-frequency information reconstructed image Y;

[0012] Step 5: Design the fusion frequency module Fusion to reconstruct the final high-frequency information image Y and the denoised image low-frequency information Y d Fusion is performed to obtain the final image enhancement result Y output ;

[0013] Step 6: Build a neural network based on the image frequency decomposition and low-light enhancement method of CNN and Transformer, train it for 1500 epochs, verify the training results, and save the neural network model;

[0014] Step 7: Test the neural network model saved in step 6, load the network model trained in step 6, input the test set into the trained network model, and then save the test results to obtain an enhanced image.

[0015] Step 1 is implemented as follows:

[0016] First, the visible light image in the low-light scene, i.e., the low-light image X, is sent to the image frequency decomposition module IFD. Two sets of dilated convolutions are used to perform convolution operations on the input low-light image X to obtain the image frequency decomposition factor C. a , using the frequency decomposition factor Ca Multiply the input low-light image X to obtain high-frequency information Y H , decompose the frequency into factors C a Take the inverse and multiply the input low-light image X to obtain the image low-frequency information Y L , C a 、Y H and Y L As shown in formulas (1)(2)(3):

[0017] C a =σ(F conv (X)) (1)

[0018] Y H =C a ⊙X (2)

[0019] Y L =(1-C a )⊙X (3)

[0020] Among them F conv Represents the operation of two sets of void convolutions, σ is the sigmoid function, which converts the data into values ​​in the range of 0-1, and ⊙ represents element-by-element multiplication.

[0021] Step 2 is implemented as follows:

[0022] Step 2.1: First, the low-frequency information Y of the image decomposed by the IFD module is L Send it to the SNRP module and use the Unet network to obtain the denoised image low-frequency information Y d , Y d As shown in formula (4):

[0023] Y d =F snrp (Y L ) (4)

[0024] Among them F snrp It is a SNRP module composed of Unet;

[0025] Step 2.2: The low-frequency information Y of the image after denoising obtained in step 2.1 is converted to d And the input image low frequency information Y L Calculate and generate the signal-to-noise ratio map M of the low-frequency image. M is shown in formula (5) (6):

[0026] T=Y d / abs((Y g )-(Y d )), (5)

[0027] M=clamp(T / mean(T)) (6)

[0028] Where abs represents the absolute value operation, T is the preliminary calculated low-frequency signal-to-noise ratio map, mean represents the averaging operation, clamp() represents the normalization function in the PyTorch neural network framework to ensure that the value is in the range of 0 to 1, and M is the final low-frequency signal-to-noise ratio map.

[0029] Step 3 is implemented as follows:

[0030] Step 3.1: First, the high-frequency information Y of the image decomposed by the IFD module H The image is sent to the SGHFR network, and CNN and Transformer are used to obtain the local features Y1 and global features Y2 of the image. Due to its own block principle, Transformer needs to first perform linear projection to obtain the visible light feature sequence Z. Vis ;

[0031] Step 3.2: The feature sequence Z Vis It passes through 1 Transformer layer in sequence; the Transformer layer consists of an MSA module and an MLP module. The specific steps of the MSA module are: first use layer normalization, i.e. LN, to process the feature sequence, then send the processed features to the MSA block, the feature sequence without LN operation is residually connected with the output of MSA, and the result of the residual connection is sent to the MLP module. Similar to the MSA module, the MLP module first uses LN to process the feature sequence, then sends the processed features to the MLP block, the feature sequence without LN operation is residually connected with the output of MLP, and finally obtains the output result of the Transformer layer

[0032] MSA is shown in formulas (1)(2)(3):

[0033] MultiHead(Q,K,V)=Concat(head1,…,head h )W O (1);

[0034] head i =Attention(QW i Q ,KW i K ,VW i V ) (2);

[0035]

[0036] in, d model Indicates the sequence encoding length, Q, K, V represent the query, key and value in matrix form respectively, d k Indicates the dimension of the vector key;

[0037] The MSA module is divided into the following three steps: First, learn multiple Q, K, V matrices separately, which corresponds to multiple learning Q, K, V weight matrices W i Q 、W i K and W i V , input matrix X and corresponding weight matrix W i Multiply them to get the corresponding Q, K, V, then calculate the self-attention for each attention head separately and get the corresponding output Z i Finally, all Z i Spliced ​​into a whole, and then multiplied by an output matrix W O , and finally obtain the local features Y1 and global features Y2 of the image.

[0038] Step 4 is implemented as follows:

[0039] The high-frequency feature fusion module FCT specifically selectively fuses the local features Y1 and global features Y2 of the image obtained in step 3 through the gate control unit ConvWEGT to achieve the interaction between local features and global features;

[0040] ConvWEGT first uses the local features Y1 extracted by CNN and the global features Y2 extracted by Transformer as the two inputs of ConvWEGT. Then, according to the weight R controlled by ConvWEGT, it completes the forgetting and memorizing of features, selectively extracts features, and realizes the information interaction between CNN and Transformer. The gate unit R is shown in formula (10):

[0041] R=σ(w conv (F concat (Y1,Y2,M))) (10);

[0042] σ is the sigmoid function, which converts the data into a value in the range of 0-1 as a gating signal, W conv () is a set of convolution operations, F concat () represents the splicing function in the PyTorch neural network framework, which is used to splice the three images Y1, Y2, and M together;

[0043] After obtaining the fusion weight R, the final fused high-frequency reconstructed image Y is calculated as shown in formula (11):

[0044] Y=(1-R)⊙Y1+(R)⊙Y2 (11);

[0045] In formula (11), ⊙ represents element-by-element multiplication.

[0046] 7. The low-light image enhancement method for high-frequency texture reconstruction according to claim 6, wherein step 5 is specifically implemented according to the following steps:

[0047] The fusion frequency module Fusion reconstructs the final high-frequency information image Y and the denoised image low-frequency information Y d Fusion is performed, and the final image enhancement result Y is obtained through a convolution layer and an activation function layer. output The kernel sizes of all convolutional layers are 3×3, and the activation function uses the ReLU activation function. The number of input channels of the first convolutional layer F1 is 6, and the number of output channels is 3. The number of channels of the fusion result is 3, that is, the color image. The width and height of the image are the same. The fusion result is expressed as the following formula (12):

[0048] Y output =F conv1 (F Relu (F concat (Y d ,Y))) (12)

[0049] Among them, F concat () represents the splicing function in the PyTorch neural network framework, which is used to convert Y d ,Y two pictures are stitched together. F Relu () represents the RELU activation function in the PyTorch neural network framework, F conv1 () represents a convolutional layer in the PyTorch neural network framework, and the output fusion result Y output The number of channels is 3, which is a color map.

[0050] Step 6 is implemented as follows:

[0051] Step 6.1, the loss function consists of three parts, namely L dec , L re and L rec , optimize the loss function and select the weight-adaptive loss function, k represents the number of loss functions, W is the weight of each loss function, b is the bias of each loss function, and the weight-adaptive loss function is expressed as follows (13):

[0052]

[0053] Step 6.2: First, calculate the correctness of the high- and low-frequency decomposition of the image by the IFD module, and use wavelet transform to obtain the true low-frequency image Y of the input image X. g , and then calculate the real low-frequency image and the low-frequency image information Y decomposed by the IFD module L Each pixel difference value, which is used for the real low-frequency image and the low-frequency image information Y decomposed by the IFD module L L dec The formula of the loss function is as follows:

[0054]

[0055] where ∈ is 10 -3 , to prevent the result from being 0, ||( )||2 means to obtain the L2 norm distance within the brackets.

[0056] Step 6.3: First, calculate the denoising effect of low-frequency image information in the SNRP module. d The difference in gradient change before and after is the result;

[0057] Among them, L is used for denoising low-frequency image information. dre The formula of the loss function is as follows:

[0058]

[0059] in and They represent the calculation of pixel gradient changes along the x-axis and y-axis for the denoised low-frequency image respectively;

[0060] Step 6.4: Calculate L rec , input the final fused image Y output And the real image X under normal lighting corresponding to the input low-light image g , calculate the difference between the final enhanced image and the real image,

[0061] L rec The calculation formula is as follows:

[0062]

[0063] in It represents the operation of extracting features using the VGG network in the neural network, and ||( )||1 represents the L1 norm distance;

[0064] Step 6.5, Loss SUM Dynamic learning loss weight, L dec , L re and Lrec By sharing information through adaptive weights, complementing each other, and updating network parameters, the network gradually converges;

[0065] Step 6.6: Select a GPU with 24GB of video memory for training. Use the ADAM optimizer for the network and set the initial learning rate to 0.0001. Use the cosine annealing learning rate scheduler with a step count of 1500. The neural network adaptively adjusts the learning rate each time, training for a total of 1500 epochs.

[0066] Step 7 is implemented as follows:

[0067] Save the neural network parameters trained in step 6, the number of Epochs being trained, the optimizer ADAM, and the scheduler to obtain the trained network model. After each round of training, test the network model trained in that round and save the SSIM value. Before the 1000th round, the SSIM indicator fluctuated greatly, but was generally on an upward trend. After the 1000th round, it stabilized and eventually reached 0.89.

[0068] The beneficial effect of the present invention is that the low-light image enhancement method of high-frequency texture reconstruction maximizes the retention of local features and global representation by interactively fusing convolution local features with Transformer-based global representations. The SGHFR module is used to reconstruct the high-frequency information of the image, which can restore the edge contour detail information of the image and compensate for the local information loss caused by the use of convolution operations to a certain extent. The SNRP is used to denoise the low-frequency information to complete the image denoising function. The low-light image enhancement is achieved by separately processing the detailed features of the high-frequency information and low-frequency information of the input low-light image, and adding a weight-adaptive loss function to adaptively adjust the weight of the loss function to constrain the network to achieve the optimal fusion result. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 Schematic diagram of the overall structure of the low-light image enhancement method for high-frequency texture reconstruction of the present invention;

[0070] Figure 2 Schematic diagram of the structure of the Transformer layer in the feature extraction module SGHFR of the low-light image enhancement method for high-frequency texture reconstruction of the present invention;

[0071] Figure 3 This is a schematic diagram of the effect of the low-light image enhancement method of high-frequency texture reconstruction in a low-light scene. DETAILED DESCRIPTION

[0072] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] The low-light image enhancement method of the present invention is combined with high-frequency texture reconstruction Figure 1 、 Figure 2 , the SGHFP is used to integrate the CNN module and the Transformer module, so that the network can focus on the local and global information of the high-frequency information of the image at the same time, and finally fuse the network output features into high-frequency information to reconstruct the image. By dynamically adjusting the weights of the loss function, manual assignment is avoided to constrain the network to achieve the optimal fusion result, and the final enhanced image is obtained from the visible light image in a low-light environment. The enhanced image has richer detail information and clearer structural contours.

[0074] Please follow the steps below to implement it:

[0075] Step 1: Design an image frequency decomposition module IFD, namely Image Frequency Decomposition block, which decomposes the high-frequency information Y of the image by frequency decomposing the input low-light image X. H and low-frequency information Y L Two parts, respectively performing enhancement processing operations on these two parts of frequency information;

[0076] Step 1 is implemented as follows:

[0077] First, the visible light image in the low-light scene, i.e., the low-light image X, is sent to the image frequency decomposition module IFD. Two sets of dilated convolutions are used to perform convolution operations on the input low-light image X to obtain the image frequency decomposition factor C. a , using the frequency decomposition factor C a Multiply the input low-light image X to obtain high-frequency information Y H , decompose the frequency into factors C a Take the inverse and multiply the input low-light image X to obtain the image low-frequency information Y L , C a 、Y H and Y L As shown in formulas (1)(2)(3):

[0078] C a =σ(F conv (X)) (1)

[0079] Y H =C a ⊙X (2)

[0080] Y L =(1-C a )⊙X (3)

[0081] Among them F convRepresents the operation of two sets of void convolutions, σ is the sigmoid function, which converts the data into values ​​in the range of 0-1, and ⊙ represents element-by-element multiplication.

[0082] Step 2: Design a signal-to-noise ratio perception module SNRP based on the Unet network, namely Signal-to-Noise Ratio Perception block. The signal-to-noise ratio perception module processes the input low-frequency image information to obtain the denoised image low-frequency information Y d and generate a signal-to-noise ratio map M of the low-frequency image;

[0083] Step 2 is implemented as follows:

[0084] Step 2.1: First, the low-frequency information Y of the image decomposed by the IFD module is L Send it to the SNRP module and use the Unet network to obtain the denoised image low-frequency information Y d , Y d As shown in formula (4):

[0085] Y d =F snrp (Y L ) (4)

[0086] Among them F snrp It is a SNRP module composed of Unet;

[0087] Step 2.2: The low-frequency information Y of the image after denoising obtained in step 2.1 is converted to d And the input image low frequency information Y L Calculate and generate the signal-to-noise ratio map M of the low-frequency image. M is shown in formula (5) (6):

[0088] T=Y d / abs((Y g )-(Y d )), (5)

[0089] M=clamp(T / mean(T)) (6)

[0090] Where abs represents the absolute value operation, T is the preliminary calculated low-frequency signal-to-noise ratio map, mean represents the averaging operation, clamp() represents the normalization function in the PyTorch neural network framework to ensure that the value is in the range of 0 to 1, and M is the final low-frequency signal-to-noise ratio map.

[0091] Step 3: Design a SNR Guided High Frequency Reconstruction network (SGHFR) based on CNN and Transformer. This network obtains the local features Y1 and global features Y2 of the input image’s high-frequency information, and deeply extracts the features of the high-frequency image information.

[0092] Step 3 is implemented as follows:

[0093] Step 3.1: First, the high-frequency information Y of the image decomposed by the IFD module H The image is sent to the SGHFR network, and CNN and Transformer are used to obtain the local features Y1 and global features Y2 of the image. Due to its own block principle, Transformer needs to first perform linear projection to obtain the visible light feature sequence Z. Vis ;

[0094] Step 3.2: The feature sequence Z Vis Pass through 1 Transformer layer in sequence; Figure 2 As shown in Figure 2, the Transformer layer consists of an MSA module and an MLP module. The specific steps of the MSA module are: first, use layer normalization, i.e., LN, to process the feature sequence, then send the processed features to the MSA block, and the feature sequence without LN operation is residually connected with the output of MSA, and the result of the residual connection is sent to the MLP module. Similar to the MSA module, the MLP module first uses LN to process the feature sequence, and then sends the processed features to the MLP block, and the feature sequence without LN operation is residually connected with the output of MLP, and finally the output result of the Transformer layer is obtained.

[0095] MSA is shown in formulas (1)(2)(3):

[0096] MultiHead(Q,K,V)=Concat(head1,…,head h )W O (1);

[0097] head i =Attention(QW i Q ,KW i K ,VW i V ) (2);

[0098]

[0099] in, d model Indicates the sequence encoding length, Q, K, V represent the query, key and value in matrix form respectively, d k Indicates the dimension of the vector key;

[0100] The MSA module is divided into the following three steps: First, learn multiple Q, K, V matrices separately, which corresponds to multiple learning Q, K, V weight matrices W i Q 、W i K and W i V , input matrix X and corresponding weight matrix W i Multiply them to get the corresponding Q, K, V, then calculate the self-attention for each attention head separately and get the corresponding output Z i Finally, all Z i Spliced ​​into a whole, and then multiplied by an output matrix W O , and finally obtain the local features Y1 and global features Y2 of the image.

[0101] Step 4: Design a high-frequency feature fusion module FCT, namely the Fusion Cnn Transformer Module, which selectively fuses the local features Y1 and the global features Y2 of the image to obtain the enhanced high-frequency information reconstructed image Y;

[0102] Step 4 is implemented as follows:

[0103] The high-frequency feature fusion module FCT specifically selectively fuses the local features Y1 and global features Y2 of the image obtained in step 3 through the gate control unit ConvWEGT to achieve the interaction between local features and global features;

[0104] ConvWEGT first uses the local features Y1 extracted by CNN and the global features Y2 extracted by Transformer as the two inputs of ConvWEGT. Then, according to the weight R controlled by ConvWEGT, it completes the forgetting and memorizing of features, selectively extracts features, and realizes the information interaction between CNN and Transformer. The gate unit R is shown in formula (10):

[0105] R=σ(w conv (F concat (Y1,Y2,M))) (10);

[0106] σ is the sigmoid function, which converts the data into a value in the range of 0-1 as a gating signal, Wconv () is a set of convolution operations, F concat () represents the splicing function in the PyTorch neural network framework, which is used to splice the three images Y1, Y2, and M together;

[0107] After obtaining the fusion weight R, the final fused high-frequency reconstructed image Y is calculated as shown in formula (11):

[0108] Y=(1-R)⊙Y1+(R)⊙Y2 (11);

[0109] In formula (11), ⊙ represents element-by-element multiplication.

[0110] Step 5: Design the fusion frequency module Fusion, that is, Fusion Module, to reconstruct the final high-frequency information image Y and the denoised image low-frequency information Y d Fusion is performed to obtain the final image enhancement result Y output ;

[0111] Step 5 is implemented as follows:

[0112] The fusion frequency module Fusion reconstructs the final high-frequency information image Y and the denoised image low-frequency information Y d Fusion is performed, and the final image enhancement result Y is obtained through a convolution layer and an activation function layer. output The kernel sizes of all convolutional layers are 3×3, and the activation function uses the ReLU activation function. The number of input channels of the first convolutional layer F1 is 6, and the number of output channels is 3. The number of channels of the fusion result is 3, that is, the color image. The width and height of the image are the same. The fusion result is expressed as the following formula (12):

[0113] Y output =F conv1 (F Relu (F concat (Y d ,Y))) (12)

[0114] Among them, F concat () represents the splicing function in the PyTorch neural network framework, which is used to convert Y d ,Y two pictures are stitched together. F Relu () represents the RELU activation function in the PyTorch neural network framework, F conv1 () represents a convolutional layer in the PyTorch neural network framework, and the output fusion result Y output The number of channels is 3, which is a color map.

[0115] Step 6: Build a neural network based on the image frequency decomposition and low-light enhancement method of CNN and Transformer, train it for 1500 epochs, verify the training results, and save the neural network model;

[0116] Step 6 is implemented as follows:

[0117] Step 6.1, the loss function consists of three parts, namely L dec , L re and L rec , optimize the loss function and select the weight-adaptive loss function, k represents the number of loss functions, W is the weight of each loss function, b is the bias of each loss function, and the weight-adaptive loss function is expressed as follows (13):

[0118]

[0119] Step 6.2: First, calculate the correctness of the high- and low-frequency decomposition of the image by the IFD module, and use wavelet transform to obtain the true low-frequency image Y of the input image X. g , and then calculate the real low-frequency image and the low-frequency image information Y decomposed by the IFD module L Each pixel difference value, which is used for the real low-frequency image and the low-frequency image information Y decomposed by the IFD module L L dec The formula of the loss function is as follows:

[0120]

[0121] where ∈ is 10 -3 , to prevent the result from being 0, ||( )||2 means to obtain the L2 norm distance within the brackets.

[0122] Step 6.3, first calculate the denoising effect of the low-frequency information of the image in the SNRP module. Since it is very difficult to obtain a clean image after denoising an image, the calculation of the low-frequency image information denoising Y is adopted. d The difference in gradient change before and after is the result;

[0123] Among them, L is used for denoising low-frequency image information. dre The formula of the loss function is as follows:

[0124]

[0125] in and They represent the calculation of pixel gradient changes along the x-axis and y-axis for the denoised low-frequency image respectively;

[0126] Step 6.4: Calculate Lrec , input the final fused image Y output And the real image X under normal lighting corresponding to the input low-light image g , calculate the difference between the final enhanced image and the real image,

[0127] L rec The calculation formula is as follows:

[0128]

[0129] in It represents the operation of extracting features using the VGG network in the neural network, and ||( )||1 represents the L1 norm distance;

[0130] Step 6.5, Loss SUM Dynamic learning loss weight, L dec , L re and L rec By sharing information through adaptive weights, complementing each other, and updating network parameters, the network gradually converges;

[0131] Step 6.6: Select a GPU with 24GB of video memory for training. Use the ADAM optimizer for the network and set the initial learning rate to 0.0001. Use the cosine annealing learning rate scheduler with a step count of 1500. The neural network adaptively adjusts the learning rate each time, training for a total of 1500 epochs.

[0132] Step 7: Test the neural network model saved in step 6, load the network model trained in step 6, input the test set into the trained network model, and then save the test results to obtain an enhanced image.

[0133] Step 7 is implemented as follows:

[0134] Save the neural network parameters trained in step 6, the number of Epochs being trained, the optimizer ADAM, and the scheduler to obtain the trained network model. After each round of training, test the network model trained in that round and save the SSIM value. Before the 1000th round, the SSIM indicator fluctuated greatly, but was generally on an upward trend. After the 1000th round, it stabilized and eventually reached 0.89.

[0135] like Figure 3 The figure shows the effect of the present invention in a low-light scene. Each column in the figure represents an image under the same environment. The first column of images is the input low-light image, the second row is the result of enhancing the input low-light image, and the third column of images is the real image under normal light corresponding to the first column of images.

[0136] The low-light image enhancement method for high-frequency texture reconstruction employs a SGHFP integrated CNN module and a Transformer module, enabling the network to simultaneously focus on both local and global high-frequency information in the image. Ultimately, the network output features are fused to reconstruct the high-frequency information. By dynamically adjusting the weights of the loss function, manual weight assignment is avoided, and the network is constrained to achieve the optimal fusion result. The final enhanced image, obtained from a visible light image in a low-light environment, exhibits richer detail and clearer structural outlines, demonstrating its practical significance.

[0137] Example 1

[0138] The low-light image enhancement method of the present invention is combined with high-frequency texture reconstruction Figure 1 、 Figure 2 , the SGHFP is used to integrate the CNN module and the Transformer module, so that the network can focus on the local and global information of the high-frequency information of the image at the same time, and finally fuse the network output features into high-frequency information to reconstruct the image. By dynamically adjusting the weights of the loss function, manual assignment is avoided to constrain the network to achieve the optimal fusion result, and the final enhanced image is obtained from the visible light image in a low-light environment. The enhanced image has richer detail information and clearer structural contours.

[0139] Example 2

[0140] The low-light image enhancement method for high-frequency texture reconstruction of the present invention is specifically implemented according to the following steps:

[0141] Step 1: Design an image frequency decomposition module IFD, which decomposes the high-frequency information Y of the image by performing frequency decomposition on the input low-light image X. H and low-frequency information Y L Two parts, respectively performing enhancement processing operations on these two parts of frequency information;

[0142] Step 2: Design a signal-to-noise ratio perception module SNRP based on the Unet network, namely Signal-to-Noise Ratio Perception block. The signal-to-noise ratio perception module processes the input low-frequency image information to obtain the denoised image low-frequency information Y d and generate a signal-to-noise ratio map M of the low-frequency image;

[0143] Step 3: Design a SNR Guided High Frequency Reconstruction network (SGHFR) based on CNN and Transformer. This network obtains the local features Y1 and global features Y2 of the input image’s high-frequency information, and deeply extracts the features of the high-frequency image information.

[0144] Step 4: Design a high-frequency feature fusion module FCT, namely the Fusion Cnn Transformer Module, which selectively fuses the local features Y1 and the global features Y2 of the image to obtain the enhanced high-frequency information reconstructed image Y;

[0145] Step 5: Design the fusion frequency module Fusion, that is, Fusion Module, to reconstruct the final high-frequency information image Y and the denoised image low-frequency information Y d Fusion is performed to obtain the final image enhancement result Y output ;

[0146] Step 6: Build a neural network based on the image frequency decomposition and low-light enhancement method of CNN and Transformer, train it for 1500 epochs, verify the training results, and save the neural network model;

[0147] Step 7: Test the neural network model saved in step 6, load the network model trained in step 6, input the test set into the trained network model, and then save the test results to obtain an enhanced image.

[0148] Example 3

[0149] The low-light image enhancement method for high-frequency texture reconstruction of the present invention is specifically implemented according to the following steps:

[0150] Step 1: Design an image frequency decomposition module IFD, which decomposes the high-frequency information Y of the image by performing frequency decomposition on the input low-light image X. H and low-frequency information Y L Two parts, respectively performing enhancement processing operations on these two parts of frequency information;

[0151] Step 1 is implemented as follows:

[0152] First, the visible light image in the low-light scene, i.e., the low-light image X, is sent to the image frequency decomposition module IFD. Two sets of dilated convolutions are used to perform convolution operations on the input low-light image X to obtain the image frequency decomposition factor C. a , using the frequency decomposition factor Ca Multiply the input low-light image X to obtain high-frequency information Y H , decompose the frequency into factors C a Take the inverse and multiply the input low-light image X to obtain the image low-frequency information Y L , C a 、Y H and Y L As shown in formulas (1)(2)(3):

[0153] C a =σ(F conv (X)) (1)

[0154] Y H =C a ⊙X (2)

[0155] Y L =(1-C a )⊙X (3)

[0156] Among them F conv Represents the operation of two sets of void convolutions, σ is the sigmoid function, which converts the data into values ​​in the range of 0-1, and ⊙ represents element-by-element multiplication.

[0157] Step 2: Design a signal-to-noise ratio perception module SNRP based on the Unet network, namely Signal-to-Noise Ratio Perception block. The signal-to-noise ratio perception module processes the input low-frequency image information to obtain the denoised image low-frequency information Y d and generate a signal-to-noise ratio map M of the low-frequency image;

[0158] Step 2 is implemented as follows:

[0159] Step 2.1: First, the low-frequency information Y of the image decomposed by the IFD module is L Send it to the SNRP module and use the Unet network to obtain the denoised image low-frequency information Y d , Y d As shown in formula (4):

[0160] Y d =F snrp (Y L ) (4)

[0161] Among them F snrp It is a SNRP module composed of Unet;

[0162] Step 2.2: The low-frequency information Y of the image after denoising obtained in step 2.1 is converted to d And the input image low frequency information YL Calculate and generate the signal-to-noise ratio map M of the low-frequency image. M is shown in formula (5) (6):

[0163] T=Y d / abs((Y g )-(Y d )), (5)

[0164] M=clamp(T / mean(T)) (6)

[0165] Where abs represents the absolute value operation, T is the preliminary calculated low-frequency signal-to-noise ratio map, mean represents the averaging operation, clamp() represents the normalization function in the PyTorch neural network framework to ensure that the value is in the range of 0 to 1, and M is the final low-frequency signal-to-noise ratio map.

[0166] Step 3: Design a SNR Guided High Frequency Reconstruction network (SGHFR) based on CNN and Transformer. This network obtains the local features Y1 and global features Y2 of the input image’s high-frequency information, and deeply extracts the features of the high-frequency image information.

[0167] Step 3 is implemented as follows:

[0168] Step 4: Design a high-frequency feature fusion module FCT, namely the Fusion Cnn Transformer Module, which selectively fuses the local features Y1 and the global features Y2 of the image to obtain the enhanced high-frequency information reconstructed image Y;

[0169] Step 5 is implemented as follows:

[0170] The fusion frequency module Fusion reconstructs the final high-frequency information image Y and the denoised image low-frequency information Y d Fusion is performed, and the final image enhancement result Y is obtained through a convolution layer and an activation function layer. output The kernel sizes of all convolutional layers are 3×3, and the activation function uses the ReLU activation function. The number of input channels of the first convolutional layer F1 is 6, and the number of output channels is 3. The number of channels of the fusion result is 3, that is, the color image. The width and height of the image are the same. The fusion result is expressed as the following formula (12):

[0171] Y output =F conv1 (F Relu (F concat (Y d,Y))) (12)

[0172] Among them, F concat () represents the splicing function in the PyTorch neural network framework, which is used to convert Y d ,Y two pictures are stitched together. F Relu () represents the RELU activation function in the PyTorch neural network framework, F conv1 () represents a convolutional layer in the PyTorch neural network framework, and the output fusion result Y output The number of channels is 3, which is a color map.

[0173] Step 6: Build a neural network based on the image frequency decomposition and low-light enhancement method of CNN and Transformer, train it for 1500 epochs, verify the training results, and save the neural network model;

[0174] Step 6 is implemented as follows:

[0175] Step 6.1, the loss function consists of three parts, namely L dec , L re and L rec , optimize the loss function and select the weight-adaptive loss function, k represents the number of loss functions, W is the weight of each loss function, b is the bias of each loss function, and the weight-adaptive loss function is expressed as follows (13):

[0176]

[0177] Step 6.2: First, calculate the correctness of the high- and low-frequency decomposition of the image by the IFD module, and use wavelet transform to obtain the true low-frequency image Y of the input image X. g , and then calculate the real low-frequency image and the low-frequency image information Y decomposed by the IFD module L Each pixel difference value, which is used for the real low-frequency image and the low-frequency image information Y decomposed by the IFD module L L dec The formula of the loss function is as follows:

[0178]

[0179] where ∈ is 10 -3 , to prevent the result from being 0, ||( )||2 means to obtain the L2 norm distance within the brackets.

[0180] Step 6.3, first calculate the denoising effect of the low-frequency information of the image in the SNRP module. Since it is very difficult to obtain a clean image after denoising an image, the calculation of the low-frequency image information denoising Y is adopted. dThe difference in gradient change before and after is the result;

[0181] Among them, L is used for denoising low-frequency image information. dre The formula of the loss function is as follows:

[0182]

[0183] in and They represent the calculation of pixel gradient changes along the x-axis and y-axis for the denoised low-frequency image respectively;

[0184] Step 6.4: Calculate L rec , input the final fused image Y output And the real image X under normal lighting corresponding to the input low-light image g , calculate the difference between the final enhanced image and the real image,

[0185] L rec The calculation formula is as follows:

[0186]

[0187] in It represents the operation of extracting features using the VGG network in the neural network, and ||( )||1 represents the L1 norm distance;

[0188] Step 6.5, Loss SUM Dynamic learning loss weight, L dec , L re and L rec By sharing information through adaptive weights, complementing each other, and updating network parameters, the network gradually converges;

[0189] Step 6.6: Select a GPU with 24GB of video memory for training. Use the ADAM optimizer for the network and set the initial learning rate to 0.0001. Use the cosine annealing learning rate scheduler with a step count of 1500. The neural network adaptively adjusts the learning rate each time, training for a total of 1500 epochs.

[0190] Step 7: Test the neural network model saved in step 6, load the network model trained in step 6, input the test set into the trained network model, and then save the test results to obtain an enhanced image.

[0191] Step 7 is implemented as follows:

[0192] Save the neural network parameters trained in step 6, the number of Epochs being trained, the optimizer ADAM, and the scheduler to obtain the trained network model. After each round of training, test the network model trained in that round and save the SSIM value. Before the 1000th round, the SSIM indicator fluctuated greatly, but was generally on an upward trend. After the 1000th round, it stabilized and eventually reached 0.89.

Claims

1. A low-light image enhancement method based on high-frequency texture reconstruction, characterized in that: Please follow the steps below to implement: Step 1: Design an image frequency decomposition module IFD, which is based on the input low-light image X Perform frequency decomposition to decompose the high-frequency information of the image Y H and low-frequency information Y L Two parts, respectively performing enhancement processing operations on these two parts of frequency information; Step 2: Design a signal-to-noise ratio perception module SNRP based on the Unet network. The signal-to-noise ratio perception module processes the input low-frequency image information to obtain the denoised low-frequency image information. Y d and generate a signal-to-noise ratio map of the low-frequency image M ; Step 3: Design a high-frequency information reconstruction network SGHFR based on CNN and Transformer, through which the local features of the high-frequency information of the input image are obtained. Y 1 and global features Y 2. Deeply extract features of high-frequency image information; Step 4: Design the high-frequency feature fusion module FCT to combine the local features of the image Y 1 and global features Y 2 are selectively fused together to obtain the enhanced high-frequency information reconstructed image Y; Step 5: Design the fusion frequency module Fusion to reconstruct the final high-frequency information image Y and the denoised image low-frequency information Y d Perform fusion to obtain the final image enhancement result Y output ; Step 6: Build a neural network based on the image frequency decomposition and low-light enhancement method of CNN and Transformer, train it for 1500 epochs, verify the training results, and save the neural network model; Step 7: Test the neural network model saved in step 6, load the network model trained in step 6, input the test set into the trained network model, and then save the test results to obtain an enhanced image; The SGHFP integrated CNN module and Transformer module are used to enable the network to focus on both local and global information of the image's high-frequency information, and ultimately fuse the network output features into high-frequency information to reconstruct the image. By dynamically adjusting the weights of the loss function, manual assignment is avoided to constrain the network to achieve the optimal fusion result. The final enhanced image is obtained from the visible light image in a low-light environment, and the enhanced image has richer detail information and clearer structural contours.

2. The low-light image enhancement method for high-frequency texture reconstruction according to claim 1, characterized in that: The step 1 is specifically implemented according to the following steps: First, the visible light image in the low light scene, that is, the low illumination image X Send to the image frequency decomposition module IFD , using two sets of dilated convolutions to input low-light images X Perform convolution operation to obtain image frequency decomposition factor C a , using the frequency factorization C a And the input low-light image X Multiply to get high-frequency information Y H , factorize the frequency C a Negated and input low-light image X Multiply to get the low-frequency information of the image Y L , C a 、Y H and Y L As shown in formulas (1), (2), and (3): C a = (F conv (X))(1) （2） （3） in F conv Represents two sets of dilated convolution operations, It is the sigmoid function, which converts the data into values ​​within the range of 0-1. Represents element-wise multiplication.

3. The low-light image enhancement method for high-frequency texture reconstruction according to claim 2, characterized in that: The step 2 is specifically implemented according to the following steps: Step 2.1: First, the low-frequency information of the image decomposed by the IFD module Y L Send it to the SNRP module and use the Unet network to obtain the denoised low-frequency information of the image Y d , Y d As shown in formula (4): Y d = F snrp (Y L )(4) in F snrp It is a SNRP module composed of Unet; Step 2.2: Denoise the low-frequency image information obtained in step 2.1 Y d and the input image low-frequency information Y L Calculate and generate the signal-to-noise ratio map of the low-frequency image M , M As shown in formula (5) and (6): （5） M = clamp(T / mean(T))(6) in abs Represents the absolute value operation, T is the preliminary calculated low-frequency signal-to-noise ratio map, mean represents the averaging operation, clamp() Represents the normalization function in the PyTorch neural network framework, ensuring that the value is in the range of 0 to 1. M is the final low-frequency signal-to-noise ratio map.

4. The low-light image enhancement method for high-frequency texture reconstruction according to claim 3, characterized in that: The step 3 is specifically implemented according to the following steps: Step 3.1: First, the high-frequency information of the image decomposed by the IFD module Y H Send it to the SGHFR network and use CNN and Transformer to obtain the local features of the image Y 1 and global features Y 2. Due to its own block principle, Transformer needs to first perform linear projection to obtain the visible light feature sequence ; Step 3.2: Sequence the features Pass through 1 Transformer layer in sequence; The Transformer layer consists of an MSA module and an MLP module. The specific steps of the MSA module are: first use layer normalization, i.e. LN, to process the feature sequence, then send the processed features to the MSA block, the feature sequence without LN operation is residually connected with the output of MSA, and the result of the residual connection is sent to the MLP module. The MLP module first uses LN to process the feature sequence, then sends the processed features to the MLP block, the feature sequence without LN operation is residually connected with the output of MLP, and finally obtains the output result of the Transformer layer ; MSA is shown in formulas (1), (2), and (3): （1）； （2）； （3）； in, , , , , represents the sequence encoding length, Q , K , V Represent the matrix form query , key and value , Represents a vector key Dimensions; The MSA module is divided into three steps: First, learn multiple Q , K , V Matrix, so it corresponds to multiple learning Q , K , V The weight matrix 、 and , input matrix X And the corresponding weight matrix Multiply them together to get the corresponding Q , K , V , then, calculate the self-attention for each attention head separately and get the corresponding output Finally, all Concatenate into a whole and then multiply by an output matrix , and finally obtain the local features of the image Y 1 and global features Y 2.

5. The low-light image enhancement method for high-frequency texture reconstruction according to claim 4, characterized in that: The step 4 is specifically implemented according to the following steps: The high-frequency feature fusion module FCT specifically combines the local features of the image obtained in step 3 Y 1 and global features Y 2. Feature fusion is selectively performed through the gate control unit ConvWEGT to achieve the interaction between local features and global features; ConvWEGT first extracts the local features from CNN Y 1 and the global features extracted by Transformer Y 2 as the two inputs of ConvWEGT, and then the weights updated according to the control in ConvWEGT R , complete the forgetting and memorizing of features, selectively extract features, realize the information interaction between CNN and Transformer, gate unit R As shown in formula (10): R = （w conv （F concat （Y 1 ,Y 2 ,M）））（ 10）； Is the sigmoid function, through Convert the data into values ​​in the range of 0-1 to serve as the gating signal. W conv （） is a set of convolution operations, F concat （） Represents the splicing function in the PyTorch neural network framework, which is used to Y 1 ,Y 2 ,M Three images are stitched together; After obtaining the fusion weight R, the final fused high-frequency reconstructed image Y, Y The calculation of is shown in formula (11): （11）； In formula (11), Represents element-wise multiplication.

6. The low-light image enhancement method for high-frequency texture reconstruction according to claim 5, characterized in that: The step 5 is specifically implemented according to the following steps: The fusion frequency module Fusion reconstructs the image with the final high-frequency information Y And the low-frequency information of the denoised image Y d Fusion is performed and the final image enhancement result is obtained through a convolution layer and an activation function layer. Y output , the kernelsizes of one convolutional layer are , the activation function adopts ReLU activation function, the number of input channels of the first convolution layer F1 is 6, and the number of output channels is 3. The number of channels of the fusion result is 3, that is, the color image, the width and height of the image are the same. The fusion result is expressed as the following formula (12): Y output = F conv1 （F Relu （F concat （Y d ,Y））） （12） in, F concat （） Represents the splicing function in the PyTorch neural network framework, which is used to Y d ,Y Stitch the two pictures together. F Relu （） Represents the RELU activation function in the PyTorch neural network framework, F conv1 （） Represents a convolutional layer in the PyTorch neural network framework, outputting the fusion result Y output The number of channels is 3, which is a color map.

7. The low-light image enhancement method for high-frequency texture reconstruction according to claim 6, characterized in that: The step 6 is specifically implemented according to the following steps: Step 6.1, the loss function consists of three parts, namely L dec 、 L re and L rec , optimize the loss function and select the weight-adaptive loss function, k represents the number of loss functions, W is the weight of each loss function, b is the bias of each loss function, and the weight-adaptive loss function is expressed as follows (13): （13）； Step 6.2: First, calculate the correctness of the high- and low-frequency decomposition of the image by the IFD module, and use the wavelet transform to obtain the true low-frequency image of the input image X. Y g , and then calculate the real low-frequency image and the low-frequency image information decomposed by the IFD module Y L The difference value of each pixel, which is used for the real low-frequency image and the low-frequency image information decomposed by the IFD module Y L of L dec The formula of the loss function is as follows: L dec = （14）； in is 10 −3 , to prevent the result from being 0, Indicates finding the L2 norm distance in the brackets; Step 6.3: First, calculate the denoising effect of low-frequency image information in the SNRP module. Y d The difference in gradient change before and after is the result; Among them, the one used for low-frequency image information denoising L dre The formula of the loss function is as follows: （15）； in and They represent the calculation of pixel gradient changes along the x-axis and y-axis for the denoised low-frequency image respectively; Step 6.4, calculation L rec , input the final fused image Y output and Input low-light image corresponding to the real image under normal lighting X g ， Calculate the difference between the final enhanced image and the real image, L rec The calculation formula is as follows: L rec = （16）； in Represents the operation of extracting features using the VGG network in the neural network, represents the L1 norm distance; Step 6.5 Dynamic learning loss weights, L dec 、 L re and L rec By sharing information through adaptive weights, complementing each other, and updating network parameters, the network gradually converges; Step 6.6: Select a GPU with 24GB of video memory for training. Use the ADAM optimizer for the network and set the initial learning rate to 0.0001. Use the cosine annealing learning rate scheduler with a step count of 1500. The neural network adaptively adjusts the learning rate each time, training for a total of 1500 epochs.

8. The low-light image enhancement method for high-frequency texture reconstruction according to claim 7, characterized in that: The step 7 is specifically implemented according to the following steps: Save the neural network parameters trained in step 6, the number of epochs being trained, the optimizer ADAM, and the scheduler to obtain the trained network model. After each round of training, test the network model trained in that round and save the SSIM value. Before the 1000th round, the SSIM indicator fluctuated greatly, but was generally on an upward trend. After the 1000th round, it stabilized and eventually reached 0.89.

Images