Wavelet change weak light image enhancement method

Abstract

The invention aims to provide a weak light image enhancement method based on wavelet variation, which comprises the following steps of: firstly, designing a wavelet variation module, and performing wavelet variation and noise removal on an input low-illumination image Input to obtain reconstruction feature information Y1; then designing a deep detail perception module, and performing deep detail extraction on the reconstructed feature information Y1 to obtain detail feature information MFSD; and a feature fusion module is designed, reconstruction feature information Y1 and detail feature information MFSD of the image are fused together, an enhanced reconstruction image Yw and the like are obtained, an existing public weak light data set LOL serves as a test set and is input into the trained network model, then a test result is stored, and an enhanced image is obtained. The problem that an enhancement algorithm in the prior art is not easy to obtain an image with clear local fine-grained texture and continuous large-scale texture is solved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer digital image processing, and in particular relates to a wavelet-transformed weak-light image enhancement method. Background Art

[0002] With the widespread use of shooting equipment in low-light environments, the problem of insufficient image brightness has become increasingly prominent. In scenes such as nighttime monitoring and low-light landscape photography, image quality is often limited, such as high noise, blurred details, and poor visibility. Such low-light images perform poorly in visual perception tasks such as object recognition and classification, seriously affecting the accuracy and robustness of the tasks. Therefore, how to effectively improve the quality of low-light images has become an urgent problem that needs to be solved. In recent years, low-light image enhancement technology has received widespread attention as an important image processing method. In particular, methods based on deep learning, relying on their powerful data learning and feature extraction capabilities, can improve image brightness, reduce noise interference, and retain important detail information in the image, showing significant results. In the fields of security monitoring and target recognition, deep learning-driven image enhancement technology has become one of the key research directions.

[0003] Existing low-light image enhancement algorithms can be mainly divided into several categories, including encoder-decoder structure, generative adversarial network (GAN), traditional convolutional neural network (CNN) and Transformer model, and diffusion model, etc. Since low-light images generally have low signal-to-noise ratio and complex texture structure, how to effectively separate and enhance the effective information in the image and maintain the clarity of image details is still a major challenge in current research. Summary of the invention

[0004] The purpose of the present invention is to provide a wavelet-transformed weak-light image enhancement method, which solves the problem that the enhancement algorithm in the prior art is not easy to obtain an image with clear local fine-grained texture and continuous large-scale texture.

[0005] The technical solution adopted by the present invention is a low-light image enhancement method based on wavelet transformation, which is specifically implemented according to the following steps:

[0006] Step 1, design a wavelet transformation module, and obtain the reconstructed feature information Y1 by performing wavelet transformation and noise removal on the input low-light image Input;

[0007] Step 2: Design a deep detail perception module to extract deep details from the reconstructed feature information Y1 obtained in step 1 to obtain detail feature information M_FSD;

[0008] Step 3: Design a feature fusion module to fuse the image reconstruction feature information Y1 and detail feature information M_FSD to obtain the enhanced reconstructed image Y w ;

[0009] Step 4: Design a block-based self-attention module to reconstruct the image Y w Perform local and global detail enhancement to obtain the final enhanced image Output;

[0010] Step 5: Build a neural network, train it, verify the training results, use a custom loss function to constrain all neural network parameters, let the neural network back propagate automatically adjust the parameters to achieve the minimum loss function value, and save the neural network model;

[0011] Step 6: Load the neural network model trained in step 5, use the existing public low-light data set LOL as a test set, input it into the trained network model, and then save the test results to obtain an enhanced image.

[0012] The present invention is also characterized in that:

[0013] Step 1 is implemented according to the following steps:

[0014] First, the visible light image in the low-light scene, i.e., the low-illumination image Input (X), is sent to the wavelet transform module. The wavelet transform module consists of the wavelet decomposition operation DWT, the wavelet synthesis operation IWT and four ordinary convolutions in the pytorch neural network framework. The wavelet decomposition operation DWT is used to decompose the input low-illumination image Input to obtain a low-frequency image LL, a horizontal high-frequency image LH, a vertical high-frequency image HL and a diagonal high-frequency image HH. Then, four parallel convolution operations are performed to extract features from these four images respectively to obtain feature maps LL1, LH1, HL1 and HH1 respectively. These four feature maps are subjected to the wavelet synthesis operation IDT to obtain the reconstructed feature information Y1. The calculation of LL, LH, HL, HH, LL1, LH1, HL1, HH1 and Y1 is shown in formulas (1)(2)(3)(4)(5)(6)(7)(8)(9):

[0015] LL=F DWT (X) (1)

[0016] LH=F DWT (X) (2)

[0017] HL=F DWT (X) (3)

[0018] HH=F DWT (X) (4)

[0019] LL1=FCONV1 (X) (5)

[0020] LH1=F CONV2 (X) (6)

[0021] HL1=F CONV3 (X) (7)

[0022] HH1=F CONV4 (X) (8)

[0023] Y1=F IWT (LL1,LH1,HL1,HH1) (9)

[0024] Among them, F DWT and F IDT Represents the wavelet decomposition operation DWT and wavelet synthesis operation IWT in the pytorch neural network framework, F conv1 、F conv2 、F conv3 and F conv4 It is the convolution operation in the pytorch neural network framework.

[0025] Step 2 is implemented as follows:

[0026] First, the reconstructed feature information Y1 obtained in step 1 is sent to the depth detail perception module, which consists of a downsampling and an upsampling operation. The downsampling and upsampling are both composed of a convolution operation and a Relu activation function in the Pytorch neural network framework. After a downsampling and an upsampling operation, the detail feature information M_FSD is obtained;

[0027] M_FSD is shown in formula (10)(11)(12):

[0028] M_FSD=F UP (F DOWN (Y1)) (10)

[0029] F DOWN =Relu(F CONV5 ()) (11)

[0030] F UP =Relu(F CONV6 ()) (12)

[0031] Among them, F DOWN and F UP represents a downsampling and an upsampling operation, F conv5 and F conv6Both are convolution operations in the pytorch neural network framework, and Relu is the activation function in the pytorch neural network framework.

[0032] Step 3 is implemented as follows:

[0033] First, the image reconstruction feature information Y1 and detail feature information M_FSD of step 1 and step 2 are input into the feature fusion module. The feature fusion module consists of two dilated convolution operations and two Relu activation functions in the Pytorch neural network framework. The feature fusion weight factors α1 and α2 are obtained. The feature fusion weight factor α1 is multiplied with the input reconstruction feature information Y1 to obtain the feature information X1. The feature fusion weight factor α2 is multiplied with the detail feature information M_FSD to obtain the feature information X2. The feature information X1 and X2 are added to obtain the fused reconstructed image Y w ; α1, α2, X1, X2 and Y w The calculation is shown in formula (13)(14)(15)(16)(17):

[0034] α1=Relu(F DCONV1 (Y1)) (13)

[0035] α2=Relu(F DCONV2 (M_FSD)) (14)

[0036] X1=α1×Y1 (15)

[0037] X2=α2×M_FSD (16)

[0038] Y W =X1+X2 (17)

[0039] Among them, F DCONV1 and F DCONV2 Represents the operation of two sets of dilated convolutions. The Relu function is the activation function in the pytorch neural network framework. × represents element-by-element multiplication, and + represents element-by-element addition.

[0040] Step 4 is implemented according to the following steps:

[0041] Step 4.1: First, reconstruct the image Y w As the input of the block-based self-attention module, the block-based self-attention module is composed of a standard mutil self-attention module MSA in the pytorch neural network framework. The self-attention mechanism needs to first perform linear projection to obtain the visible light feature sequence Z Vis ;

[0042] Step 4.2: The feature sequence ZVis Through the MSA module: first use the layer normalization LN in the pytorch neural network framework to process the feature sequence, then send the processed features to the MSA block, and perform a residual connection between the feature sequence without LN operation and the output of MSA, and finally obtain the output result Output of the multi-scale self-attention mechanism;

[0043] MSA is shown in formula (18)(19)(20):

[0044] MultiHead(Q,K,V)=Concat(head1,…,head h )W O (18);

[0045]

[0046]

[0047] in, d model represents the sequence encoding length, Q, K, and V represent the query, key, and value in matrix form, respectively, and d k Indicates the dimension of the vector key;

[0048] The operation of the MSA module is divided into the following three steps: First, multiple Q, K, and V matrices are learned separately, so the corresponding weight matrices of Q, K, and V are learned and Input matrix X and corresponding weight matrix W i Multiply them to get the corresponding Q, K, V, then calculate the self-attention for each attention head separately and get the corresponding output Z i Finally, all Z i Spliced ​​into a whole, and then multiplied by an output matrix W O , and finally the final enhanced image Output of the image is obtained.

[0049] Step 5 is implemented according to the following steps:

[0050] Step 5.1, the loss function consists of four parts, namely L vgg and L dec , optimize the loss function and select the weight-adaptive loss function, k represents the number of loss functions, W is the weight of each loss function, b is the bias of each loss function, and the weight-adaptive loss function is expressed as follows (21):

[0051]

[0052] Step 5.2: Calculate the correctness of the global feature extraction module, input the final enhanced image Output and the real image X under normal lighting corresponding to the input low-light image g , calculate the difference L between the final enhanced image and the real image vgg ;

[0053]

[0054] in, represents the operation of extracting features using the VGG network in the neural network, and ||()||1 represents the L1 norm distance;

[0055] Step 5.3: Calculate the enhancement effect of the final image, input the final enhanced image Output and the real image X under normal lighting corresponding to the input low-light image g , calculate the difference L between the final enhanced image Output and the real image dec ,

[0056] L dec The calculation formula is as follows:

[0057]

[0058] where ∈ is 10 -3 , to prevent the result from being 0, ‖( )‖2 means to obtain the L2 norm distance in the brackets;

[0059] Step 5.4, Loss SUM Dynamic learning loss weight, L vgg and L dec By sharing information through adaptive weights, complementing each other, and updating network parameters, the network gradually converges;

[0060] Step 5.5, choose to train on a GPU with a video memory of 16GB, use the ADAM optimizer as the network optimizer, set the initial learning rate to 0.0001, and set the number of adjustment steps to 500. The neural network will adaptively adjust the learning rate each time, and train a total of 500 Epochs. Save the trained neural network parameters, the number of Epochs being trained, the optimizer ADAM, and the scheduler to obtain the trained network model.

[0061] Step 6 is implemented according to the following steps:

[0062] After each round of training, the network model of that round of training is tested and the value of the structural similarity SSIM is saved. The value range of SSIM is between 0 and 1. The closer it is to 1, the more complete the detailed information of the image.

[0063] The beneficial effect of the present invention is that the wavelet-transformed low-light image enhancement method combines frequency domain information reconstruction and spatial domain information change through a two-stage enhancement method, retains local details and global structures to the maximum extent, and uses a block-based self-attention module to extract fine-grained detail information of the image, which can supplement the local texture detail information of the image, compensate for the local information loss caused by the convolution operation to a certain extent, ensure the clarity of the local fine-grained texture, and also extract the global contextual structural features of the image, obtain the structural information of the entire image, and ensure the continuity of large-scale texture details. The wavelet transformation module is used to remove noise and detail enhancement from the perspective of the frequency domain, and the block-based self-attention module is used to achieve texture restoration, and a weight-adaptive loss function is added to adaptively adjust the weight of the loss function to constrain the network to achieve the optimal enhancement result. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a schematic diagram of the overall structure of the weak-light image enhancement method based on wavelet transformation of the present invention;

[0065] Figure 2 It is a schematic diagram of the effect of the low-light image enhancement method based on wavelet transformation in low-light scenes of the present invention. DETAILED DESCRIPTION

[0066] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] The low-light image enhancement method based on wavelet transformation of the present invention is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0068] Step 1, design a wavelet transformation module, and obtain the reconstructed feature information Y1 by performing wavelet transformation and noise removal on the input low-light image Input;

[0069] Step 1 is implemented according to the following steps:

[0070] First, the visible light image in the low-light scene, i.e., the low-illumination image Input (X), is sent to the wavelet transform module. The wavelet transform module consists of the wavelet decomposition operation DWT, the wavelet synthesis operation IWT and four ordinary convolutions in the pytorch neural network framework. The wavelet decomposition operation DWT is used to decompose the input low-illumination image Input to obtain a low-frequency image LL, a horizontal high-frequency image LH, a vertical high-frequency image HL and a diagonal high-frequency image HH. Then, four parallel convolution operations are performed to extract features from these four images respectively to obtain feature maps LL1, LH1, HL1 and HH1 respectively. These four feature maps are subjected to the wavelet synthesis operation IDT to obtain the reconstructed feature information Y1. The calculation of LL, LH, HL, HH, LL1, LH1, HL1, HH1 and Y1 is shown in formulas (1)(2)(3)(4)(5)(6)(7)(8)(9):

[0071] LL=F DWT (X) (1)

[0072] LH=F DWT (X) (2)

[0073] HL=F DWT (X) (3)

[0074] HH=F DWT (X) (4)

[0075] LL1=F CONV1 (X) (5)

[0076] LH1=F CONV2 (X) (6)

[0077] HL1=F CONV3 (X) (7)

[0078] HH1=F CONV4 (X) (8)

[0079] Y1=F IWT (LL1,LH1,HL1,HH1) (9)

[0080] Among them, F DWT and F IDT Represents the wavelet decomposition operation DWT and wavelet synthesis operation IWT in the pytorch neural network framework, F conv1 、F conv2 、F conv3 and F conv4 It is the convolution operation in the pytorch neural network framework.

[0081] Step 2: Design a deep detail perception module to extract deep details from the reconstructed feature information Y1 obtained in step 1 to obtain detail feature information M_FSD;

[0082] Step 2 is implemented as follows:

[0083] First, the reconstructed feature information Y1 obtained in step 1 is sent to the depth detail perception module, which consists of a downsampling and an upsampling operation. The downsampling and upsampling are both composed of a convolution operation and a Relu activation function in the Pytorch neural network framework. After a downsampling and an upsampling operation, the detail feature information M_FSD is obtained;

[0084] M_FSD is shown in formula (10)(11)(12):

[0085] M_FSD=F UP (F DOWN (Y1)) (10)

[0086] F DOWN =Relu(F CONV5 ()) (11)

[0087] F UP =Relu(F CONV6 ()) (12)

[0088] Among them, F DOWN and F UP represents a downsampling and an upsampling operation, F conv5 and F conv6 Both are convolution operations in the pytorch neural network framework, and Relu is the activation function in the pytorch neural network framework.

[0089] Step 3: Design a feature fusion module to fuse the image reconstruction feature information Y1 and detail feature information M_FSD to achieve denoising of information in the frequency domain and enhancement of details, and obtain the enhanced reconstructed image Y w ;

[0090] Step 3 is implemented as follows:

[0091] First, the image reconstruction feature information Y1 and detail feature information M_FSD of step 1 and step 2 are input into the feature fusion module. The feature fusion module consists of two dilated convolution operations and two Relu activation functions in the Pytorch neural network framework. The feature fusion weight factors α1 and α2 are obtained. The feature fusion weight factor α1 is multiplied with the input reconstruction feature information Y1 to obtain the feature information X1. The feature fusion weight factor α2 is multiplied with the detail feature information M_FSD to obtain the feature information X2. The feature information X1 and X2 are added to obtain the fused reconstructed image Y w ; α1, α2, X1, X2 and Y w The calculation is shown in formula (13)(14)(15)(16)(17):

[0092] α1=Relu(F DCONV1 (Y1)) (13)

[0093] α2=Relu(F DCONV2 (M_FSD)) (14)

[0094] X1=α1×Y1 (15)

[0095] X2=α2×M_FSD (16)

[0096] Y W =X1+X2 (17)

[0097] Among them, F DCONV1 and F DCONV2 Represents the operation of two sets of dilated convolutions. The Relu function is the activation function in the pytorch neural network framework. × represents element-by-element multiplication, and + represents element-by-element addition.

[0098] Step 4: Design a block-based self-attention module to reconstruct the image Y w Perform local and global detail enhancement to obtain the final enhanced image Output;

[0099] Step 4 is implemented according to the following steps:

[0100] Step 4.1: First, reconstruct the image Y w As the input of the block-based self-attention module, the block-based self-attention module is composed of a standard mutil self-attention module MSA in the pytorch neural network framework. Because of its own block principle, the self-attention mechanism needs to first perform linear projection to obtain the visible light feature sequence Z Vis ;

[0101] Step 4.2: The feature sequence Z VisThrough the MSA module: first use the layer normalization LN in the pytorch neural network framework to process the feature sequence, then send the processed features to the MSA block, and perform a residual connection between the feature sequence without LN operation and the output of MSA, and finally obtain the output result Output of the multi-scale self-attention mechanism;

[0102] MSA is shown in formula (18)(19)(20):

[0103] MultiHead(Q,K,V)=Concat(head1,…,h,…,head h )W O (18);

[0104]

[0105]

[0106] in, d model represents the sequence encoding length, Q, K, and V represent the query, key, and value in matrix form, respectively, and d k Indicates the dimension of the vector key;

[0107] The operation of the MSA module is divided into the following three steps: First, multiple Q, K, V matrices are learned separately, so the corresponding weight matrices W of multiple learning Q, K, V are i Q , W i K and W i V , input matrix X and corresponding weight matrix W i Multiply them to get the corresponding Q, K, V, then calculate the self-attention for each attention head separately and get the corresponding output Z i Finally, all Z i Spliced ​​into a whole, and then multiplied by an output matrix W O , and finally the final enhanced image Output of the image is obtained.

[0108] Step 5: Build a neural network based on the wavelet-based low-light image enhancement method, train it, verify the training results, use a custom loss function to constrain all neural network parameters, let the neural network back propagate automatically adjust the parameters to achieve the minimum loss function value, and save the neural network model;

[0109] Step 5 is implemented according to the following steps:

[0110] Step 5.1, the loss function consists of four parts, namely L vgg and Ldec , optimize the loss function and select the weight-adaptive loss function, k represents the number of loss functions, W is the weight of each loss function, b is the bias of each loss function, and the weight-adaptive loss function is expressed as follows (21):

[0111]

[0112] Step 5.2: Calculate the correctness of the global feature extraction module, input the final enhanced image Output and the real image X under normal lighting corresponding to the input low-light image g , calculate the difference L between the final enhanced image and the real image vgg ;

[0113]

[0114] in, represents the operation of extracting features using the VGG network in the neural network, and ||()||1 represents the L1 norm distance;

[0115] Step 5.3: Calculate the enhancement effect of the final image, input the final enhanced image Output and the real image X under normal lighting corresponding to the input low-light image g , calculate the difference L between the final enhanced image Output and the real image dec ,

[0116] L dec The calculation formula is as follows:

[0117]

[0118] where ∈ is 10 -3 , to prevent the result from being 0, ||( )||2 means to obtain the L2 norm distance in the brackets;

[0119] Step 5.4, Loss SUM Dynamic learning loss weight, L vgg and L dec By sharing information through adaptive weights, complementing each other, and updating network parameters, the network gradually converges;

[0120] Step 5.5, choose to train on a GPU with a video memory of 16GB, use the ADAM optimizer as the network optimizer, set the initial learning rate to 0.0001, and set the number of adjustment steps to 500. The neural network will adaptively adjust the learning rate each time, and train a total of 500 Epochs. Save the trained neural network parameters, the number of Epochs being trained, the optimizer ADAM, and the scheduler to obtain the trained network model.

[0121] Step 6: Test the neural network model trained in step 5. Load the neural network model trained in step 5, use the existing public low-light data set LOL as a test set, input it into the trained network model, and then save the test results to obtain an enhanced image.

[0122] Step 6 is implemented according to the following steps:

[0123] After each round of training, the network model of that round of training is tested and the value of the structural similarity SSIM is saved. The value range of SSIM is between 0 and 1. The closer it is to 1, the more complete the detailed information of the image.

[0124] Example 1

[0125] The low-light image enhancement method based on wavelet transformation of the present invention is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0126] Step 1, design a wavelet transformation module, and obtain the reconstructed feature information Y1 by performing wavelet transformation and noise removal on the input low-light image Input;

[0127] Step 2: Design a deep detail perception module to extract deep details from the reconstructed feature information Y1 obtained in step 1 to obtain detail feature information M_FSD;

[0128] Step 3: Design a feature fusion module to fuse the image reconstruction feature information Y1 and detail feature information M_FSD to achieve denoising of information in the frequency domain and enhancement of details, and obtain the enhanced reconstructed image Y w ;

[0129] Step 4: Design a block-based self-attention module to reconstruct the image Y w Perform local and global detail enhancement to obtain the final enhanced image Output;

[0130] Step 5: Build a neural network based on the wavelet transform low-light image enhancement method, train for 500 epochs, verify the training results, use the custom loss function to constrain all neural network parameters, let the neural network back propagate automatically adjust the parameters to achieve the minimum loss function value, and save the neural network model;

[0131] Step 6: Test the neural network model trained in step 5. Load the neural network model trained in step 5, use the existing public low-light data set LOL as a test set, input it into the trained network model, and then save the test results to obtain an enhanced image.

[0132] Example 2

[0133] The low-light image enhancement method based on wavelet transformation of the present invention is as follows: Figure 1 As shown, the specific implementation steps are as follows:

[0134] Step 1, design a wavelet transformation module, and obtain the reconstructed feature information Y1 by performing wavelet transformation and noise removal on the input low-light image Input;

[0135] Step 1 is implemented according to the following steps:

[0136] First, the visible light image in the low-light scene, i.e., the low-illumination image Input (X), is sent to the wavelet transform module. The wavelet transform module consists of the wavelet decomposition operation DWT, the wavelet synthesis operation IWT and four ordinary convolutions in the pytorch neural network framework. The wavelet decomposition operation DWT is used to decompose the input low-illumination image Input to obtain a low-frequency image LL, a horizontal high-frequency image LH, a vertical high-frequency image HL and a diagonal high-frequency image HH. Then, four parallel convolution operations are performed to extract features from these four images respectively to obtain feature maps LL1, LH1, HL1 and HH1 respectively. These four feature maps are subjected to the wavelet synthesis operation IDT to obtain the reconstructed feature information Y1. The calculation of LL, LH, HL, HH, LL1, LH1, HL1, HH1 and Y1 is shown in formulas (1)(2)(3)(4)(5)(6)(7)(8)(9):

[0137] LL=F DWT (X) (1)

[0138] LH=F DWT (X) (2)

[0139] HL=F DWT (X) (3)

[0140] HH=F DWT (X) (4)

[0141] LL1=F CONV1 (X) (5)

[0142] LH1=F CONV2 (X) (6)

[0143] HL1=F CONV3 (X) (7)

[0144] HH1=F CONV4 (X) (8)

[0145] Y1=F IWT (LL1,LH1,HL1,HH1) (9)

[0146] Among them, F DWT and FIDT Represents the wavelet decomposition operation DWT and wavelet synthesis operation IWT in the pytorch neural network framework, F conv1 、F conv2 、F conv3 and F conv4 It is the convolution operation in the pytorch neural network framework.

[0147] Step 2: Design a deep detail perception module to extract deep details from the reconstructed feature information Y1 obtained in step 1 to obtain detail feature information M_FSD;

[0148] Step 2 is implemented as follows:

[0149] First, the reconstructed feature information Y1 obtained in step 1 is sent to the depth detail perception module, which consists of a downsampling and an upsampling operation. The downsampling and upsampling are both composed of a convolution operation and a Relu activation function in the Pytorch neural network framework. After a downsampling and an upsampling operation, the detail feature information M_FSD is obtained;

[0150] M_FSD is shown in formula (10)(11)(12):

[0151] M_FSD=F UP (F DOWN (Y1)) (10)

[0152] F DOWN =Relu(F CONV5 ()) (11)

[0153] F UP =Relu(F CONV6 ()) (12)

[0154] Among them, F DOWN and F UP represents a downsampling and an upsampling operation, F conv5 and F conv6 Both are convolution operations in the pytorch neural network framework, and Relu is the activation function in the pytorch neural network framework.

[0155] Step 3: Design a feature fusion module to fuse the image reconstruction feature information Y1 and detail feature information M_FSD to achieve denoising of information in the frequency domain and enhancement of details, and obtain the enhanced reconstructed image Y w ;

[0156] Step 4: Design a block-based self-attention module to reconstruct the image Y wPerform local and global detail enhancement to obtain the final enhanced image Output;

[0157] Step 4 is implemented according to the following steps:

[0158] Step 4.1: First, reconstruct the image Y w As the input of the block-based self-attention module, the block-based self-attention module is composed of a standard mutil self-attention module MSA in the pytorch neural network framework. Because of its own block principle, the self-attention mechanism needs to first perform linear projection to obtain the visible light feature sequence Z Vis ;

[0159] Step 4.2: The feature sequence Z Vis Through the MSA module: first use the layer normalization LN in the pytorch neural network framework to process the feature sequence, then send the processed features to the MSA block, and perform a residual connection between the feature sequence without LN operation and the output of MSA, and finally obtain the output result Output of the multi-scale self-attention mechanism;

[0160] MSA is shown in formula (18)(19)(20):

[0161] MultiHead(Q,K,V)=Concat(head1,…,h,…,head h )W O (18);

[0162]

[0163]

[0164] in, d model represents the sequence encoding length, Q, K, and V represent the query, key, and value in matrix form, respectively, and d k Indicates the dimension of the vector key;

[0165] The operation of the MSA module is divided into the following three steps: First, multiple Q, K, and V matrices are learned separately, so the corresponding weight matrices of Q, K, and V are learned and Input matrix X and corresponding weight matrix W i Multiply them to get the corresponding Q, K, V, then calculate the self-attention for each attention head separately and get the corresponding output Z i Finally, all Z i Spliced ​​into a whole, and then multiplied by an output matrix W O, and finally the final enhanced image Output of the image is obtained.

[0166] Step 5: Build a neural network based on the wavelet transform low-light image enhancement method, train 500 epochs, verify the training results, use the custom loss function to constrain all neural network parameters, let the neural network back propagate automatically adjust the parameters to achieve the minimum loss function value, and save the neural network model;

[0167] Step 5 is implemented according to the following steps:

[0168] Step 5.1, the loss function consists of four parts, namely L vgg and L dec , optimize the loss function and select the weight-adaptive loss function, k represents the number of loss functions, W is the weight of each loss function, b is the bias of each loss function, and the weight-adaptive loss function is expressed as follows (21):

[0169]

[0170] Step 5.2: Calculate the correctness of the global feature extraction module, input the final enhanced image Output and the real image X under normal lighting corresponding to the input low-light image g , calculate the difference L between the final enhanced image and the real image vgg ;

[0171]

[0172] in, represents the operation of extracting features using the VGG network in the neural network, and ||()||1 represents the L1 norm distance;

[0173] Step 5.3: Calculate the enhancement effect of the final image, input the final enhanced image Output and the real image X under normal lighting corresponding to the input low-light image g , calculate the difference L between the final enhanced image Output and the real image dec ,

[0174] L dec The calculation formula is as follows:

[0175]

[0176] where ∈ is 10 -3 , to prevent the result from being 0, ‖( )‖2 means to obtain the L2 norm distance in the brackets;

[0177] Step 5.4, Loss SUM Dynamic learning loss weight, L vggand L dec By sharing information through adaptive weights, complementing each other, and updating network parameters, the network gradually converges;

[0178] Step 5.5, choose to train on a GPU with a video memory of 16GB, use the ADAM optimizer as the network optimizer, set the initial learning rate to 0.0001, and set the number of adjustment steps to 500. The neural network will adaptively adjust the learning rate each time, and train a total of 500 Epochs. Save the trained neural network parameters, the number of Epochs being trained, the optimizer ADAM, and the scheduler to obtain the trained network model.

[0179] Step 6: Test the neural network model trained in step 5. Load the neural network model trained in step 5, use the existing public low-light data set LOL as a test set, input it into the trained network model, and then save the test results to obtain an enhanced image.

[0180] Step 6 is implemented according to the following steps:

[0181] After each round of training, the network model trained in that round is tested and the value of structural similarity SSIM is saved. The value range of SSIM is between 0 and 1. The closer it is to 1, the more complete the detailed information of the image is. Before the 100th round, the SSIM index fluctuated greatly, but it was generally on an upward trend. After the 300th round, it tended to stabilize and finally reached 0.92.

[0182] Example 3

[0183] The performance of the entire neural network is verified using structural similarity (SSIM). The numerical range of SSIM is between 0 and 1. The closer it is to 1, the clearer the details of the image. The low-light image dataset (LOL) provided by the pyotch neural network framework is used. This dataset has 485 pairs of low-light and normal-light image pairs as training sets, and 15 pairs of low-light and normal-light image pairs as test sets. Each training of 485 pairs is shuffled and sent to the neural network for a total of 500 rounds of training. SSIM is used to verify the final image enhancement effect, and the neural network parameters with SSIM values ​​closer to 1 are saved. The performance of image enhancement is tested again with the test set to ensure the consistency of SSIM value training and testing.

[0184] like Figure 2 The figure shows the effect diagram of the present invention in a low-light scene. Each column in the figure represents an image in the same environment. The first column of images is the input low-light image, the second row is the result of enhancing the input low-light image, and the third column of images is the real image under normal light corresponding to the first column of images.

[0185] The low-light image enhancement method based on wavelet transformation of the present invention adopts wavelet transformation module, depth detail perception module, feature fusion module and block-based self-attention module, so that the network can pay attention to the fine-grained texture details and large-scale texture details of the image at the same time. Finally, the network output features are enhanced to reconstruct the image, and the weights of the loss function are dynamically adjusted to avoid manual assignment to constrain the network to achieve the optimal enhancement result. The final enhanced image is obtained from the visible light image in the low-light environment, and the enhanced image has richer detail information and clearer texture structure, which has certain practical significance.

Claims

1. A low-light image enhancement method based on wavelet transformation, characterized in that: Follow the steps below to implement it: Step 1, design the wavelet transform module; Step 2: Design a depth detail perception module; Step 3: Design feature fusion module; Step 4: Design a block-based self-attention module; Step 5: Build a neural network, perform training, verify the training results, and save the neural network model; Step 6: Load the neural network model trained in step 5, use the existing public low-light data set LOL as a test set, input it into the trained network model, and then save the test results to obtain an enhanced image.

2. The wavelet-based low-light image enhancement method according to claim 1, characterized in that: The step 1 is specifically implemented according to the following steps: First, the visible light image in the low-light scene, i.e., the low-illumination image Input (X), is sent to the wavelet transform module. The wavelet transform module consists of the wavelet decomposition operation DWT, the wavelet synthesis operation IWT and four ordinary convolutions in the pytorch neural network framework. The wavelet decomposition operation DWT is used to decompose the input low-illumination image Input to obtain a low-frequency image LL, a horizontal high-frequency image LH, a vertical high-frequency image HL and a diagonal high-frequency image HH. Then, four parallel convolution operations are performed to extract features from these four images respectively to obtain feature maps LL1, LH1, HL1 and HH1 respectively. These four feature maps are subjected to the wavelet synthesis operation IDT to obtain the reconstructed feature information Y1. The calculation of LL, LH, HL, HH, LL1, LH1, HL1, HH1 and Y1 is shown in formulas (1)(2)(3)(4)(5)(6)(7)(8)(9): LL=F DWT (X) (1) LH=F DWT (X) (2) <h2 style=";text-align:left;direction:ltr">HL=F<h2 style=";text-align:left;direction:ltr"> DWT <h2 style=";text-align:left;direction:ltr"> (X) (3) HH=F DWT (X) (4) LL1=F CONV1 (X) (5) LH1=F CONV2 (X) (6) HL1=F CONV3 (X) (7) HH1=F CONV4 (X) (8) Y1=F IWT (LL1,LH1,HL1,HH1) (9) Among them, F DWT and F IDT Represents the wavelet decomposition operation DWT and wavelet synthesis operation IWT in the pytorch neural network framework, F conv1 、F conv2 、F conv3 and F conv4 It is the convolution operation in the pytorch neural network framework.

3. The wavelet-based low-light image enhancement method according to claim 2, characterized in that: The step 2 is specifically implemented according to the following steps: First, the reconstructed feature information Y1 obtained in step 1 is sent to the depth detail perception module, which consists of a downsampling and an upsampling operation. The downsampling and upsampling are both composed of a convolution operation and a Relu activation function in the Pytorch neural network framework. After a downsampling and an upsampling operation, the detail feature information M_FSD is obtained; M_FSD is shown in formula (10)(11)(12): M_FSD=F UP (F DOWN (Y1)) (10) F DOWN =Relu(F CONV5 ()) (11) F UP =Relu(F CONV6 ()) (12) Among them, F DOWN and F UP represents a downsampling and an upsampling operation, F conv5 and F conv6 Both are convolution operations in the pytorch neural network framework, and Relu is the activation function in the pytorch neural network framework.

4. The wavelet-based low-light image enhancement method according to claim 3, characterized in that: The step 3 is specifically implemented according to the following steps: First, the image reconstruction feature information Y1 and detail feature information M_FSD of step 1 and step 2 are input into the feature fusion module. The feature fusion module consists of two dilated convolution operations and two Relu activation functions in the Pytorch neural network framework. The feature fusion weight factors α1 and α2 are obtained. The feature fusion weight factor α1 is multiplied with the input reconstruction feature information Y1 to obtain the feature information X1. The feature fusion weight factor α2 is multiplied with the detail feature information M_FSD to obtain the feature information X2. The feature information X1 and X2 are added to obtain the fused reconstructed image Y w ; α1, α2, X1, X2 and Y w The calculation is shown in formula (13)(14)(15)(16)(17): α1=Relu(F DCONV1 (Y1)) (13) α2=Relu(F DCONV2 (M_FSD)) (14) X1=α1×Y1 (15) X2=α2×M_FSD (16) Y W =X1+X2 (17) Among them, F DCONV1 and F DCONV2 Represents the operation of two sets of dilated convolutions. The Relu function is the activation function in the pytorch neural network framework. × represents element-by-element multiplication, and + represents element-by-element addition.

5. The wavelet-based low-light image enhancement method according to claim 4, characterized in that: The step 4 is specifically implemented according to the following steps: Step 4.1: First, reconstruct the image Y w As the input of the block-based self-attention module, the block-based self-attention module is composed of a standard mutil self-attention module MSA in the pytorch neural network framework. The self-attention mechanism needs to first perform linear projection to obtain the visible light feature sequence Z Vis ; Step 4.2: The feature sequence Z Vis Through the MSA module: first use the layer normalization LN in the pytorch neural network framework to process the feature sequence, then send the processed features to the MSA block, and perform a residual connection between the feature sequence without LN operation and the output of MSA, and finally obtain the output result Output of the multi-scale self-attention mechanism; MSA is shown in formula (18)(19)(20): MultiHead(Q,K,V)=Concat(head1,…,head h )W O (18); head i =Attention(QW i Q ,KW i K ,VW i V ) (19); in, d model represents the sequence encoding length, Q, K, and V represent the query, key, and value in matrix form, respectively. k Indicates the dimension of the vector key; The operation of the MSA module is divided into the following three steps: First, multiple Q, K, V matrices are learned separately, so the corresponding weight matrices W of multiple learning Q, K, V are i Q , W i K and W i V , input matrix X and corresponding weight matrix W i Multiply them to get the corresponding Q, K, V, then calculate the self-attention for each attention head separately and get the corresponding output Z i Finally, all Z i Spliced ​​into a whole, and then multiplied by an output matrix W O , and finally the final enhanced image Output of the image is obtained.

6. The wavelet-based low-light image enhancement method according to claim 5, characterized in that: The step 5 is specifically implemented according to the following steps: Step 5.1, the loss function consists of four parts, namely L vgg and L dec , optimize the loss function and select the weight-adaptive loss function, k represents the number of loss functions, W is the weight of each loss function, b is the bias of each loss function, and the weight-adaptive loss function is expressed as follows (21): Step 5.2: Calculate the correctness of the global feature extraction module, input the final enhanced image Output and the real image X under normal lighting corresponding to the input low-light image g , calculate the difference L between the final enhanced image and the real image vgg ; in, represents the operation of extracting features using the VGG network in the neural network, and ||( )||1 represents the L1 norm distance; Step 5.3: Calculate the enhancement effect of the final image, input the final enhanced image Output and the real image X under normal lighting corresponding to the input low-light image g , calculate the difference L between the final enhanced image Output and the real image dec , L dec The calculation formula is as follows: where ∈ is 10 -3 , to prevent the result from being 0, ||( )||2 means to obtain the L2 norm distance in the brackets; Step 5.4, Loss SUM Dynamic learning loss weight, L vgg and L dec By sharing information through adaptive weights, complementing each other, and updating network parameters, the network gradually converges; Step 5.5, choose to train on a GPU with a video memory of 16GB, use the ADAM optimizer as the network optimizer, set the initial learning rate to 0.0001, and set the number of adjustment steps to 500. The neural network will adaptively adjust the learning rate each time, and train a total of 500 Epochs. Save the trained neural network parameters, the number of Epochs being trained, the optimizer ADAM, and the scheduler to obtain the trained network model.

7. The wavelet-based low-light image enhancement method according to claim 6, characterized in that: The step 6 is specifically implemented according to the following steps: After each round of training, the network model of that round of training is tested and the value of the structural similarity SSIM is saved. The value range of SSIM is between 0 and 1. The closer it is to 1, the more complete the detailed information of the image.

Images