Low-light enhancement method based on Transformer and image fusion

Abstract

The present invention discloses a low-light enhancement method based on Transformer and image fusion, specifically: Step 1, use DoubleGamma transform to preprocess the infrared and visible light images in low-light scenes separately; Step 2, model the local and global correlations in the infrared input image and the visible light input image; Step 3, use Fusionblock to fuse the feature sequence of the infrared image and the feature sequence of the visible light image together, and use Enhanceblock to enhance the features of the fusion result; Step 4, reconstruct the fusion result based on the fusion features; Step 5, update the network parameters based on the loss, so that the network gradually converges to obtain a network that outputs the optimal fused image. The network model established by the method of the present invention can effectively extract the deep features of infrared and visible light images and construct a fused image.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer digital image processing, and in particular relates to a low-light enhancement method based on Transformer and image fusion. Background Art

[0002] In all aspects of our lives, 80% of the information we obtain daily comes from vision, but we have very little knowledge of the information hidden in the dark, which makes us have a strong demand for low-light night vision.

[0003] Deep learning-based image enhancement algorithms are a key research area in low-light image enhancement, playing an irreplaceable role in diverse fields, including civil and police applications, security monitoring, object detection, and biomedicine. Deep learning-based fusion methods fully utilize the complementary information between visible and infrared images. Furthermore, the Transformer is applied to computer vision tasks to reduce model complexity and address the limitations of convolutional neural networks (CNNs) in capturing long-range information.

[0004] Most of the current image fusion methods use encoders and decoders to extract and reconstruct features from infrared and visible light images. However, existing image fusion algorithms can only reconstruct visible light and infrared fusion images under normal lighting conditions, multi-exposure and under-exposure fusion images, and fusion images of multi-focus scenes. It is not easy to obtain clear infrared and visible light fusion images in low-light scenes. Summary of the Invention

[0005] The purpose of the present invention is to provide a low-light enhancement method based on Transformer and image fusion, which solves the problem that it is not easy to obtain clear infrared and visible light fusion images in low-light scenes.

[0006] The technical solution adopted by the present invention is:

[0007] The low-light enhancement method based on Transformer and image fusion is implemented in the following steps:

[0008] Step 1, design the preprocessing DoubleGamma module. The DoubleGamma module consists of Gamma' transformation and Gamma" transformation. Gamma' transformation processes the input infrared image, and Gamma" transformation processes the input visible light image. The infrared image I' is obtained. Inf and visible light image I′ Vis ;

[0009] Step 2: Design the feature extraction module TFE based on the Transformer architecture, namely Transformer-FeatureExtraction, by transforming I′ in step 1 Inf and I′ Vis Modeling the local and global correlations in the image to extract deep features of the input image;

[0010] Step 3, design a feature fusion enhancement module FE composed of Fusionblock and Enhanceblock; use Fusionblock to transform the infrared image feature sequence and the feature sequence of visible light images Fuse them together, where l = 1…L, and then use Enhanceblock to enhance the features of the fusion result;

[0011] Step 4: Design the feature reconstruction module FR, namely Feature Reconstruction, based on the fusion feature Z feats Reconstruction fusion result I F ;

[0012] Step 5, design including L SSIM , L TV and L EXP The loss function is used to update the network parameters according to the loss, so that the network gradually converges and finally obtains a network that outputs the optimal fused image.

[0013] Step 6: Train the neural network built based on the low-light enhancement algorithm of Transformer and image fusion for 100 epochs, verify the training results, and save the neural network model;

[0014] Step 7: Test the neural network model saved in step 6 and output the fused image.

[0015] The present invention is also characterized in that:

[0016] Step 1 is as follows: first, the infrared image I in the low-light scene Inf and visible light image I Vis Convert it into a grayscale image as the input image, and then use DoubleGamma transformation on the converted grayscale image to convert the infrared image I Inf Using the Gamma' transform in DoubleGamma, we get I' Inf , visible light image I Vis Use the Gamma″ transform in DoubleGamma to get I′ Vis , used to highlight the image feature information; the formula for DoubleGamma transformation is as follows:

[0017] I′=I γ (1);

[0018] Where I is the input image, including I Inf and I Vis , I′ is the image enhanced by DoubleGamma transformation, including I′ Inf and I′ Vis γ is the scaling factor for pixel values. The DoubleGamma transform can selectively enhance the contrast of low-grayscale areas or high-grayscale areas based on the value of γ. The scaling factor for pixel values in the Gamma′ transform is 0.8, and the scaling factor for pixel values in the Gamma″ transform is 0.5.

[0019] Step 2 is as follows:

[0020] Step 2.1, transform the infrared image I′ after DoubleGamma transformation Inf and visible light image I′ Vis Enter the embedded linear projection to obtain the infrared feature sequence Z Inf and visible light characteristic sequence Z Vis ;

[0021] Step 2.2, the feature sequence Z Inf and Z Vis Through L Transformer layers in sequence, the output of each layer is expressed as and Where l = 1…L; the structure of the Transformer layer consists of a multi-head attention mechanism MSA block and a multi-layer perceptron MLP block; MSA is shown in formula (2):

[0022] MultiHead(Q,K,V)=Concat(head1,…,head h )W O (2);

[0023] head i =Attention(QW i Q ,KW i K ,VW i V ) (3);

[0024]

[0025] in, d modelIndicates the sequence encoding length, Q, K, V represent the query, key and value in matrix form respectively, d k Indicates the dimension of the vector key.

[0026] Step 3 is as follows:

[0027] Step 3.1: Fusionblock module feature fusion is as follows: and Perform three-layer wavelet packet decomposition to obtain the decomposition coefficient C and the length S of each layer, C1 and C2 are and The decomposition coefficients of each layer, S1 and S2 are and The length of each layer decomposition coefficient;

[0028] Extract and The approximate components A1 and A2 of the wavelet decomposition are added and then averaged to obtain A;

[0029] extract and The detail components of each layer of wavelet decomposition H1, V1, D1 and H2, V2, D2;

[0030] The regional energy-based fusion rule is used for the detail components H1 and H2, V1 and V2, D1 and D2 of each layer to obtain H, V, and D. The specific process of the regional energy-based fusion rule is as follows: first, the matching degree of the two images is calculated. If the matching degree is greater than the threshold, the weighted average fusion method is used. Otherwise, if the matching degree is less than the threshold, the wavelet coefficient with larger local energy is selected as the wavelet coefficient of the fused image.

[0031] Use A, H, V, and D to perform wavelet reconstruction to obtain fusion features;

[0032] Step 3.2: Enhanceblock feeds the fused features into two sequentially connected convolution blocks Convblock to obtain Z feats .

[0033] Step 4 is as follows:

[0034] Step 4.1: The fusion result Z feats After passing through 5 convolution layers in sequence, the fusion result I is obtained. F , where the number of channels of the fusion result is 1, i.e. grayscale image, the width and height of the image are the same, and the fusion result is expressed as the following formula (5):

[0035] I F =(C,W,H) (5);

[0036] Among them, C represents the number of channels of the image. The number of channels is 1, that is, a grayscale image. W represents the width of the image. H represents the height of the image. The width and height of the image are the same.

[0037] Step 5 is as follows:

[0038] The total loss function is expressed as the following formula (6):

[0039] Loss=λL SSIM +L TV +L EXP (6);

[0040] Set a hyperparameter to influence their weights;

[0041] Step 5.1: First calculate the average pixel value E(I′) of the visible light image and infrared image after DoubleGamma preprocessing Vis |W) and E(I′ Inf |W), if E(I′ Vis |W)>E(I′ Inf |W), then calculate SSIM M (I′ Vis ,I F |W) value, otherwise, calculate SSIM M (I′ Inf ,I F |W), and the calculation result is used as Score(I′ Vis ,I′ Inf ,I F |W) value, and then calculate L SSIM ;

[0042] Among them, SSIM is used for infrared and visible light image fusion M The formula of the loss function is as follows:

[0043]

[0044] Where σ represents the standard deviation, and X represents I′ after DoubleGamma transformation Inf and I′ Vis , Y represents the fused image I F , σXY represents the cross-correlation between X and Y, W represents the sliding window, which moves element by element from left to right and top to bottom, and C is a constant; SSIM is measured by calculating the average value of the pixels in the sliding window using E(I|W) M The formula for the fraction E(I|W) is as follows:

[0045]

[0046] Among them, P i is the value of pixel i; SSIM M The score is calculated as follows:

[0047]

[0048] L SSIM The calculation formula is as follows:

[0049]

[0050] Where N represents the total number of sliding windows in a single image;

[0051] Step 5.2, calculate L TV , input visible light image I′ Vis And fusion result I F , first calculate the difference R(i,j) between the visible light image and the fused image after DoubleGamma preprocessing, and then calculate L TV ;

[0052] L TV The calculation formula is as follows:

[0053] R(i,j)=I′ Vis (i,j)-I F (i,j) (11);

[0054]

[0055] Where R(i,j) represents the difference between the visible light and fused images, L TV represents the total change loss function, ||||2 represents the l2 distance;

[0056] Step 5.3, calculate L EXP , design exposure control loss function L EXP To control the exposure level; the exposure control loss measures the distance between the average intensity value of the local area and the good exposure level E; E is set to 0.6; L EXP It can be expressed as:

[0057]

[0058] Where M represents the number of non-overlapping local regions of size 16×16, I k is the average intensity value of the local area in the enhanced image;

[0059] Step 5.4, L SSIM , L TV and L EXPThe Loss is obtained by adding them together, and the network parameters are updated according to the Loss, so that the network gradually converges.

[0060] Step 6 is as follows:

[0061] Step 6.1, select the network optimizer as ADAM optimizer, set the initial learning rate to 0.01, and the scheduler strategy to multi-step decay strategy. The decay steps are 25 and 100 respectively, and the decay is half each time. A total of 100 Epochs are trained. By observing L SSIM , L TV and L EXP Three indicators dynamically adjust hyperparameters: learning rate lr and number of training rounds Epoch;

[0062] Step 6.2, use the SummaryWriter of the Python third-party library tensorboard to write the training process L SSIM , L TV and L EXP Output the reference indicators to tensorboard and test the results on the validation set. Then save the model and save the neural network parameters trained in step 6.1, the number of epochs being trained, the optimizer ADAM, and the scheduler to obtain the trained network model.

[0063] Step 7 is as follows:

[0064] Load the network model trained in step 6, input the test set into the trained network model, and then save the test results to obtain a fused image.

[0065] The beneficial effect of the present invention is that the present invention is based on the low-light enhancement method of Transformer and image fusion, uses double gamma transform for image enhancement, extracts deep information of the image through Transformer, uses TFE to enhance the infrared and visible light features, and uses FR for image reconstruction, which can accurately describe the detailed features of the input infrared and visible light to achieve low-light image enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 This is a schematic diagram of the overall structure of the low-light enhancement method based on Transformer and image fusion of the present invention;

[0067] Figure 2 Schematic diagram of the structure of the feature extraction module TFE in the low-light enhancement method based on Transformer and image fusion of the present invention;

[0068] Figure 3Schematic diagram of the structure of the Transformer layer in the feature extraction module TFE of the low-light enhancement method based on Transformer and image fusion of the present invention;

[0069] Figure 4 Schematic diagram of the structure of the Convblock of the feature fusion enhancement module FE in the low-light enhancement method based on Transformer and image fusion of the present invention;

[0070] Figure 5 Schematic diagram of the structure of the feature reconstruction module FR in the low-light enhancement method based on Transformer and image fusion of the present invention;

[0071] Figure 6 This is the low-light enhancement method based on Transformer and image fusion in the present invention. SSIM Calculation flow chart of

[0072] Figure 7 This is a schematic diagram of the changes in the evaluation index SSIM during the training process of the low-light enhancement method based on Transformer and image fusion of the present invention;

[0073] Figure 8 This is a schematic diagram of the effect of the low-light enhancement method based on Transformer and image fusion in low-light scenes. DETAILED DESCRIPTION

[0074] The following describes in detail the low-light enhancement method based on Transformer and image fusion of the present invention with reference to the accompanying drawings and specific embodiments.

[0075] like Figure 1 As shown, the embodiments of the present invention include: a low-light enhancement method based on Transformer and image fusion, using double gamma transform for image enhancement, extracting deep information of the image through Transformer, and using FR for image reconstruction, which can accurately describe the detailed features of the input infrared and visible light to achieve low-light image enhancement.

[0076] The low-light enhancement method based on Transformer and image fusion is implemented in the following steps:

[0077] Step 1: First, the infrared image I in the low-light scene Inf and visible light image I Vis Convert it into a grayscale image as the input image, and then use DoubleGamma transformation on the converted grayscale image to convert the infrared image I Inf Using the Gamma' transform in DoubleGamma, we get I'Inf , visible light image I Vis Use the Gamma″ transform in DoubleGamma to get I′ Vis , used to highlight image feature information.

[0078] The formulas for Gamma′ and Gamma″ transformations are:

[0079] I′=I γ (1);

[0080] Where I is the input image, including I Inf and I Vis , I′ is the image enhanced by DoubleGamma transformation, including I′ Inf and I′ Vis , where γ is the scaling factor of the pixel values. The DoubleGamma transform can selectively enhance the contrast of low-grayscale areas or high-grayscale areas based on the value of γ. γ is a very important parameter in image grayscale correction. Its value determines the grayscale mapping method between the input image and the output image, that is, whether to enhance low-grayscale areas, i.e., shadow areas, or high-grayscale areas, i.e., highlights. The scaling factor of the pixel values for the Gamma′ transform is 0.8, and the scaling factor of the pixel values for the Gamma″ transform is 0.5.

[0081] Step 2: Design TFE. TFE is a feature extraction module based on Transformer architecture. It processes the infrared image I′ obtained after step 1. Inf and visible light image I′ Vis The local and global correlations in the image are modeled to extract the deep features of the input image. Step 2 is mainly used to extract deep features, which can be divided into two steps:

[0082] Step 2.1, transform the infrared image I′ after DoubleGamma transformation Inf and visible light image I′ Vis Enter the embedded linear projection to obtain the infrared feature sequence Z Inf and visible light characteristic sequence Z Vis ;

[0083] Step 2.2, the feature sequence Z Inf and Z Vis Through L Transformer layers in sequence, the output of each layer is expressed as and Where l=1…L. Figure 3As shown in the figure, the structure of the Transformer layer consists of an MSA block and an MLP block. The specific steps of the MSA module are: first, use layer normalization, i.e. LN, to process the feature sequence, then send the processed features to the MSA block, and perform a residual connection between the feature sequence that has not been subjected to the LN operation and the output of the MSA, and then send the result of the residual connection to the MLP module. Similar to the MSA module, the MLP module first uses LN to process the feature sequence, and then sends the processed features to the MLP block, and then performs a residual connection between the feature sequence that has not been subjected to the LN operation and the output of the MLP, and finally obtains the output result of the Transformer layer. and

[0084] The formula of MSA is shown in (2):

[0085] MultiHead(Q,K,V)=Concat(head1,…,head h )W O (2);

[0086] head i =Attention(QW i Q ,KW i K ,VW i V ) (3);

[0087]

[0088] in, d model Indicates the sequence encoding length, Q, K, V represent the query, key and value in matrix form respectively, d k Indicates the dimension of the vector key;

[0089] The main steps of MSA are divided into the following three steps: First, learn multiple Q, K, V matrices separately, which corresponds to multiple learning Q, K, V weight matrices W i Q 、W i K and W i V , input matrix X and corresponding weight matrix W i Multiply them to get the corresponding Q, K, V, then calculate the self-attention for each attention head separately and get the corresponding output Z i , finally, all Z i Spliced into a whole, and then multiplied by an output matrix W O ;

[0090] Step 3, design FE, which consists of two parts: Fusionblock and Enhanceblock. First, use Fusionblock to transform the infrared image feature sequence into and the feature sequence of visible light images Fuse them together, and then use the Enhanceblock module to enhance the features of the fusion result. The specific steps are as follows:

[0091] Step 3.1, the specific process of Fusionblock module feature fusion is as follows:

[0092] (1) The result obtained in step 2 and Perform three-layer wavelet packet decomposition to obtain the decomposition coefficient C and the length S of each layer, C1 and C2 are and The decomposition coefficients of each layer, S1 and S2 are and The length of each layer decomposition coefficient;

[0093] (2) Extract and The approximate components A1 and A2 of the wavelet decomposition are added and then averaged to obtain A;

[0094] (3) Extraction and The detail components of each layer of wavelet decomposition H1, V1, D1 and H2, V2, D2;

[0095] (4) The regional energy-based fusion rule is used for the detail components H1 and H2, V1 and V2, D1 and D2 of each layer to obtain H, V, and D. The specific process of the regional energy-based fusion rule is as follows: first, the matching degree of the two input features is calculated. If the matching degree is greater than the threshold, it means that the local energies corresponding to the two input features are close, so the weighted average fusion method is adopted. On the contrary, if the matching degree is less than the threshold, it means that the local energies corresponding to the two input features are quite different, so the wavelet coefficient with larger local energy is selected as the wavelet coefficient for feature fusion;

[0096] (5) Use A, H, V, and D to perform wavelet reconstruction to obtain fusion features.

[0097] Step 3.2, Enhanceblock is composed of two Convblocks, where Convblock is composed of two CNNs and a ReLU activation layer. The size of the convolution kernel in CNN is 3×3 and the padding is 1. Enhanceblock sends the fusion result to two sequentially connected convolution blocks Convblock to obtain Z feats,like Figure 4 As shown in the figure, each Convblock consists of two convolutional layers and one ReLU activation layer. The size of the convolution kernel is 3×3 and the padding is 1.

[0098] Step 4, design FR, based on the fusion feature Z feats Reconstruction fusion result I F .

[0099] Step 4.1, the structure diagram of FR is as follows Figure 4 As shown in the figure: FR consists of 5 convolutional layers. The kernel sizes of the 5 convolutional layers are all 3×3, and the activation function uses the ReLU activation function. The first convolutional layer F1 has 128 input channels and 128 output channels. The second convolutional layer F2 has 128 input channels and 64 output channels. The third convolutional layer F3 has 64 input channels and 32 output channels. The fourth convolutional layer F4 has 32 input channels and 16 output channels. The fifth convolutional layer F5 has 16 input channels and 1 output channel.

[0100] Step 4.2, the fusion result Z feats After passing through 5 convolution layers in sequence, the fusion result I is obtained. F , where the number of channels of the fusion result is 1, that is, the grayscale image, the width and height of the image are the same, and the fusion result is expressed as:

[0101] I F =(C,W,H) (5);

[0102] Where C represents the number of channels of the image. The number of channels is 1, that is, a grayscale image. W represents the width of the image. H represents the height of the image. The width and height of the image are the same.

[0103] Step 5: Design the loss function. The loss function consists of three parts, including L SSIM , L TV and L EXP , to achieve unsupervised learning, L SSIM It is an effective measure of the structural similarity between infrared images and visible light images, combining the two components of structure and contrast to comprehensively measure image quality. TV Denotes the total change loss function, L EXP Represents the exposure control loss function. The total loss function is expressed as:

[0104] Loss=λL SSIM +L TV +L EXP (6);

[0105] Since the three losses are not of the same order of magnitude, a hyperparameter λ is set to affect their weights;

[0106] Step 5 is as follows:

[0107] Step 5.1, calculate L SSIM ,like Figure 6 As shown, first calculate the average pixel value E(I′) of the visible light image and infrared image after DoubleGamma preprocessing. Vis |W) and E(I′ Inf |W), if E(I′ Vis |W)>E(I′ Inf |W), then calculate SSIM M (I′ Vis ,I F |W) value, otherwise, calculate SSIM M (I′ Inf ,I F |W), and the calculation result is used as Score(I′ Vis ,I′ Inf ,I F |W) value, and then calculate L SSIM ;

[0108] Among them, SSIM is used for infrared and visible light image fusion M The formula of the loss function is as follows:

[0109]

[0110] Where σ represents the standard deviation, and X represents I′ after DoubleGamma transformation Inf and I′ Vis , Y represents the fused image I F , σXY represents the cross-correlation between X and Y, W represents the sliding window, the window size is 11×11, and it moves element by element from left to right and from top to bottom. C is a constant, C=9×10 -4 SSIM is measured by calculating the average value of pixels in the sliding window using E(I|W). M The formula for the fraction E(I|W) is as follows:

[0111]

[0112] Among them, P i is the value of pixel i. M The score is calculated as follows:

[0113]

[0114] L SSIMThe calculation formula is as follows:

[0115]

[0116] Where N represents the total number of sliding windows in a single image;

[0117] Step 5.2, calculate L TV , input visible light image I′ Vis And fusion result I F , first calculate the difference R(i,j) between the visible light image and the fused image after DoubleGamma preprocessing, and then calculate L TV ;

[0118] L TV The calculation formula is as follows:

[0119] R(i,j)=I′ Vis (i,j)-I F (i,j) (11);

[0120]

[0121] Where R(i,j) represents the difference between the visible light and fused images, L TV represents the total change loss function, ||||2 represents the l2 distance;

[0122] Step 5.3, calculate L EXP , in order to suppress underexposed or overexposed areas, the exposure control loss function L is designed EXP To control the exposure level. The exposure control loss measures the distance between the average intensity value of the local area and the good exposure level E. Set E to 0.6. L EXP It can be expressed as:

[0123]

[0124] Where M represents the number of non-overlapping local regions of size 16×16, I k is the average intensity value of the local area in the enhanced image;

[0125] Step 5.4, L SSIM , L TV and L EXP The Loss is obtained by adding them together, and the network parameters are updated according to the Loss, so that the network gradually converges.

[0126] Step 6: Train the neural network built based on the low-light enhancement algorithm of Transformer and image fusion for 100 epochs, verify the training results, and save the neural network model;

[0127] Step 6 is as follows: Step 6.1, select the network optimizer as ADAM optimizer, set the initial learning rate to 0.01, and the scheduler strategy to multi-step decay strategy. The decay steps are 25 and 100 respectively, and the decay is half each time. A total of 100 Epochs are trained. By observing L SSIM , L TV and L EXP Three indicators dynamically adjust hyperparameters: learning rate lr and number of training rounds Epoch;

[0128] Step 6.2, use the SummaryWriter of the Python third-party library tensorboard to write the training process L SSIM , L TV and L EXP Output the reference indicators to tensorboard and test the results on the validation set. Then save the model and save the neural network parameters trained in step 6.1, the number of epochs being trained, the optimizer ADAM and the scheduler to obtain the trained network model. After each round of training, save the SSIM value and draw the SSIM curve for 100 rounds as shown below. Figure 7 As shown in the figure, the SSIM index fluctuated greatly before the 20th round, but was generally on an upward trend. After the 20th round, it tended to stabilize and eventually reached 0.93.

[0129] Step 7: Test the neural network model saved in step 6 and output the fused image.

[0130] Step 7 is specifically as follows: loading the network model trained in step 6, inputting the test set into the trained network model, and then saving the test results to obtain a fused image.

[0131] like Figure 8 The figure shows the effect diagram of the present invention in a low-light scene. Each column in the figure represents an image in the same environment. The first row of images is the input visible light image, and the second row of images is the infrared image corresponding to the first row of images. Feature extraction is performed on the visible light image and the infrared image respectively to obtain deep feature information. Then, feature fusion and enhancement operations are performed, and finally feature reconstruction is performed to obtain the visible light and infrared image fusion results in the third row.

[0132] The low-light enhancement method based on Transformer and image fusion is proposed in this paper. It extracts deep features from visible light images and infrared images using a Transformer-based structure, reconstructs the features into images using FR, and accelerates the convergence of the network through the loss function. A fused image is obtained from infrared and visible light images in a low-light environment. The fused image has richer detail information and has certain practical significance.

Claims

1. A low-light enhancement method based on Transformer and image fusion, characterized by: The specific implementation steps are as follows: Step 1, design the preprocessing DoubleGamma module. The DoubleGamma module consists of Gamma' transformation and Gamma" transformation. Gamma' transformation processes the input infrared image, and Gamma" transformation processes the input visible light image. The infrared image I' is obtained. Inf and visible light image I′ Vis ; Step 2: Design TFE based on Transformer architecture by Inf and I′ Vis Modeling the local and global correlations in the image to extract deep features of the input image; Step 3, design the FE consisting of Fusionblock and Enhanceblock; Use Fusionblock to sequence infrared image features and the feature sequence of visible light images Fuse them together, where l = 1…L, and then use Enhanceblock to enhance the features of the fusion result; Step 4, design FR, based on the fusion feature Z feats Reconstruction fusion result I F ; Step 5, design including L SSIM , L TV and L EXP The loss function is used to update the network parameters according to the loss, so that the network gradually converges and finally obtains a network that outputs the optimal fused image; Step 6: Train the neural network built based on the low-light enhancement algorithm of Transformer and image fusion for 100 epochs, verify the training results, and save the neural network model; Step 7: Test the neural network model saved in step 6 and output the fused image.

2. The low-light enhancement method based on Transformer and image fusion according to claim 1, characterized in that: Step 1 is as follows: first, the infrared image I in the low-light scene Inf and visible light image I Vis Convert it into a grayscale image as the input image, and then use DoubleGamma transformation on the converted grayscale image to convert the infrared image I Inf Using the Gamma' transform in DoubleGamma, we get I' Inf , visible light image I Vis Use the Gamma″ transform in DoubleGamma to get I′ Vis ; The formulas for Gamma′ and Gamma″ are as follows: I′=I γ (1); Where I is the input image, including I Inf and I Vis , I′ is the image enhanced by Gamma′ and Gamma″ transformation, including I′ Inf and I′ Vis , γ is the scaling strength of the pixel value; DoubleGamma transformation can selectively enhance the contrast of low grayscale areas or high grayscale areas according to different values of γ; the scaling coefficient of the pixel value of Gamma′ transformation is 0.8, and the scaling coefficient of the pixel value of Gamma″ transformation is 0.

5.

3. The low-light enhancement method based on Transformer and image fusion according to claim 2, characterized in that: The step 2 is specifically as follows: Step 2.1, transform the infrared image I′ after DoubleGamma transformation Inf and visible light image I′ Vis Enter the embedded linear projection to obtain the infrared feature sequence Z Inf and visible light characteristic sequence Z Vis ; Step 2.2, the feature sequence Z Inf and Z Vis Through L Transformer layers in sequence, the output of each layer is expressed as and The structure of the Transformer layer consists of a multi-head attention mechanism MSA block and a multi-layer perceptron MLP block; MSA is shown in formula (2): MultiHead(Q,K,V)=Concat(head1,…,head h )W O (2); in, d model Indicates the sequence encoding length, Q, K, V represent the query, key and value in matrix form respectively, d k Indicates the dimension of the vector key.

4. The low-light enhancement method based on Transformer and image fusion according to claim 3, characterized in that: The step 3 is specifically as follows: Step 3.1: Fusionblock module feature fusion is as follows: and Perform three-layer wavelet packet decomposition to obtain the decomposition coefficient C and the length S of each layer, C1 and C2 are and The decomposition coefficients of each layer, S1 and S2 are and The length of each layer decomposition coefficient; Extract and The approximate components A1 and A2 of the wavelet decomposition are added and then averaged to obtain A; extract and The detail components of each layer of wavelet decomposition H1, V1, D1 and H2, V2, D2; The regional energy-based fusion rule is used for the detail components H1 and H2, V1 and V2, D1 and D2 of each layer to obtain H, V, and D. The specific process of the regional energy-based fusion rule is as follows: first, the matching degree of the two images is calculated. If the matching degree is greater than the threshold, the weighted average fusion method is used. Otherwise, if the matching degree is less than the threshold, the wavelet coefficient with larger local energy is selected as the wavelet coefficient of the fused image. Use A, H, V, and D to perform wavelet reconstruction to obtain fusion features; Step 3.2: Enhanceblock feeds the fused features into two sequentially connected convolution blocks Convblock to obtain Z feats .

5. The low-light enhancement method based on Transformer and image fusion according to claim 4, characterized in that: Step 4 is as follows: Step 4.1: The fusion result Z feats After passing through five convolutional layers in sequence, the fusion result I is obtained. F , where the number of channels of the fusion result is 1, i.e. grayscale image, the width and height of the image are the same, and the fusion result is expressed as the following formula (5): I F =(C,W,H) (5); Among them, C represents the number of channels of the image. The number of channels is 1, that is, a grayscale image. W represents the width of the image. H is the height of the image. The width and height of the image are the same.

6. The low-light enhancement method based on Transformer and image fusion according to claim 5, characterized in that: The step 5 is specifically as follows: The total loss function is expressed as the following formula (6): Loss=λL SSIM +L TV +L EXP (6); Set a hyperparameter to influence their weights; Step 5.1: First calculate the average pixel value E(I′) of the visible light image and infrared image after DoubleGamma preprocessing Vis |W) and E(I′ Inf |W), if E(I′ Vis |W)>E(I′ Inf |W), then calculate SSIM M (I′ Vis ,I F |W) value, otherwise, calculate SSIM M (I′ Inf ,I F |W), and the calculation result is used as Score(I′ Vis ,I′ Inf ,I F |W) value, and then calculate L SSIM ; Among them, SSIM is used for infrared and visible light image fusion M The formula of the loss function is as follows: Where σ represents the standard deviation, and X represents I′ after DoubleGamma transformation Inf and I′ Vis , Y represents the fused image I F , σXY represents the cross-correlation between X and Y, W represents the sliding window, which moves element by element from left to right and from top to bottom, C is a constant, and SSIM is measured by calculating the average value of the pixels in the sliding window using E(I|W) M The formula for the fraction E(I|W) is as follows: Among them, P i is the value of pixel i; SSIM M The score is calculated as follows: L SSIM The calculation formula is as follows: Where N represents the total number of sliding windows in a single image; Step 5.2, calculate L TV , input visible light image I′ Vis And fusion result I F , first calculate the difference R(i,j) between the visible light image and the fused image after DoubleGamma preprocessing, and then calculate L TV ; L TV The calculation formula is as follows: R(i,j)=I′ Vis (i,j)-I F (i,j) (11); Where R(i,j) represents the difference between the visible light and fused images, L TV represents the total change loss function, ||||2 represents the l2 distance; Step 5.3, calculate L EXP , design exposure control loss function L EXP To control the exposure level; the exposure control loss measures the distance between the average intensity value of the local area and the good exposure level E; E is set to 0.6; L EXP It can be expressed as: Where M represents the number of non-overlapping local regions of size 16×16, I k is the average intensity value of the local area in the enhanced image; Step 5.4, L SSIM , L TV and L EXP The Loss is obtained by adding them together, and the network parameters are updated according to the Loss, so that the network gradually converges.

7. The low-light enhancement method based on Transformer and image fusion according to claim 6, characterized in that: The step 6 is specifically as follows: Step 6.1, select the network optimizer as ADAM optimizer, set the initial learning rate to 0.01, and the scheduler strategy to multi-step decay strategy. The decay steps are 25 and 100 respectively, and the decay is half each time. A total of 100 Epochs are trained. By observing L SSIM , L TV and L EXP Three indicators dynamically adjust hyperparameters: learning rate lr and number of training rounds Epoch; Step 6.2, use the SummaryWriter of the Python third-party library tensorboard to write the training process L SSIM , L TV and L EXP Output the reference indicators to tensorboard and test the results on the validation set. Then save the model and save the neural network parameters trained in step 6.1, the number of epochs being trained, the optimizer ADAM, and the scheduler to obtain the trained network model.

8. The low-light enhancement method based on Transformer and image fusion according to claim 7, characterized in that: The step 7 is specifically as follows: Load the network model trained in step 6, input the test set into the trained network model, and then save the test results to obtain a fused image.

Images