Low-light image fusion method based on lightweight feature extraction and color restoration

Abstract

This invention discloses a low-light image fusion method based on lightweight feature extraction and color restoration. A lightweight feature extraction (LFE) module extracts local and global features of the image in a lightweight manner, preserving image features to the maximum extent. A fusion module (FB) is used for image fusion and reconstruction to obtain a grayscale fused image that accurately describes the detailed features of the input image. A color restoration network (CR) is used to colorize the grayscale fused image, resulting in a color fused image that better conforms to human visual perception. Finally, a weighted adaptive loss function is applied, adaptively adjusting the weights of the loss function to constrain the network to achieve the optimal fusion result.

Description

Technical Field

[0001] This invention belongs to the field of computer digital image processing technology, specifically relating to a low-light image fusion method based on lightweight feature extraction and color restoration. Background Technology

[0002] When the light detected by a camera is insufficient, the brightness of the captured image is significantly reduced, resulting in a low-light image. Low-light images are often accompanied by problems such as insufficient brightness and blurred details. When the brightness and detail of an image fail to reach the required level, image enhancement processing is necessary. In such cases, various enhancement methods exist, among which deep learning-based fusion methods have attracted considerable attention. These methods not only fully consider the complementary nature of infrared and visible light images but also generate fused images with color, better conforming to human visual perception. Deep learning-based image enhancement algorithms have become an important research field, demonstrating indispensable roles in various areas such as security monitoring, military reconnaissance, target detection, and autonomous driving.

[0003] Currently, existing image fusion methods can be broadly categorized into four main types: autoencoder-based fusion methods, generative adversarial network (GAN)-based fusion methods, traditional convolutional neural network (CNN)-based fusion methods, and Transformer-based fusion methods. However, despite the existence of various methods, some challenges remain in processing infrared and visible light image fusion in low-light scenes.

[0004] Current image fusion algorithms struggle to produce clear infrared and visible light fused images in low-light scenes, likely due to their difficulty in processing low-light images. Furthermore, some methods may introduce significant computational overhead during image fusion, making them less lightweight and limiting their practical application. Additionally, some fusion algorithms generate grayscale images, which may reduce the human eye's perception of the fused image. Summary of the Invention

[0005] The purpose of this invention is to provide a low-light image fusion method based on lightweight feature extraction and color restoration, which solves the problem that the human eye has low perception of the fused image because the image result generated by the fusion algorithm is grayscale.

[0006] The technical solution adopted in this invention is a low-light image fusion method based on lightweight feature extraction and color restoration. A network for fusion image construction is established, including a lightweight feature extraction module, a feature fusion module, and a color restoration network. The lightweight feature extraction module extracts local and global features of the image in a lightweight manner. The fusion module is used to perform image fusion and reconstruction, obtaining a grayscale fusion image of the input image with detailed features. The color restoration network colors the grayscale fusion image to obtain a color fusion image that conforms to human visual perception. Finally, a weighted adaptive loss function is added, adaptively adjusting the weights of the loss function to obtain the optimal fusion result.

[0007] The invention is further characterized by:

[0008] The low-light image fusion method based on lightweight feature extraction and color restoration is implemented according to the following steps:

[0009] Step 1: Construct a network for fusing images, including a lightweight feature extraction module, a feature fusion module, and a color restoration network;

[0010] Step 2: Acquire infrared images (I) under the same low-light scene. Inf and visible light image I Vis A lightweight feature extraction module is used to extract features from infrared and visible light images, obtaining local features F. CNN and global features F T ;

[0011] Step 2: Use the feature fusion module to combine the local features F of the image. CNN and global features F T When combined, they yield the fusion feature Z. feats According to the fusion feature Z feats Reconstruct the grayscale fusion result that accurately describes the detailed features of the input image. F ;

[0012] Step 3: Process the grayscale fused image I using a color restoration network. F Coloring is performed to obtain a color-fused image I that is perceived by human vision. FColor ;

[0013] Step 4: Design a weighted adaptive loss function based on color consistency loss and structural similarity loss. Adaptively adjust the weights of the loss function. Multiple losses complement each other by sharing information, and finally obtain the network with the optimal fused image.

[0014] Step 5: Train the network for the optimal fused image for 70 epochs to obtain the trained optimal fused image network;

[0015] Step 6: Input the low-light image to be enhanced into the trained optimal fusion image network to obtain a low-light enhanced image with color.

[0016] The lightweight feature extraction module in step 1 consists of a multi-head attention mechanism (MSA) block and a multilayer perceptron (MLP) block; the multi-head attention mechanism (MSA) block is shown in formulas (1), (2), and (3):

[0017] MultiHead(Q,K,V)=Concat(head1,…,head h W O (1);

[0018] head i =Attention(QW i Q ,KW i K VW i V (2);

[0019]

[0020] in, d model The sequence encoding length is represented by Q, K, and V, which represent the query, key, and value in matrix form, respectively. k Indicates the dimension of the vector key;

[0021] The feature fusion module consists of two sequentially connected convolutional blocks (Convblock).

[0022] In step 2, based on the fusion feature Z feats Reconstruct the grayscale fusion result that accurately describes the detailed features of the input image. F The specific process is as follows:

[0023] The design includes a feature reconstruction module with five convolutional layers. All five convolutional layers have a kernel size of 3×3 and use ReLU activation. The first convolutional layer F1 has 128 input and 128 output channels; the second convolutional layer F2 has 128 input and 64 output channels; the third convolutional layer F3 has 64 input and 32 output channels; the fourth convolutional layer F4 has 32 input and 16 output channels; and the fifth convolutional layer F5 has 16 input and 1 output channel. The fused result Z... feats The grayscale fusion result I is obtained by passing the data through five convolutional layers in sequence. F .

[0024] Grayscale fusion result I F The number of channels is 1, i.e., grayscale image, and the width and height of the image are the same. The fusion result is expressed as the following formula (4):

[0025] I F =(C,W,H) (4);

[0026] Where C represents the number of channels in the image, with a channel count of 1, i.e., a grayscale image; W represents the width of the image; and H represents the height of the image, with the width and height being the same.

[0027] Step 4 is as follows:

[0028] Visible light image I Vis Convert from RGB color space to YCbCr color space and extract visible light image I Vis The Cb and Cr channels contain color information; by fusing the grayscale results, I F With visible light image I Vis The Cb and Cr channels are stitched together to obtain a 3-channel image I. FCbCr As input to the color restoration network, the image is converted back to the RGB color space to obtain a color-rich fused image I. FColor .

[0029] The color restoration network structure is as follows (5):

[0030] I FColor =CR(I FCbCr (5).

[0031] Step 5 is as follows:

[0032] Step 4.1: Designing a weight-adaptive loss function consists of two parts, namely L... SSIM L Color The loss function was optimized to achieve unsupervised learning. SSIM It is an effective measure of structural similarity between infrared and visible light images, combining both structure and contrast components to comprehensively measure image quality. Color Let k represent the color consistency loss function; select a weighted adaptive loss function, where k represents the number of loss functions, W is the weight of each loss function, and b is the bias of each loss function. The weighted adaptive loss function is expressed as follows (6):

[0033]

[0034] Step 4.2: Calculate the visible light image I Vis and 3-channel image I FCbCr Average pixel value E(I)Vis |W) and E(I FCbCr |W), if E(I) Vis |W)>E(I FCbCr |W), then calculate the visible light image I Vis Fusion SSIM M Loss function value; otherwise, calculate the loss function value for the 3-channel image I. FCbCr Fusion SSIM M The loss function calculates the result as a score (Score(I)). Vis ,I FCbCr ,I FColor The value of |W) is then used to calculate an effective measure of structural similarity L between infrared and visible light images. SSIM ;

[0035] Step 4.3: Based on the input grayscale fusion image I Inf 3-channel image I FCbCr Color fusion result I FColor Calculate the difference L between the input image and the output fused image. Color The calculation formula is as follows:

[0036]

[0037] Where C represents the number of channels in the image, and ∠(·,·) represents the fused image I on the R, G, and B channels. FColor Compared with the original visible image I FCbCr The discrete cosine similarity between the two is calculated pixel by pixel, where K represents the number of pixels in the image and i represents an element in R, G, and B;

[0038] Step 4.4, Loss SUM Dynamically learn the loss weights, and L SSIM L Color By sharing information through adaptive weights, complementing each other, and updating network parameters, the network gradually converges, resulting in a network with the optimal fused image.

[0039] Step 4.2 is used for the 3-channel image I FCbCr and visible light image I Vis Fusion SSIM M The formula for the loss function is as follows:

[0040]

[0041] Where σ represents the standard deviation, and X represents the input I. Inf and I FCbCr Y represents the fused color image I FColorσXY represents the cross-correlation between X and Y, W represents the sliding window, which moves element by element from left to right and top to bottom, and C is a constant; SSIM is measured by calculating the average value of pixels in the sliding window using E(I|W). M The fraction E(I|W) is given by the following formula:

[0042]

[0043] Where I represents the input I. Vis and I FCbCr m and n both represent the image size, P i It is the value of pixel i; SSIM M The formula for calculating fractions is as follows:

[0044]

[0045] L SSIM The calculation formula is as follows:

[0046]

[0047] Where N represents the total number of sliding windows in a single image.

[0048] Step 6 is as follows:

[0049] Step 5.1: Select a GPU with 32GB of video memory for training. The optimizer for the network with the best fused images is the ADAM optimizer. Set the initial learning rate to 0.01 and the scheduler strategy to a multi-step decay strategy with decay steps of 25 and 100 respectively. Decrease by half each time and train for a total of 70 epochs.

[0050] Step 5.2: Save the network parameters of the optimal fused image trained in Step 5.1, the number of epochs being trained, the optimizer ADAM, and the scheduler to obtain the trained network model.

[0051] The beneficial effects of this invention are:

[0052] This invention presents a low-light image fusion method based on lightweight feature extraction and color restoration. A lightweight feature extraction (LFE) module extracts local and global features of the image in a lightweight manner, preserving image features to the maximum extent. A fusion module (FB) is used for image fusion and reconstruction to obtain a grayscale fused image that accurately describes the detailed features of the input image. A color restoration network (CR) is used to colorize the grayscale fused image, resulting in a color fused image that better conforms to human visual perception. Finally, a weighted adaptive loss function is applied, adaptively adjusting the weights of the loss function to constrain the network to achieve the optimal fusion result. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the overall structure of the network model used in this invention;

[0054] Figure 2 This is a schematic diagram of the structure of a portion of the convolutional PConv in the lightweight feature extraction module (LFE) of this invention;

[0055] Figure 3 This is a schematic diagram of the lightweight Transformer layer in the lightweight feature extraction module (LFE) of this invention;

[0056] Figure 4 This is a schematic diagram of the structure of the color restoration network (CR) of the present invention;

[0057] Figure 5 This is a schematic diagram illustrating the changes in the evaluation metric CC during the training process of this invention;

[0058] Figure 6 This is a schematic diagram illustrating the effect of the low-light image fusion method based on lightweight feature extraction and color restoration of the present invention in a low-light scene. Detailed Implementation

[0059] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0060] like Figure 1 As shown, this invention is a low-light image fusion method based on lightweight feature extraction and color restoration. It uses a lightweight feature extraction module (LFE) to extract local and global features of the image in a lightweight manner, which can preserve image features to the maximum extent. It uses a fusion module (FB) to perform image fusion and reconstruction to obtain a grayscale fused image that can accurately describe the detailed features of the input image. It uses a color restoration network (CR) to colorize the grayscale fused image to obtain a color fused image that is more in line with human visual perception. Finally, it uses a weighted adaptive loss function to adaptively adjust the weights of the loss function to constrain the network to achieve the optimal fusion result.

[0061] This invention is a low-light image fusion method based on lightweight feature extraction and color restoration, specifically implemented according to the following steps:

[0062] Step 1: Acquire infrared images (I) under the same low-light scene. Inf and visible light image I Vis A lightweight feature extraction module is used to extract features from infrared and visible light images, obtaining local features F. CNN and global features F T ;

[0063] Partial convolution (PConv) is a convolutional neural network that applies regular Convolution to extract spatial features only on a portion of the input channels, while keeping the remaining channels unchanged. Figure 2 As shown. For consecutive or regular memory accesses, the first or last consecutive channel is treated as a representative of the entire feature map for computation.

[0064] The Lightweight Transformer layer utilizes skip attention to achieve a lightweight Transformer. Specifically, it uses the self-attention computation of preceding layers to approximate the attention of multiple subsequent layers, thus achieving lightweight Transformer feature extraction. Figure 3 As shown. The lightweight Transformer consists of a multi-head attention mechanism (MSA) block and a multilayer perceptron (MLP) block; the multi-head attention mechanism (MSA) block is shown in equations (1), (2), and (3):

[0065] MultiHead(Q,K,V)=Concat(head1,…,head h W O (1);

[0066] head i =Attention(QW i Q ,KW i K VW i V (2);

[0067]

[0068] in, d model The sequence encoding length is represented by Q, K, and V, which represent the query, key, and value in matrix form, respectively. k Indicates the dimension of the vector key.

[0069] The main steps of MSA are as follows: First, multiple Q, K, V matrices are learned individually, which corresponds to multiple weight matrices W for learning Q, K, V. i Q W i K and W i V Input matrix X and corresponding weight matrix W i Multiply these values ​​to obtain the corresponding Q, K, and V. Then, calculate the self-attention separately for each attention head and obtain the corresponding output Z. iFinally, all Z i Concatenate them into a single unit, and then multiply by an output matrix W. O .

[0070] Step 2: Use the feature fusion module to combine the local features F of the image. CNN and global features F T The fused feature Z is obtained by fusing two sequentially connected convolutional blocks (Convblocks). feats According to the fusion feature Z feats Reconstruct the grayscale fusion result that accurately describes the detailed features of the input image. F The specific process is as follows:

[0071] The design includes a feature reconstruction module with five convolutional layers. All five convolutional layers have a kernel size of 3×3 and use ReLU activation. The first convolutional layer F1 has 128 input and 128 output channels; the second convolutional layer F2 has 128 input and 64 output channels; the third convolutional layer F3 has 64 input and 32 output channels; the fourth convolutional layer F4 has 32 input and 16 output channels; and the fifth convolutional layer F5 has 16 input and 1 output channel. The fused result Z... feats The grayscale fusion result I is obtained by passing the data through five convolutional layers in sequence. F .

[0072] Grayscale fusion result I F The number of channels is 1, i.e., grayscale image, and the width and height of the image are the same. The fusion result is expressed as the following formula (4):

[0073] I F =(C,W,H) (4);

[0074] Where C represents the number of channels in the image, with a channel count of 1, i.e., a grayscale image; W represents the width of the image; and H represents the height of the image, with the width and height being the same.

[0075] Step 3: Process the grayscale fused image I using a color restoration network. F Coloring is performed to obtain a color-fused image I that is perceived by human vision. FColor The specific process is as follows:

[0076] Visible light image I Vis Convert from RGB color space to YCbCr color space and extract visible light image I Vis The Cb and Cr channels contain color information; by fusing the grayscale results, I F With visible light image IVis The Cb and Cr channels are stitched together to obtain a 3-channel image I. FCbCr As input to the color restoration network, the image is converted back to the RGB color space to obtain a color-rich fused image I. FColor .like Figure 4 As shown. Due to the low-light environment and poor lighting conditions, the visible light image I... Vis Many foreground target objects are not visible in the image, but the fusion result obtained by this invention is... FColor It effectively overcomes this difficulty, making objects clearly visible. Furthermore, compared to infrared imaging... Inf In comparison, the fusion result I obtained by this invention FColor It has richer texture and color information, making the image more visually appealing.

[0077] The Color Restoration Network (CR) is a network architecture based on UNet, also known as an encoder-decoder architecture. Because the overall structure of the network resembles a large letter U, it is called UNet. Figure 4 As shown. This structure allows for the visualization of 3-channel images I... FCbCr Become I FColor It has richer texture and color information, making the image more visually appealing. The color restoration network structure is as follows (5):

[0078] I FColor =CR(I FCbCr (5).

[0079] Step 4: Design a weighted adaptive loss function based on color consistency loss and structural similarity loss. The weights of the loss function are adaptively adjusted, and multiple losses complement each other by sharing information, ultimately yielding a network with the optimal fused image. The specific process is as follows:

[0080] Step 4.1: Designing a weight-adaptive loss function consists of two parts, namely L... SSIM L Color The loss function was optimized to achieve unsupervised learning. SSIM It is an effective measure of structural similarity between infrared and visible light images, combining both structure and contrast components to comprehensively measure image quality. Color Let k represent the color consistency loss function; select a weighted adaptive loss function, where k represents the number of loss functions, W is the weight of each loss function, and b is the bias of each loss function. The weighted adaptive loss function is expressed as follows (6):

[0081]

[0082] Step 4.2: Calculate the visible light image IVis and 3-channel image I FCbCr Average pixel value E(I) Vis |W) and E(I FCbCr |W), if E(I) Vis |W)>E(I FCbCr |W), then calculate the visible light image I Vis Fusion SSIM M Loss function value; otherwise, calculate the loss function value for the 3-channel image I. FCbCr Fusion SSIM M The loss function calculates the result as a score (Score(I)). Vis ,I FCbCr ,I FColor The value of |W) is then used to calculate an effective measure of structural similarity L between infrared and visible light images. SSIM ;

[0083] For 3-channel image I FCbCr and visible light image I Vis Fusion SSIM M The formula for the loss function is as follows:

[0084]

[0085] Where σ represents the standard deviation, and X represents the input I. Inf and I FCbCr Y represents the fused color image I FColor σXY represents the cross-correlation between X and Y, W represents the sliding window, which moves element by element from left to right and top to bottom, and C is a constant; SSIM is measured by calculating the average value of pixels in the sliding window using E(I|W). M The fraction E(I|W) is given by the following formula:

[0086]

[0087] Where I represents the input I. Vis and I FCbCr m and n both represent the image size, P i It is the value of pixel i; SSIM M The formula for calculating fractions is as follows:

[0088]

[0089] L SSIM The calculation formula is as follows:

[0090]

[0091] Where N represents the total number of sliding windows in a single image.

[0092] Step 4.3: Based on the input grayscale fusion image I Inf 3-channel image I FCbCr Color fusion result I FColor Calculate the difference L between the input image and the output fused image. Color The calculation formula is as follows:

[0093]

[0094] Where C represents the number of channels in the image, and ∠(·,·) represents the fused image I on the R, G, and B channels. FColor Compared with the original visible image I FCbCr The discrete cosine similarity between the two images is calculated pixel-by-pixel, where K represents the number of pixels in the image and i represents an element in R, G, and B. The color consistency loss is designed for two purposes. First, it effectively reduces color distortion caused by image fusion. Second, it reduces color distortion caused by visible light image I... Vis Information from the R, G, and B channels is converged into a 3-channel image I. FCbCr In the process, the final color-blended image I is obtained. FColor It can capture more scene details.

[0095] Step 4.4, Loss SUM Dynamically learn the loss weights, and L SSIM L Color By sharing information through adaptive weights, complementing each other, and updating network parameters, the network gradually converges, resulting in a network with the optimal fused image.

[0096] Step 5: Train the network for the optimal fused image for 70 epochs to obtain the trained optimal fused image network; the specific process is as follows:

[0097] Step 5.1: Select a GPU with 32GB of video memory for training. The optimizer for the network with the best fused images is the ADAM optimizer. Set the initial learning rate to 0.01 and the scheduler strategy to a multi-step decay strategy with decay steps of 25 and 100 respectively. Decrease by half each time and train for a total of 70 epochs.

[0098] Step 5.2: Save the network parameters of the optimal fused image trained in Step 5.1, the current training epoch, the optimizer ADAM, and the scheduler to obtain the trained network model. After each training epoch, save the CC value and plot the CC curve for 70 epochs as shown below. Figure 5 As shown, the CC index fluctuated significantly before the 20th round, but was generally on an upward trend. After the 20th round, it stabilized and eventually reached 0.923.

[0099] Step 6: Input the low-light image to be enhanced into the trained optimal fusion image network to obtain a low-light enhanced image with color.

[0100] Example

[0101] like Figure 6 The image shown is an illustration of the effect of using the method of this invention in a low-light scene. Figure 6 The subscripts (a)-(f) in the middle represent the effect diagrams in this scene. Figure 6 (a) is the input infrared image. Figure 6 (b) is a visible light image. Figure 6 Image (c) is a grayscale image of the visible light image. Feature extraction is performed on both the visible light and infrared images to obtain deep-level feature information. Then, feature fusion and reconstruction are performed to obtain the final image. Figure 6 In step (e), color restoration is finally performed to obtain the color image fusion result. Figure 6 (f) Among them, Figure 6 (d) shows the color image fusion result of the CDDFuse method.

[0102] This invention presents a low-light image fusion method based on lightweight feature extraction and color restoration. A lightweight feature extraction (LFE) module extracts local and global features of the image in a lightweight manner, preserving image features to the maximum extent. A fusion module (FB) is used for image fusion and reconstruction to obtain a grayscale fused image that accurately describes the detailed features of the input image. A color restoration network (CR) is then used to colorize the grayscale fused image, resulting in a color fused image that better conforms to human visual perception. Finally, a weighted adaptive loss function is applied, adaptively adjusting the weights of the loss function to constrain the network to achieve the optimal fusion result. This method obtains a color fused image from infrared and visible light images in low-light environments, and the color fused image possesses richer detail information, demonstrating significant practical value.

Claims

1. A low-light image fusion method based on light-weight feature extraction and color restoration, characterized in that, The network for constructing a fused image comprises a lightweight feature extraction module, a feature fusion module and a color restoration network; the lightweight feature extraction module extracts local features and global features of an image in a lightweight manner, the feature fusion module is used for image fusion and reconstruction to obtain a gray fused image of the input image in terms of detail features, the color restoration network colors the gray fused image to obtain a color fused image conforming to human visual perception; a weight adaptive loss function is further added to adaptively adjust the weight of the loss function to obtain an optimal fusion result; the following steps are implemented in detail: Step 1, constructing a network for fused images, comprising a lightweight feature extraction module, a feature fusion module and a color restoration network; Step 2, collect infrared images under the same weak light scene and visible light images , extract features of the infrared images and the visible light images through a light feature extraction module to obtain local features and global features ; Step 2: Use the feature fusion module to integrate local features of the image. and global features When combined, they exhibit fusion characteristics. Based on fusion characteristics Reconstruct the grayscale fusion result that accurately describes the detailed features of the input image. ; Step 3, color restoration network on the grayscale fused image colorization to obtain a color fused image perceived by human visual perception ; Step 4: Design a weighted adaptive loss function based on color consistency loss and structural similarity loss. Adaptively adjust the weights of the loss function; multiple losses complement each other by sharing information, ultimately yielding a network with the optimal fused image. This involves processing visible light images... Convert from RGB color space to YCbCr color space and extract visible light image. The Cb and Cr channels contain color information; by blending grayscale results With visible light images The Cb and Cr channels are stitched together to obtain a 3-channel image. As input to the color restoration network, the image is converted back to the RGB color space to obtain a richly colored fused image. ; Step 4, the specific process is as follows: Step 4.1, the loss function with adaptive weight includes two parts, which are , , the loss function is optimized to realize unsupervised learning, is an effective measure of structural similarity between infrared images and visible light images, which combines structural and contrast components to measure image quality, represents the color consistency loss function; the loss function with adaptive weight is selected, k represents the number of loss functions, W is the weight of each loss function, b is the bias of each loss function, and the loss function with adaptive weight is represented by the following formula (6): (6)； Step 4.2, computing the visible light image and the average pixel value of the 3-channel image and if then compute the visible light image fused SSIM M loss function value, otherwise, compute the visible light image fused SSIM M loss function, the result of the computation as a score ; then compute an effective measure of structural similarity between the infrared image and the visible light image ;​ Step 4.3, fusing the input grayscale image according to the input gray scale fusion image , 3-channel image and color fusion result , calculating the difference between the input image and the output fused image , the calculation formula is as follows: (11)； wherein, C represents the number of channels in the image, represents a pixel-wise calculation of the discrete cosine similarity between the fused image R , G , B on the channels, and the original visible image , K represents the number of pixels in the image, i represents R , G , B an element in the matrix Step 4.4、 Dynamically learn the loss weight, and , Share information through adaptive weight, complement each other, update network parameters, so that the network gradually converges to get the optimal fusion image network; Step 5, the optimal fused image network is trained for 70 epochs to obtain a trained optimal fused image network; Step 6, inputting a weak light image to be enhanced into the trained optimal fused image network to obtain a weak light enhanced image with color.

2. The weak light image fusion method based on light feature extraction and color restoration according to claim 1, characterized in that, The lightweight feature extraction module in step 1 is composed of a multi-head attention mechanism MSA block and a multi-layer perception MLP block; the multi-head attention mechanism MSA block is as shown in formulas (1), (2) and (3): (1)； (2)； (3)； wherein , , , , denotes the sequence coding length, Q , K , V denote the matrix form of query , key and value , denotes the vector key dimension; The feature fusion module comprises two sequentially connected convolution blocks Convblock.

3. The weak light image fusion method based on light feature extraction and color restoration according to claim 2, characterized in that, The fusion feature described in step 2 The gray-scale fusion result which can accurately describe the detail features of the input image is reconstructed The specific process is: The feature reconstruction module includes five convolutional layers, the kernel sizes of the five convolutional layers are all , the ReLU activation function is used as the activation function, the input channel number of the first convolutional layer F1 is 128, the output channel number of the convolutional layer is 128, the input channel number of the second convolutional layer F2 is 128, the output channel number of the convolutional layer is 64, the input channel number of the third convolutional layer F3 is 64, the output channel number of the convolutional layer is 32, the input channel number of the fourth convolutional layer F4 is 32, the output channel number of the convolutional layer is 16, the input channel number of the fifth convolutional layer F5 is 16, and the output channel number of the convolutional layer is 1; the fusion result passes through the five convolutional layers in sequence to obtain the gray-scale fusion result .

4. The weak light image fusion method based on light feature extraction and color restoration according to claim 3, characterized in that, The gray-scale fusion result The number of channels is 1, i.e. a gray-scale image, and the width and height of the image are the same. The fusion result is expressed as formula (4) as follows: (4)； wherein, C represents the number of channels of the image, the number of channels being 1, i.e. a grayscale image, W represents the width of the image, H is the height of the image, the width and the height of the image being the same.

5. The weak light image fusion method based on light feature extraction and color restoration according to claim 4, characterized in that, The color restoration network structure is as shown in formula (5): (5)。 6. The weak light image fusion method based on light feature extraction and color restoration according to claim 5, characterized in that, Step 4.2 is used for 3-channel images and visible light images Fusion SSIM M The formula for the loss function is as follows: (7)； where, denotes the standard deviation, X denotes the input and , Y denotes the fused color image , denotes X and Y the cross-correlation between W denotes the sliding window, moving element-wise from left to right and from top to bottom, C is a constant; the average value of the pixels in the sliding window is computed using to measure the score of SSIM M , is given by the formula: (8)； wherein, I represents an input and , m , n both represent an image size, is a value of a pixel i ; SSIM M The calculation formula of the score is as follows: (9)； The calculation formula is as follows: (10)； wherein N denotes the total number of sliding windows in a single image.

7. The weak light image fusion method based on light feature extraction and color restoration according to claim 6, characterized in that, Step 5, the specific process is as follows: Step 5.1, training on a GPU with 32GB of memory, the optimizer of the optimal fused image network is an ADAM optimizer, the initial learning rate is set to 0.01, the scheduler strategy is a multi-step decay strategy, the decay steps are 25 and 100 respectively, the learning rate is halved each time, and a total of 70 epochs are trained; Step 5.2, saving the optimal fused image network parameters, the number of Epochs being trained, the optimizer ADAM and the scheduler of step 5.1 to obtain a trained network model.

Images