A multi-exposure image fusion method based on lightweight attention mechanism

Abstract

The present invention discloses a multi-exposure image fusion method based on a lightweight attention mechanism, comprising the following steps: step S1, inputting images of different exposures into a feature extraction module to obtain feature information corresponding to different source images; step S2, inputting feature information corresponding to different source images into a feature fusion module to obtain a preliminary fusion result; step S3, inputting the preliminary fusion result into a detail correction module to obtain a final fused image. The technical solution of the present invention can improve the efficiency and quality of image fusion while reducing the amount of parameters and calculations.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a multi-exposure image fusion method based on a lightweight attention mechanism. Background Art

[0002] With the continuous development of technology, mobile phone photography has become the most common way of shooting in people's daily lives. However, when people take photos in daily life, they are affected by the length of exposure time, which leads to the phenomenon of halo, gradient inversion, color distortion, etc. This is because people take multiple photos in a row in the same scene, which causes the captured images to be affected by light from different angles, resulting in underexposure or overexposure of the captured images.

[0003] Multi-exposure image fusion technology can effectively utilize the brightness information in multiple exposure images, restore the information affected by light in the original exposure image, enhance the transparency of the information in the image, and thus generate a high dynamic range image of the same scene. Traditional technologies mainly use methods based on convolutional neural networks (CNN) and Transformer to achieve multi-exposure image fusion. These methods only focus on the final fusion effect, without considering the amount of calculation and the number of parameters. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a multi-exposure image fusion method based on a lightweight attention mechanism, which improves the efficiency and quality of image fusion while reducing the amount of parameters and calculations.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] A multi-exposure image fusion method comprises the following steps:

[0007] Step S1, inputting images with different exposure levels into a feature extraction module to obtain feature information corresponding to different source images;

[0008] Step S2, inputting feature information corresponding to different source images into a feature fusion module to obtain a preliminary fusion result;

[0009] Step S3: input the preliminary fusion result into the detail correction module to obtain the final fused image.

[0010] Preferably, the feature extraction module consists of depthwise separable convolution, multi-scale feature extraction, pooling attention and axial attention mechanism; the shallow feature information of the image is obtained through depthwise separable convolution, the multi-scale feature extraction block obtains the global information and detail information of different scales in the image, and then the axial attention mechanism is used to extract the global structural information of the image, and finally the pooling attention is used to obtain the spatial information of the image from multiple sets of feature information.

[0011] Preferably, the feature fusion module consists of three groups of pooling attention, three groups of axial attention mechanisms and three groups of brightness fusion blocks; the pooling attention is used to perform lightweight processing on the feature information and obtain information of different dimensions, and the axial attention mechanism is used to perform multi-dimensional feature fusion on the information of different dimensions; the average brightness information of different source images is fused and added with their brightness features through the brightness fusion block to obtain the brightness information of the preliminary fused image.

[0012] Preferably, the detail correction module consists of multiple 3×3 convolution blocks and two reconstruction blocks. Relu activation function and normalization are performed after each convolution. The image after preliminary fusion is used as input, and the edge information of objects in the image is sharpened in each convolution block.

[0013] The present invention uses two attention mechanisms, Axis Attention and PoolFormer, to obtain image information, which reduces the number of parameters and calculations while improving the efficiency and quality of image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0015] Figure 1 This is a flow chart of a multi-exposure image fusion method according to an embodiment of the present invention;

[0016] Figure 2 This is the architecture diagram of the Multi-Scale Feature Extraction Block (MFEB);

[0017] Figure 3 This is the architecture diagram of the Axis Attention mechanism;

[0018] Figure 4 This is the PoolFormer architecture diagram;

[0019] Figure 5 Correct the architecture diagram for details;

[0020] Figure 6 Schematic diagram of the original image (underexposed on the upper left, overexposed on the lower left) and the fused image (right). DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Embodiment 1:

[0024] like Figure 1 As shown, an embodiment of the present invention provides a multi-exposure image fusion method, comprising the following steps:

[0025] Step S1, inputting images with different exposure levels into a feature extraction module to obtain feature information corresponding to different source images;

[0026] Step S2, inputting feature information corresponding to different source images into a feature fusion module to obtain a preliminary fusion result;

[0027] Step S3: input the preliminary fusion result into the detail correction module to obtain the final fused image.

[0028] As an implementation method of an embodiment of the present invention, the feature extraction module is composed of a deep separable convolution (DWSConv), a multi-scale feature extraction block (MFEB), a pooling attention (PoolFormer) and an axial attention mechanism (Axis Attention). The shallow feature information of the image is obtained through the deep separable convolution (DWSConv), the multi-scale feature extraction block (MFEB) obtains the global information and detail information of different scales in the image, and then the axial attention mechanism (Axis Attention) is used to extract the global structural information of the image, and finally the pooling attention (PoolFormer) is used to obtain the spatial information of the image from multiple sets of feature information. Figure 2 As shown in the figure, the multi-scale feature extraction block (MFEB) mainly consists of multiple groups of convolution kernels of different sizes of 1×1, 3×3, and 5×5 and Relu activation functions to extract multi-scale features from shallow feature maps. Figure 3 The Axis Attention mechanism shown in the figure is mainly composed of two specific axis attentions and a bidirectional gated feedback block. In the feature extraction module, the main function of the Axis Attention mechanism is to capture the contextual information of two different axes by independently calculating the attention along each axis, thereby extracting rich global structural information features. Figure 4The pooling attention (PoolFormer) shown in the figure is mainly composed of global average pooling and multi-layer perceptron (MLP). In the feature extraction module, the spatial feature information is obtained by aggregating the information in the local area using the pooling layer. The main function of this feature extraction module is to extract different aspects of image information while lightweighting the network and reducing information loss.

[0029] As an implementation method of an embodiment of the present invention, the feature fusion module is composed of three groups of pooling attention (PoolFormer), three groups of axial attention mechanisms (Axis Attention) and three groups of brightness fusion blocks (LightFusion). First, the corresponding feature information of different source images is spliced, and then the feature information is lightweight processed by using pooling attention (PoolFormer) and the spatial dimension information of different sources is captured by global average pooling to obtain the average brightness information of the corresponding position of the source image. Then, the obtained preliminary information is used to enhance the model's perception of image context information and reduce the complexity of model calculation by using two specific axial attention mechanisms of the axial attention mechanism (Axis Attention), promote the fusion effect of different source images and improve the adaptability of the model. Finally, the average brightness information of different source images is fused and added with their brightness features to obtain the brightness information of the preliminary fused image.

[0030] As an implementation method of the embodiment of the present invention, Figure 5 As shown in the figure, the detail correction module (DCM) consists of multiple 3×3 convolution blocks and two reconstruction blocks, where each convolution is followed by a Relu activation function and normalization. The detail correction module takes the image after preliminary fusion as input, sharpens the edge information of objects in the image in each convolution block, and improves the brightness of the overall color of the image, thereby eliminating artifacts in the fused image and further enhancing the fusion effect.

[0031] The present invention reduces the number of model parameters and computational complexity by introducing the Axis Attention mechanism and PoolFormer. The Axis Attention mechanism mainly applies attention along a specific axis. This method reduces redundant information and obtains the global feature information of the image through multi-scale feature extraction, which can extract richer structural information. The feature information extracted by the axial attention is spliced ​​together and then the spatial information of the image is obtained through PoolFormer. In the traditional self-attention mechanism, the amount of calculation increases quadratically with the increase of the sequence length. PoolFormer reduces the overall computational amount of the model by introducing global average pooling to replace the standard multi-head self-attention (MSA) in the self-attention mechanism. Finally, the image after preliminary fusion is subjected to color correction and detail enhancement to ensure the quality of the fusion result image.

[0032] Embodiment 2:

[0033] An embodiment of the present invention provides a multi-exposure image fusion method based on a lightweight network model, comprising the following steps:

[0034] 1. Two pairs of images with different exposures are used as the input of the network. The input image size is H×W×3. The input images are I u and I o , and then respectively for I u and I o The data preprocessing operation makes the image size 256×256×3.

[0035] 2. Convert I and O from the original RGB channels to YCbCr channels according to formula 1, where the Y channel represents the brightness channel and the CbCr channel represents the contrast and chrominance channels. Since most of the reasons for exposure are related to brightness, the feature information of the Y channel is input into the network, and the original CbCr channel information is retained. The information input to the network architecture is Y o , Y u .

[0036] Y=0.299R+0.587G+0.114B

[0037] Cb=-0.1687R-0.3313G+0.5B (1)

[0038] Cr=0.5R-0.4187G-0.0813B

[0039] 3. Y o and Y uThey are input into the corresponding feature extraction modules respectively. First, the image is preprocessed using deep separable convolution (DWSConv) to obtain the shallow feature information T of the image. u and T O , the size is n×256×256, n is the number of channels. Then, the multi-scale feature extraction block (MFEB) is used to perform 1×1, 3×3, and 5×5 convolutions on T to obtain three sets of feature information of different scales, namely a1, b1, and c1. Then, 1×1, 3×3, and 5×5 convolutions are performed on b1 to obtain a2, b2, and c2. Then, a1 and a2, c1 and c2 are concatenated according to the channel dimension and a Relu activation function is performed to obtain i1 and j1, the size is 2n×256×256. This operation is repeated three times in sequence. Finally, b4, i3, and j3 are concatenated according to the channel dimension and Relu activated to obtain K, and a 1×1 convolution is performed on K to obtain U1 and O1, the size is (9n+1)×256×256.

[0040] 4. The shallow feature information T u and T O , input into the axial attention mechanism, perform a linear normalization and two specific axis attentions to get A, then add A to T to get X, then perform linear normalization and double gating mechanism and local information enhancement (DGFN) on X, and add it to X to get U2 and O2, with a size of 1×256×256.

[0041] 5. The shallow feature information T, global detail information and structural information are concatenated according to the channel dimension and the concatenated feature information is lightweight processed using pooling attention (PoolFormer) to obtain U3 and O3, with a size of (10n+1)×256×256.

[0042] 6. The obtained U and O are input into the feature fusion module. First, the feature information corresponding to different source images is spliced ​​by dimension to obtain Z1, Z2 and Z3. Then, the pooling attention (PoolFormer) is used to perform lightweight processing on Z and obtain information E of different dimensions. Then, the axial attention mechanism (Axis Attention) is used to perform multi-dimensional feature fusion on E to obtain F1, F2 and F3, with a size of 1×256×256.

[0043] 7. The brightness information Y of different sources o and Y u Splice by dimension and use 1×1 convolution and pooling attention (PoolFormer) to extract the average brightness Y of the corresponding positions of different images and , the size is 1×256×256.

[0044] 8. Compare the obtained F and Y and Input into the brightness feature fusion block (LightFusion), perform two 3×3 convolutions and one 1×1 convolution on F to obtain T and calculate it according to formula 2 to obtain R0, where α0 is 0.7, α1 is 0.1, and α2 is 0.3.

[0045] R0=α0×Y and ×T0+α1×Y and ×T1+α2×Y and ×T2 (2)

[0046] 9. Convert CbCr un and CbCr ov The CbCr channel information of a size of 2×256×256 is obtained by weighted averaging using Formula 3.

[0047] Cr / Cb=(α1×(Cr un / Cb un ))+α2×(Cr ov / Cb ov )) (3)

[0048] 10. The obtained R0 and the processed CbCr channel information are calculated by formula 4 to obtain the preliminary fusion result R1, and the fused result is restored to the image size according to the size W and H of the original image to obtain R2.

[0049] R=1.1643×(Y-16)+1.596×(Cr-128)

[0050] R=1.1643×(Y-16)-0.3917×(Cb-128)-0.8129×(Cr-128) (4)

[0051] R=1.1643×(Y-16)+2.0173×(Cb-128)

[0052] 11. The preliminary fusion result R2 is input into the detail correction module. First, R2 is subjected to two 3×3 convolutions, two normalizations and two downsamplings of the activation function to obtain N1. Then N1 is reconstructed twice to obtain N2. Finally, N2 is subjected to two 3×3 convolutions, two normalizations and two upsamplings of the activation function to obtain the fusion result graph R.

[0053] During the training implementation phase, the network model uses the public datasets SICE and MEFB, with a total of 960 images and 460 sets of exposure image pairs for training samples. For the loss function of the network model, the L1 loss function, the structural loss function, and the image fidelity loss function are selected. The L1 loss function has a good effect on the overall contrast of the image and can prevent the image gradient from exploding. The structural loss function can adjust the overall contrast of the image. The image fidelity loss function can reduce the artifacts in the fused image and improve the overall smoothness of the image to obtain richer detail information in the image. These loss functions will be introduced below.

[0054] The L1 loss function is the basic loss function in the image fusion task. In the peak signal-to-noise ratio (PSNR), the L1 loss is often used to calculate the fused image, which is better than the L2 loss training, because the L1 loss mainly calculates the square of the absolute pixel value of the reference image and the fused image, and the total error is very small. The L2 loss calculates the variance of the absolute pixel value of the reference image and the fused image. In some specific exposure areas, the variance is very large, so the L1 loss is selected in the loss function selection. The formula of the L1 loss function is shown in Formula 4. Where f g is the reference image, f r To fuse the images

[0055] L1=||f g -f r ||1 (4)

[0056] The structural loss function has a good performance in maintaining image structural information. The MEF-SSIM loss function is selected from the aspects of image structure preservation, contrast and detail preservation, and flexibility and adaptability. The MEF-SSIM loss function is a loss based on structural similarity indicators. It retains edge details and texture information from multiple angles such as image brightness, contrast, and structural information to obtain higher quality and more visually obvious results. The formula of the MEF-SSIM loss function is shown in Formula 5 and Formula 6. Where x and y are the reference image and the fused image respectively, μ x and μ y is the average of x and y, and is the variance of x and y, σ xy is the covariance of x and y.

[0057]

[0058] L MEF-SSIM (F,R)=1-SSIM(F,R) (6)

[0059] Image fidelity loss can quantify and optimize the similarity between the fused image and the reference image. In order to enhance the continuity of local information of the image, the optimization of gradient smoothing is added to the original image fidelity loss function. The specific loss function is shown in Formula 7. Where α is 1, β is 1, I f is the reference image, I u For underexposed images, I o is an overexposed image, max(·) is the maximum value operation, and ||·||1 is the first normal form calculation.

[0060] L if =α(U||I f -I u ||1+O||I f -I o ||1)+β(||I f -max(I u ,I o )||1) (7)

[0061] The weighted average of L1 loss, MEF-SSIM structural loss and image fidelity loss function is used as the total loss function of the network model, as shown in Formula 8, where λ1 is 0.2, λ2 is 0.2, and λ3 is 0.6.

[0062] L total =λ1L1+λ2L MEF-SSIM +λ3L if (8)

[0063] When people use the camera built into their mobile phones or digital cameras to take photos, if the object being photographed is under sunlight or light, the image taken will form an overexposure effect, so that a large part of the image is covered by strong light. If the object being photographed is in a backlit or backlit scene, the image taken will be underexposed, making the overall image darker. Both of the above shooting situations cannot obtain complete information in the image. If the image taken is put into the network model of the invention, a high dynamic range image can be obtained, thereby obtaining complete information in the image. In mobile phone photography, due to hardware limitations, it is difficult to capture all the details of a high dynamic range scene in a single shot. The multi-exposure image fusion technology of the embodiment of the present invention can be used to synthesize high dynamic range images, thereby improving the quality of photos. In the field of autonomous driving, the camera of an autonomous driving vehicle needs to accurately perceive the environment under various lighting conditions. The multi-exposure image fusion technology can help generate clearer environmental images, thereby improving the vehicle's recognition and decision-making capabilities. In the field of security, cameras often need to work under extreme lighting conditions, such as backlight or at night. Using multi-exposure image fusion technology can improve the visibility and quality of surveillance images and enhance the effectiveness of the surveillance system.

[0064] The method for fusion of multiple exposure images in an embodiment of the present invention adopts an end-to-end approach, inputs multiple exposure images, and improves the transparency of image information and the integrity of the overall image information through multiple operations such as extraction, fusion, and correction. Figure 6 As shown in the figure, the left side is the input image and the right side is the fused image. It can be seen that the overall brightness of the input image is too high or too dark, resulting in many places in the image being unable to be observed intuitively, but the fused image has obvious improvements in both color vividness and overall image details, and all the information in the image can be clearly seen.

[0065] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A multi-exposure image fusion method based on a lightweight attention mechanism, characterized in that: The following steps are involved: Step S1, inputting images with different exposure levels into a feature extraction module to obtain feature information corresponding to different source images; Step S2, inputting feature information corresponding to different source images into a feature fusion module to obtain a preliminary fusion result; Step S3, inputting the preliminary fusion result into the detail correction module to obtain the final fused image; The feature extraction module consists of depthwise separable convolution, multi-scale feature extraction, pooling attention and axial attention mechanism. The shallow feature information of the image is obtained through depthwise separable convolution, and the multi-scale feature extraction block obtains the global information and detail information of different scales in the image. Then, the axial attention mechanism is used to extract the global structural information of the image. Finally, the pooling attention is used to obtain the spatial information of the image from multiple sets of feature information. The feature fusion module consists of three groups of pooling attention, three groups of axial attention mechanisms and three groups of brightness fusion blocks. Pooling attention is used to perform lightweight processing on feature information and obtain information of different dimensions. The axial attention mechanism is used to perform multi-dimensional feature fusion on information of different dimensions. The brightness fusion block is used to fuse and add the average brightness information of different source images with their brightness features to obtain the brightness information of the preliminary fused image. The detail correction module consists of multiple 3×3 convolution blocks and two reconstruction blocks. After each convolution, a ReLU activation function and normalization are performed. The image after preliminary fusion is used as input, and the edge information of objects in the image is sharpened in each convolution block.

Images