A multi-exposure image fusion method based on multi-dimensional collaborative refinement network

Abstract

The present invention discloses a multi-exposure image fusion method based on a multi-dimensional collaborative refinement network, including: constructing a collaborative extraction module CEM to obtain shallow features and deep features of an image; obtaining an attention feature map through a self-attention module SAM; correcting the extracted edge texture of image features using an edge correction module ERM; and using a combination of two loss functions to train an image fusion network, wherein the image fusion network includes a collaborative extraction module CEM, a self-attention module SAM, and an edge correction module ERM. In order to improve the quality of the fused image and avoid artifacts, the present application designs a collaborative extraction module to extract and optimize image features. With the cooperation of the self-attention module, the texture, brightness, and color of the multi-exposure fused image are more prominent; the edge correction module complements CEM and SAM, and it can reduce the impact of noise on features at different scales, enhance the texture details of the feature map, and ensure the edge effect of the fusion result.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a multi-exposure image fusion method based on a multi-dimensional collaborative refinement network. Background Art

[0002] In everyday shooting scenes, it is a tricky matter to modify the exposure time and aperture size to obtain a photo with a good exposure effect. Such photos are called low dynamic range (LDR) images, and they usually have poor visual expression and incomplete display of core content. Specifically, LDR images often have uneven exposure ranges, especially in overexposed and underexposed areas. As a result, observers cannot clearly see vivid details and colors.

[0003] Researchers have tried to develop a variety of hardware devices to obtain HDR images. However, due to the high cost of these devices, they cannot be widely used in daily life. Therefore, Multi-exposure Image Fusion (MEF) has become a mainstream method for generating HDR images as a cost-effective solution. The goal of MEF is to fuse LDR image sequences of different exposures, extract the features of these sequences, and fuse them into vivid HDR images.

[0004] In the past few decades, a large number of MEF methods have been proposed and well applied. These methods can be roughly divided into methods based on traditional frameworks and methods based on deep learning frameworks. In traditional fusion methods, feature transformation in spatial domain and transform domain is a common fusion strategy. However, these methods usually rely on manually designed complex architectures to complete the fusion process, which reduces time efficiency and increases development costs. In order to overcome the shortcomings of traditional methods, researchers use convolutional neural networks (CNNs) in deep learning methods to implement MEF tasks. CNN-based methods have good nonlinear fitting and feature extraction capabilities. However, this method still has some limitations, such as (1) simple fusion rules such as addition, multiplication and weighted average are usually used, so the fusion results are prone to distortion in some areas. (2) In the process of feature extraction, the extracted feature information may be forgotten, resulting in incomplete texture details of the fused image. (3) Local details are rarely paid attention to, and the fusion result may produce some edge artifacts. Summary of the invention

[0005] The purpose of the present invention is to provide an end-to-end multi-exposure image fusion method based on multi-dimensional collaborative learning, which designs three different modules to collaboratively extract features and guide the entire fusion process. The fused image shows vivid texture details and attractive visual perception.

[0006] To achieve the above objectives, the present application proposes a multi-exposure image fusion method based on a multi-dimensional collaborative refinement network, comprising:

[0007] Construct a collaborative extraction module CEM to obtain shallow and deep features of the image;

[0008] Get the attention feature map through the self-attention module SAM;

[0009] The edge correction module ERM is used to correct the edge texture of the extracted image features;

[0010] Two loss functions are combined to train an image fusion network, which includes a collaborative extraction module CEM, a self-attention module SAM and an edge correction module ERM.

[0011] Furthermore, the process of extracting image features by the collaborative extraction module CEM is quantitatively expressed by the following formula:

[0012]

[0013] where i represents the overexposed or underexposed source image, represents the feature map after the 3×3 convolutional layer; P(·) represents the multi-dimensional pooling feature extractor, and C(·) represents the content generated by the coordinated attention block CA.

[0014] Furthermore, in the collaborative extraction module, shallow features of the input image are first extracted through two 3×3 and one 1×1 convolutional layers; three coordinated attention blocks CA are added to the feature extractor and connected, and feature learning is guided by the coordinated attention blocks CA; Maxpooling and Avgpooling operations are simultaneously performed to compress the deep feature map, and then the different feature sequences generated by the two pooling layers are connected in series, and the details of each feature are amplified. After repeating the same pooling operation, the extracted deep features are upsampled through 3×3 and 1×1 convolutional layers; finally, the shallow features are combined with the deep features through element addition and connection operations.

[0015] Furthermore, the self-attention module SAM obtains the original attention feature F through the first two 3×3 convolutional layers. o After that, a 1×1 convolutional layer is used to obtain the corresponding feature space f in the first branch and the second branch respectively. a and f b ; For the feature space fa and f b Perform a flattening operation and transform the feature space f a Perform a transpose operation to obtain the feature weight map W a and W b ; Attention map A1 is obtained by introducing matrix multiplication and softmax function normalization operation, the formula is:

[0016]

[0017] Where k represents the total number of pixels in the feature weight map; through a similar method as above, the attention map A2 is obtained on the third branch:

[0018] A2=softmax(A1×f c )

[0019] Among them, f c is the feature space on the third branch.

[0020] Furthermore, for the original attention feature F o An upsampling operation is performed, and a matrix multiplication operation is performed on the attention map A1 and the attention map A2 to obtain the final attention feature map, which is quantized as:

[0021]

[0022] Furthermore, a BN layer is added after each 1×1 convolution layer and 3×3 convolution layer, and LeakyReLU is used as the activation function.

[0023] Furthermore, the edge correction module ERM uses a spatial gradient filter F G The image edge texture details are optimized as follows:

[0024]

[0025] Will Put it into two 3×3 convolutional layers to complete the edge refinement; where M and N represent the ranges of [1, m] and [1, n] respectively; i is the horizontal pixel of the feature map, and j is the vertical pixel of the feature map; The edge gradient map E of size m×n is expressed as:

[0026]

[0027] Among them, p l(t) 、p b(t) They are p t Pixels to the left and below.

[0028] Furthermore, two loss functions are combined to train the image fusion network. Specifically, the total loss function is designed first, as follows:

[0029] L total =αL SSIM +βL MSE

[0030] Among them, L MSE represents the MSE loss function, α and β are the parameters for adjusting L total The two hyperparameters of the value, L SSIM The formula for obtaining is:

[0031]

[0032] Among them, I O and I U Represents overexposed and underexposed images, I F represents the fusion result;

[0033] The definition of SSIM is as follows:

[0034]

[0035] Among them, μ and σ are the mean and standard deviation; parameters C1, C2 and C3 can keep the loss function numerically stable;

[0036] The MSE loss function is used to obtain the mean square error between the source image and the fused image result, which is defined as:

[0037]

[0038] in, C F , H F and W F are the channel, height and width of the fused image respectively; p represents the number of pixels in an image.

[0039] Furthermore, the specific process of the image fusion network generating the RGB fused image is as follows: converting the overexposed and underexposed source images from RGB three channels to YcbCr three channels; the decomposed Y channel represents the brightness channel of the source image, which is used to fuse the real information; the Cb and Cr channels represent the chromaticity information of the input image. The following formula can give more weight to the low-saturation color component and less weight to the high-saturation color component, which is defined as:

[0040]

[0041] Among them, C f It is the weighted sum of Cb and Cr; Cb i and Cri are the chromaticity values ​​of different channels in the source image respectively; τ is used as a hyperparameter; finally, the fused YCbCr result is converted to RGB result through inverse transformation.

[0042] Compared with the prior art, the above technical solution adopted by the present invention has the following advantages: In order to improve the quality of the fused image and avoid artifacts, the present application designs a collaborative extraction module (CEM) to extract and optimize image features. With the cooperation of the self-attention module (SAM), the texture, brightness and color of the multi-exposure fused image are more prominent; the edge revision module (ERM) complements CEM and SAM, which can reduce the impact of noise on features at different scales, enhance the texture details of feature mapping, and ensure the edge effect of the fusion result. Through three different modules to collaboratively extract features and guide the entire fusion process, the fused image has small distortion and good effect, showing vivid texture details and attractive visual perception. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of the multi-exposure image fusion method of the present invention;

[0044] Figure 2 It is a schematic diagram of the structure of the collaborative extraction module CEM;

[0045] Figure 3 Schematic diagram of the self-attention module SAM structure;

[0046] Figure 4 This is a schematic diagram of the structure of the edge correction module ERM;

[0047] Figure 5 Detailed process diagram for generating RGB fused images for the image fusion network;

[0048] Figure 6 A qualitative comparison diagram between the present invention and other advanced fusion methods on the SICE dataset;

[0049] Figure 7 A quantitative comparison chart between the present invention and other advanced fusion methods on the SICE dataset. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application, that is, the embodiments described are only part of the embodiments of the present application, not all of the embodiments.

[0051] Example 1

[0052] like Figure 1 As shown, the present invention provides a multi-exposure image fusion method based on a multi-dimensional collaborative refinement network, which specifically includes:

[0053] Step 1: Construct a collaborative extraction module CEM to obtain shallow and deep features of the image;

[0054] Specifically, the collaborative extraction module CEM is as follows Figure 2 As shown in the figure. A multi-dimensional pooling feature extractor is constructed, and the feature learning is guided by the Coordinate Attention (CA) block. Under the guidance of the CA block, CEM can obtain the most significant feature map from the source image. LeakyReLU and Sigmoid are used as activation functions in the module. The process of CEM extracting features can be quantified by the following formula:

[0055]

[0056] where i represents the overexposed or underexposed source image, represents the feature map after the 3×3 convolutional layer; P(·) represents the multi-dimensional pooling feature extractor, and C(·) represents the content generated by the CA block.

[0057] It should be noted that in the collaborative extraction module CEM, shallow feature extraction is first performed on the input image through two 3×3 and one 1×1 convolutional layers. The CA block is regarded as a computing unit, and three such blocks are added to the feature extractor at the same time and connected to ensure that the network can highlight the ideal details of the fusion result. In order to accurately extract complex representation graphs from images with different exposures, Maxpooling and Avgpooling are used to compress the deep feature map at the same time. Then, the different feature sequences generated by the two pooling layers are connected in series, and various feature details are amplified. After repeating the same pooling layer operation, the extracted features are upsampled through 3×3 and 1×1 convolutional layers. In order to compensate for the degradation of the extracted features after upsampling, the present invention directly connects the shallow convolution with the convolution layer after upsampling; finally, the shallow features are combined with the deep features through element addition and connection operations.

[0058] Step 2: Obtain the attention feature map through the self-attention module SAM;

[0059] Specifically, in MEF, due to changes in the shooting scene, overexposed and underexposed images cannot show complete details. In this case, a self-attention mechanism is designed for feature extraction and fusion. By drawing global relationships, the fusion results are associated with small blocks at the same position in the source image, thereby effectively solving the problem of limited kernel size in ordinary convolutional layers.

[0060] The present invention redesigns the self-attention module SAM according to the characteristics of MEF, and its structure is as follows: Figure 3 As shown. The original attention feature F is obtained through the first two 3×3 convolutional layers. o After that, 1×1 convolution is used to obtain the corresponding feature space f in the first branch and the second branch respectively. a and f b In order to facilitate calculation, the feature space f a and f b A flattening operation is performed and the feature space f a Perform a transpose operation to obtain the feature weight map W a and W b . Matrix multiplication and softmax function normalization operations are introduced to obtain the attention map A1, and its formula is:

[0061]

[0062] Where k represents the total number of pixels in the feature weight map. Similarly, A2 can be obtained on the third branch by the same method as above. Finally, the original attention feature F o Perform upsampling operation and do matrix multiplication on A1 and A2 to get the final attention feature map, which can be quantized as:

[0063]

[0064] In addition, in order to improve the computational efficiency of SAM, a maximum pooling layer is introduced. A BN layer is added after all convolutional layers to prevent gradient explosion or disappearance during training, and LeakyReLU is used as the activation function.

[0065] Step 3: Use the edge correction module ERM to correct the extracted image feature edge texture;

[0066] Specifically, before outputting the fused image, the extracted features are put into the edge correction module ERM to correct the edge texture to ensure the quality of the fusion result. Figure 4 As shown in FIG. 1 , after connecting the feature maps of CEM and SAM, the present invention uses ERM to refine the edge details of these feature maps. Specifically, the spatial gradient filter F GOptimize edge texture details as follows:

[0067]

[0068] Where M and N represent the range of [1, m] and [1, n] respectively. i is the horizontal pixel of the feature map, and j is the vertical pixel of the feature map. The edge gradient map E with a size of m×n can be expressed as:

[0069]

[0070] Among them, p l(t) 、p b(t) Yes t Pixels to the left and below; F G It is very sensitive to edge features in the image and helps the convolution layer refine the edge details of the fusion result. Put it into two 3×3 convolutional layers to complete the edge refinement.

[0071] Step 4: using two loss functions to combine and train an image fusion network, wherein the image fusion network includes a collaborative extraction module CEM, a self-attention module SAM and an edge correction module ERM;

[0072] Specifically, the present invention designs two loss functions to combine the proposed network structure for training. The total loss function is as follows:

[0073] L total =αL SSIM +βL MSE

[0074] Among them, L MSE represents the MSE loss function, α and β are the parameters for adjusting L total The two hyperparameters of L can both be set to 0.5. SSIM The formula for obtaining is:

[0075]

[0076] Among them, I O and I U Represents overexposed and underexposed images, I F represents the fusion result. ω1 and ω2 can be set to 0.5, and the definition of SSIM is as follows:

[0077]

[0078] Where μ and σ are the mean and standard deviation. Parameters C1, C2, and C3 can keep the loss function numerically stable. In particular, the structural similarity between input and output can be measured by SSIM, which is very sensitive to brightness, contrast distortion, and correlation loss.

[0079] The MSE loss function is used to obtain the mean square error between the source image and the fused image result, which is defined as:

[0080]

[0081] in, C F , H F and W F are the channel, height and width of the fused image respectively; p represents the number of pixels in an image.

[0082] The specific process of the image fusion network generating the RGB fusion image is as follows: Figure 5 As shown in FIG. 1 , the key to the fusion of RDB images is to fuse specific information on different channels. The overexposed and underexposed source images are converted from RGB three-channels to YCbCr three-channels. The decomposed Y channel represents the brightness channel of the source image, which can usually reflect the content details and texture of the image. Therefore, in the present invention, the Y channel is used to fuse real information. In other words, performing image fusion on the Y channel is similar to fusing grayscale images on a single channel.

[0083] The Cb and Cr channels represent the chromaticity information of the input image. The chromaticity of the image can be obtained by the traditional weighted sum formula, which can give more weight to low-saturation color components and less weight to high-saturation color components, which is defined as:

[0084]

[0085] Among them, C f It is the weighted sum of Cb and Cr. i and Cr i are the chromaticity values ​​of different channels in the source image respectively. τ is a hyperparameter and can be set to 128. Finally, the fused YCbCr result is converted to RGB result through inverse transformation.

[0086] This paper selects test image sequences on the SICE dataset and compares them with nine state-of-the-art multi-exposure image fusion methods. Figure 6The overall effect and local feature details are shown in the figure. In the first group of image sequences, the sky displayed by DSIFT, DEM, MESPD and MEF-Net has obvious local shadows. In addition, GBM and IFCNN are overexposed during the fusion process. U2Fusion and PMGI still have defects in color performance. Although the overall visual effect of MEF-GAN is of high quality, the details are blurred. In contrast, the method proposed in the present invention can produce vivid fusion results in both details and colors.

[0087] In addition to subjective qualitative analysis, the present invention also uses objective quantitative qualitative changes MEF-SSIM and PSNR to evaluate the performance of the fusion results. At the same time, CC and MI are introduced to reflect the comprehensiveness of quantitative comparison. The present invention extracts 29 pairs of test image sequences from 100 pairs of data sets for calculation. All quantitative results are as follows: Figure 7 As shown. Obviously, the results obtained by the present invention have reached the highest level among the four evaluation indicators. It can be seen from the distribution of the line graph that most of the indicators of the test results are higher than other methods. The higher the value of MEF-SSIM, the better the visual effect that the present invention can provide and the similarity to the source image. The value of PSNR represents that the present invention can fuse precise texture details. In addition, the high level performance of CC and MI represents that the present invention can achieve better linear correlation and more information transfer. Therefore, the fused image generated by the present invention has less distortion and better effect.

[0088] The proposed method realizes multi-exposure image fusion by constructing a collaborative extraction module, a self-attention module, and an edge correction module. Each module makes an outstanding contribution to the performance of the fusion result, showing vivid texture details and attractive visual perception. It is compared with other nine state-of-the-art methods in the SICE dataset; the results show that the proposed method outperforms the other methods both qualitatively and quantitatively. Therefore, the present invention promotes the development of multi-exposure image fusion.

[0089] The foregoing description of specific exemplary embodiments of the present invention is for the purpose of illustration and demonstration. These descriptions are not intended to limit the present invention to the precise form disclosed, and it is clear that many changes and variations can be made based on the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present invention and its practical application, so that those skilled in the art can realize and utilize various different exemplary embodiments of the present invention and various different selections and changes. The scope of the present invention is intended to be limited by the claims and their equivalents.

Claims

1. A multi-exposure image fusion method based on a multi-dimensional collaborative refinement network, characterized in that: include: Construct a collaborative extraction module CEM to obtain shallow and deep features of the image; Get the attention feature map through the self-attention module SAM; The edge correction module ERM is used to correct the edge texture of the extracted image features; The image fusion network is trained by combining two loss functions, wherein the image fusion network includes a collaborative extraction module CEM, a self-attention module SAM and an edge correction module ERM; The process of extracting image features by the collaborative extraction module CEM is quantitatively expressed by the following formula: where i represents the overexposed or underexposed source image, represents the feature map after the 3×3 convolutional layer; P(·) represents the multi-dimensional pooling feature extractor, and C(·) represents the content generated by the coordinated attention block CA; In the collaborative extraction module, shallow feature extraction is first performed on the input image through two 3×3 and one 1×1 convolutional layers; Three coordinated attention blocks CA are added to the feature extractor and connected together, and feature learning is guided by the coordinated attention blocks CA; at the same time, Maxpooling and Avgpooling operations are used to compress the deep feature map, and then the different feature sequences generated by the two pooling layers are connected in series, and the details of each feature are amplified. After repeating the same pooling operation, the extracted deep features are upsampled through 3×3 and 1×1 convolutional layers; finally, the shallow features are combined with the deep features through element addition and concatenation operations; The self-attention module SAM obtains the original attention feature F through the first two 3×3 convolutional layers. o After that, a 1×1 convolutional layer is used to obtain the corresponding feature space f in the first branch and the second branch respectively. a and f b ; For the feature space f a and f b Perform a flattening operation and transform the feature space f a Perform a transpose operation to obtain the feature weight map W a and W b ; Attention map A1 is obtained by introducing matrix multiplication and softmax function normalization operation, the formula is: Where k represents the total number of pixels in the feature weight map; through a similar method as above, the attention map A2 is obtained on the third branch: <h2 style=";text-align:left;direction:ltr">A2 = softmax(A1×f<h2 style=";text-align:left;direction:ltr"> c <h2 style=";text-align:left;direction:ltr"> ) Among them, f c is the feature space on the third branch; For the original attention feature F o An upsampling operation is performed, and a matrix multiplication operation is performed on the attention map A1 and the attention map A2 to obtain the final attention feature map, which is quantized as: Add a BN layer after each 1×1 convolution layer and 3×3 convolution layer, and use LeakyReLU as the activation function; The edge correction module ERM uses a spatial gradient filter F G The image edge texture details are optimized as follows: Will Put it into two 3×3 convolutional layers to complete the edge refinement; where M and N represent the ranges of [1,m] and [1,n] respectively; i is the horizontal pixel of the feature map, and j is the vertical pixel of the feature map; The edge gradient map E of size m×n is expressed as: Among them, p l(t) 、p b(t) They are p t Pixels to the left and below.

2. According to claim 1, a multi-exposure image fusion method based on a multi-dimensional collaborative refinement network is characterized in that: Two loss functions are combined to train the image fusion network. Specifically, the total loss function is designed first, as follows: L total =αL SSIM +βL MSE Among them, L MSE represents the MSE loss function, α and β are the parameters for adjusting L total The two hyperparameters of the value, L SSIM The formula for obtaining is: Among them, I O and I U Represents overexposed and underexposed images, I F Represents the fusion result; the definition of SSIM is as follows: Among them, μ and σ are the mean and standard deviation; parameters C1, C2 and C3 can keep the loss function numerically stable; The MSE loss function is used to obtain the mean square error between the source image and the fused image result, which is defined as: in, C F , H F and W F are the channel, height and width of the fused image respectively; p represents the number of pixels in an image.

3. According to claim 1, a multi-exposure image fusion method based on a multi-dimensional collaborative refinement network is characterized in that: The specific process of the image fusion network generating the RGB fused image is as follows: converting the overexposed and underexposed source images from RGB three channels to YCbCr three channels; the decomposed Y channel represents the brightness channel of the source image, which is used to fuse the real information; The Cb and Cr channels represent the chromaticity information of the input image. The following formula can give more weight to low-saturation color components and less weight to high-saturation color components. It is defined as: Among them, C f It is the weighted sum of Cb and Cr; Cb i and Cr i are the chromaticity values ​​of different channels in the source image respectively; τ is used as a hyperparameter; finally, the fused YCbCr result is converted to RGB result through inverse transformation.

Images