A multi-exposure image fusion system based on global-local aggregation learning

Abstract

The present invention discloses a multi-exposure image fusion system based on global-local aggregation learning, comprising: a collaborative aggregation module for acquiring image features required for fusion; a fusion module for reconstructing the fusion result according to the image features; and a cyclic refinement module for continuously mining information from the image using a cyclic mechanism to fine-tune the fusion result. The present invention can directly generate a high dynamic range image with rich details and balanced exposure based on a low dynamic range image; the collaborative aggregation module is used to mine relevant features from the source image, and a feature extraction mechanism that fuses global information and local information is successfully constructed. Secondly, a special fusion module is proposed to reconstruct the fused image, which effectively avoids artifacts and suppresses information attenuation. In addition, a cyclic refinement module is used to further fine-tune the fusion result to capture more texture details from the source image.

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and in particular to a multi-exposure image fusion system based on global-local aggregation learning. Background Art

[0002] Natural scenes are rich in light and shadow information, presenting a rich and colorful image to the human eye. However, digital cameras struggle to accurately capture well-exposed images. A single image often exhibits over- or underexposure in certain areas. This phenomenon occurs because the dynamic range of real scenes far exceeds the output dynamic range of the imaging or display device. To address this limitation, researchers typically employ both high-dynamic range (HDR) and standard imaging devices. High-dynamic range (HDR) imaging devices utilize specialized hardware to directly capture high dynamic range (LDR) images of the scene. Standard imaging devices, however, rely on multiple LDR image sequences of the target scene at varying exposure levels, synthesizing a single HDR image with crisp detail and realistic colors through an exposure fusion algorithm. Therefore, the stability of HDR images is influenced by two factors: the number of LDR images and the performance of the fusion strategy. Based on the number of LDR images, current fusion tasks can be categorized as non-extreme exposure fusion and extreme exposure fusion. However, an excessive number of LDR images inevitably increases storage requirements and processing time.

[0003] Over the past few decades, researchers have been dedicated to the multi-exposure fusion (MEF) task, producing a large number of excellent works. These works can be roughly divided into two categories: traditional methods and deep learning-based methods. For the traditional category, existing fusion methods are mainly based on transform domains and spatial domains. The former decomposes the image sequence into a transform domain and then designs fusion rules to reconstruct the fused image. The latter directly fuses the images in the pixel space, and its pixel values ​​are calculated linearly. The limitation of this type of method is that it requires manual design of fusion rules, which can easily lead to color deviation and blurred details in the fused image, and is time-inefficient.

[0004] In addition, deep learning has become very popular in recent years and has been successfully applied to image enhancement, super-resolution, image restoration and other low-level image tasks. Due to the excellent feature representation ability of deep learning, researchers have also introduced it into the field of MEF. Most existing MEF methods based on deep learning use convolutional neural networks (CNNs) to extract features. Compared with traditional methods, convolutional neural networks can directly interact with image pixels to obtain the required information, which is closer to the working method of the human visual system. However, these MEF methods also have some shortcomings: (1) Only the features of the local neighborhood are captured through convolution operations, ignoring the overall dependencies and losing some basic global context. (2) The fusion strategies used by some methods fail to fuse the features obtained from the source images well. Moreover, these methods are prone to information degradation during the fusion process, resulting in structural defects and color deviations in the fusion results. (3) Due to the large gaps between extremely exposed images, the detailed information of too bright or too dark areas is difficult to be fully utilized, so the fusion results may appear unclear and have poor visual perception. Summary of the Invention

[0005] The object of the present invention is to provide a multi-exposure image fusion system based on global-local aggregation learning, which effectively avoids artifacts, suppresses information attenuation, and improves the texture details of the fused image.

[0006] To achieve the above objectives, this application proposes a multi-exposure image fusion system based on global-local aggregation learning, comprising:

[0007] Collaborative aggregation module to obtain image features required for fusion;

[0008] Fusion module, reconstructs the fusion results according to image features;

[0009] The cyclic fine-tuning module uses a cyclic mechanism to continuously mine information from images and fine-tune the fusion results.

[0010] Furthermore, the collaborative aggregation module includes:

[0011] A non-local attention inference module for obtaining global dependencies from feature maps;

[0012] Local adaptive learning module, as a supplement to learning relevant local features.

[0013] Furthermore, the non-local attention inference module obtains the global feature information of the image by Gaussian random feature approximation and changing the multiplication order, specifically:

[0014] Use three different convolutional layers to process the initial image features Fo to obtain the intermediate feature representation sequence K, Q, V;

[0015] Get m different Gaussian random samples F1···F m , and connect them into a Gaussian random matrix F;

[0016] Sequence Q, sequence K are multiplied by Gaussian random matrix F to complete Q, K to Specific mapping of

[0017] Sequence V and Multiply again;

[0018] Normalize the results to generate a feature sequence that aggregates global information

[0019]

[0020] in, Represents Gaussian random transformation, that is, Q, K to The mapping process, N represents normalization.

[0021] Furthermore, the non-local attention inference module multiplies the initial feature map Fo by the magnification factor k (k>1) during the convolution layer processing of the initial image feature Fo to enhance the sparsity of the non-local attention weight, which is specifically expressed as where θ(·) is the feature transformation.

[0022] Furthermore, the local adaptive learning module includes a downsampling stage and an upsampling stage;

[0023] In the first downsampling stage, the initial image features Fo are subjected to global average pooling (GAP), and then the maximum pooling operation (MaxPool) is used to preserve the most important information in the neighborhood, and the resolution is reduced to After that, it is passed to the convolution layer to obtain the feature map a; in the second downsampling stage, the feature map a is processed using the average pooling operation (AvgPool), and the processed result is passed to another convolution layer to obtain a size of The intermediate feature map of

[0024] In the upsampling stage, the intermediate feature map is upsampled using Sub-pixel to obtain a size of Feature map b;

[0025] Concatenate feature map a and feature map b on the channel and then pass them into the 1×1 convolution layer; use Sub-pixel to generate the feature sequence of the local weight map And use the Sigmoid function to Normalized to [0, 1]; where h is the image height and w is the image width, Output representation for the feature map.

[0026] Furthermore, the fusion module first cascades the feature sequence With the initial image feature F o , and then perform four convolution operations on the cascaded features to obtain the initial fusion features; the initial image features F o Add it to the second convolution output to get the fused image

[0027] Furthermore, the recurrent refinement module constructs a recursive refinement network with three convolutional layers. The cyclic fine-tuning module is defined as where p t and p t-1 are the predicted feature maps of the t-th iteration and the t-1-th iteration, respectively.

[0028] Furthermore, the total loss function of this system is: Among them, L fused Is the fusion loss function, including the structural similarity loss function L SSIM , mean square error loss function L MSE And the total variation loss function L TV ; In addition, two hyperparameters α and β are introduced, and the fusion process is defined as:

[0029] L fused =L SSIM +α LMSE +βL TV

[0030] During the training phase, the structural similarity loss function can better learn structural information from the image, while the mean square error loss function is used to ensure pixel-level reconstruction; therefore, L SSIM and L MSE They are defined as:

[0031] L SSIM =1-SSIM(I f ,I i )

[0032] L MSE =||I f -I i ||2

[0033] Total variation loss L TV It is used to better preserve the gradient in the source image and further eliminate noise. It is defined as follows:

[0034] R(p,q)=I f (p,q)-I i (p,q)

[0035]

[0036] where R(p,q) represents the difference between the original image and the fused image, ||·||2 represents the L2 norm, and p and q represent the horizontal and vertical coordinates of the image pixels, respectively.

[0037] Compared with the existing technology, the above technical solutions adopted by the present invention have the following advantages: the present invention can directly generate a high dynamic range image with rich details and balanced exposure based on a low dynamic range image; and utilizes a collaborative aggregation module to mine relevant features from the source image, successfully constructing a feature extraction mechanism that fuses global information and local information. Secondly, a special fusion module is proposed to reconstruct the fused image, which effectively avoids artifacts and suppresses information attenuation. In addition, a cyclic refinement module is used to further fine-tune the fusion result and capture more texture details from the source image. This system can generate vivid fusion results in visual perception while also ensuring quantitative indicators, thus contributing to the development of multi-exposure image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of a multi-exposure image fusion system based on global-local aggregation learning;

[0039] Figure 2 This is the schematic diagram of the collaborative polymerization module;

[0040] Figure 3 This is the schematic diagram of the fusion module;

[0041] Figure 4 This is the schematic diagram of the cyclic fine calibration module;

[0042] Figure 5 The figure is a qualitative comparison between this patent and other advanced fusion methods on the SICE image dataset. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of this application more clearly understood, this application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. That is, the embodiments described are only part of the embodiments of this application, not all of them.

[0044] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present application.

[0045] Example 1

[0046] like Figure 1 As shown, the present application provides a multi-exposure image fusion system based on global-local aggregation learning, specifically including:

[0047] Collaborative aggregation module to obtain image features required for fusion;

[0048] Specifically, this module consists of two submodules: one is the non-local attention reasoning module, and the other is the local adaptive learning module. The details are as follows: Figure 2 shown.

[0049] The non-local attention inference module (NLAIN) can obtain more relevant global features in the source image. Based on the non-local attention mechanism, the kernel method is used to approximate the exponential function, which greatly suppresses the fusion of irrelevant features while reducing the computational complexity. Specifically, the global feature information of the image is obtained by approximating the Gaussian random features and changing the multiplication order. The initial image features Fo are processed with three different convolutional layers to obtain the intermediate feature representation sequence K, Q, V; this embodiment sets m different Gaussian random samples F1···F m , and concatenate them into a Gaussian random matrix F. Figure 2 (Part B) shows the specific transformation process. Sequence Q, sequence K are multiplied by Gaussian random matrix F to complete Q, K to The specific mapping of sequence V and Multiply again and normalize the result (represented by N) to generate a feature sequence that aggregates global information The specific expressions are as follows:

[0050]

[0051] In addition, to further improve the performance of NLAIN, sparse aggregation is also used to filter out irrelevant information and amplify the weight of relevant information. That is, when the convolution layer processes the initial image feature Fo, multiplying the initial feature map Fo by the amplification factor k (k>1) will force the non-local attention to give higher aggregation weights to relevant information. In essence, it enhances the sparsity of the non-local attention weights. Specifically expressed as where θ(·) is the feature transformation.

[0052] The local adaptive learning module is a supplement to the non-local attention reasoning module and is used to learn local information. It is divided into two stages: downsampling stage and upsampling stage, such as Figure 2 (Part A) In the first downsampling stage, the initial image features Fo are globally average pooled, and then the maximum pooling operation is used to preserve the most important information in the neighborhood, and the resolution is reduced to After that, it is passed to the convolution layer to obtain the feature map a; in the second downsampling stage, the feature map a is processed using the average pooling operation, and the processed result is passed to another convolution layer to obtain a feature map of size In the upsampling stage, the intermediate feature map is upsampled using Sub-pixel to obtain a size of Feature map b; concatenate feature map a and feature map b on the channel and then pass them into the 1×1 convolution layer; use Sub-pixel to generate a feature sequence of the local weight map And use the Sigmoid function to Normalized to [0, 1].

[0053] Fusion module, reconstructs the fusion results according to image features;

[0054] Specifically, such as Figure 3 As shown, this module uses two feature sequences generated by the collaborative aggregation module The global-local information of the source image is retained to reconstruct the fused image. Specifically, the feature sequence is first cascaded With the initial image feature F o , the purpose is to balance the pixel intensity distribution and overcome the loss of details in small areas at the same time. Then these sequences are convolved four times to obtain the preliminary fusion features. At the same time, in order to make up for the information lost in the extraction process of the coordinated aggregation module, the initial image feature F is introduced o , and adds an addition operation to convert the initial image feature F o Adding it to the second convolution output, we can get a preliminary friendly fusion image. In order to further improve the detail quality of the fused image, the present invention introduces a recurrent refinement module (RRM) to generate a predicted feature map p, thereby guiding Generate the final fusion result with complementary and complete information.

[0055] The cyclic fine-tuning module uses a cyclic mechanism to continuously mine information from images and fine-tune the fusion results.

[0056] Specifically, in order to improve the detail texture of the fused image, this module constructs a recursive refinement network with three convolutional layers. The details are as follows Figure 4 As shown. RRM is defined as where p t and p t-1 are the predicted feature maps of the tth iteration and the t-1th iteration respectively. The initial prediction map is defined as zero, and the number of iterations is T.max It can be set to 3. As the number of iterations increases, the data distribution of the predicted feature map is closer to the initial image feature F o distribution.

[0057] In order to encourage the system to learn pixel-level information from the source image and to capture the structure and gradient information in the image, the present invention introduces three loss functions to constrain the original image I i And the final fused image I f The total loss function takes the following form: Among them, L fused Is the fusion loss function, including the structural similarity loss function L SSIM , mean square error loss function L MSE And the total variation loss function L TV In addition, two hyperparameters α and β are introduced, and the fusion process is defined as:

[0058] L fused =L SSIM +αL MSE +βL TV

[0059] During the training phase, the structural similarity loss function helps the system better learn structural information from the image, while the mean square error loss function is used to ensure pixel-level reconstruction. SSIM and L MSE They are defined as:

[0060] L SSIM =1-SSIM(I f ,I i )

[0061] L MSE =||I f -I i ||2

[0062] Total variation loss L TV It is used to better preserve the gradient in the source image and further eliminate noise. It is defined as follows:

[0063] R(p,q)=I f (p,q)-I i (p,q)

[0064]

[0065] where R(p,q) represents the difference between the original image and the fused image, ||·||2 represents the L2 norm, and p and q represent the horizontal and vertical coordinates of the image pixels, respectively.

[0066] This paper selects test image sequences from the SICE image dataset to compare with ten state-of-the-art multi-exposure image fusion methods. Figure 5 The overall effect and local feature details are shown in the figure. From the details in the figure, it can be seen that the fusion results of the present invention are well restored in terms of both color and detail. The fusion results of some methods have obvious black shadows, such as the sky and clouds above the buildings. The colors of some areas are unbalanced in exposure, and the color information cannot be restored well. It is also easy to have blurred details and structural defects, especially the edges of house walls and trees. After comparison, the present invention can generate an image with balanced exposure, and its details and colors can be well restored.

[0067] In addition to the subjective qualitative analysis, we began to objectively analyze the image quality. We selected MEF-SSIM and PSNR, the two most commonly used and important image metrics in multi-exposure image fusion tasks, to evaluate the fusion capabilities of different methods. A quantitative analysis was performed on the SICE dataset. Based on 100 test image pairs, the average values ​​of the two metrics were calculated for 11 methods (including the method proposed in this paper). Table 1 shows the quantitative results. The results show that compared with the other ten methods, the fusion results generated by this method have higher MEF-SSIM and PSNR values. The statistics in the table also reflect that this system can provide high-quality visual results with balanced exposure and clear texture details.

[0068] Table 1 shows the quantitative comparison between this patent and other advanced fusion methods on the SICE dataset.

[0069]

[0070]

[0071] The foregoing descriptions of specific exemplary embodiments of the present invention are for purposes of illustration and description. These descriptions are not intended to limit the invention to the precise forms disclosed, and it is apparent that many variations and modifications are possible in light of the foregoing teachings. The exemplary embodiments have been selected and described for the purpose of explaining the specific principles of the invention and their practical application, thereby enabling those skilled in the art to realize and utilize a variety of exemplary embodiments of the invention and various options and modifications. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A multi-exposure image fusion system based on global-local aggregation learning, characterized in that: include: Collaborative aggregation module to obtain image features required for fusion; Fusion module, reconstructs the fusion results according to image features; The cyclic fine-tuning module uses a cyclic mechanism to continuously mine information from the image and fine-tune the fusion results; The collaborative aggregation module includes: A non-local attention inference module for obtaining global dependencies from feature maps; Local adaptive learning module, as a supplement to learning relevant local features; The non-local attention inference module obtains the global feature information of the image by Gaussian random feature approximation and changing the multiplication order, specifically: Use three different convolutional layers to process the initial image features Fo to obtain the intermediate feature representation sequence K, Q, V; Get m different Gaussian random samples F1···F m , and connect them into a Gaussian random matrix F; Sequence Q, sequence K are multiplied by Gaussian random matrix F to complete Q, K to Specific mapping of Sequence V and Multiply again; Normalize the results to generate a feature sequence that aggregates global information in, Represents Gaussian random transformation, that is, Q, K to The mapping process, N represents normalization; The non-local attention inference module multiplies the initial feature map Fo by the magnification factor k (k>1) during the convolution layer processing of the initial image feature Fo to enhance the sparsity of the non-local attention weight, which is specifically expressed as where θ(·) is the feature transformation; The local adaptive learning module includes a downsampling stage and an upsampling stage; In the first downsampling stage, the initial image features Fo are globally average pooled, and then a maximum pooling operation is used to preserve the most important information in the neighborhood, and the resolution is reduced to After that, it is passed to the convolution layer to obtain the feature map a; in the second downsampling stage, the feature map a is processed using the average pooling operation, and the processed result is passed to another convolution layer to obtain a feature map of size The intermediate feature map of In the upsampling stage, the intermediate feature map is upsampled using Sub-pixel to obtain a size of Feature map b; Concatenate feature map a and feature map b on the channel and then pass them into the 1×1 convolution layer; use Sub-pixel to generate the feature sequence of the local weight map And use the Sigmoid function to Normalized to [0, 1]; where h is the image height and w is the image width, Output representation for the feature map; The fusion module first cascades the feature sequence With the initial image feature F o , and then perform four convolution operations on the cascaded features to obtain the initial fusion features; the initial image features F o Add it to the second convolution output to get the fused image The recurrent refinement module constructs a recursive refinement network with three convolutional layers The cyclic fine-tuning module is defined as where p t and p t-1 are the predicted feature maps of the t-th iteration and the t-1-th iteration, respectively.

2. The multi-exposure image fusion system based on global-local aggregate learning according to claim 1, characterized in that: The total loss function of this system is: Among them, L fused Is the fusion loss function, including the structural similarity loss function L SSIM , mean square error loss function L MSE And the total variation loss function L TV ; In addition, two hyperparameters α and β are introduced, and the fusion process is defined as: L fused =L SSIM +αL MSE +βL TV During the training phase, the structural similarity loss function can better learn structural information from the image, while the mean square error loss function is used to ensure pixel-level reconstruction; therefore, L SSIM and L MSE They are defined as: L SSIM =1-SSIM(I f ,I i ) L MSE =||I f -I i ||2 Total variation loss L TV It is used to better preserve the gradient in the source image and further eliminate noise. It is defined as follows: R(p,q)=I f (p,q)-I i (p,q) where R(p,q) represents the difference between the original image and the fused image, ||·||2 represents the L2 norm, and p and q represent the horizontal and vertical coordinates of the image pixels, respectively.

Images