Multi-exposure image fusion method and system for optimizing rendering quality and visual reality

Abstract

The invention discloses a multi-exposure image fusion method and system for optimizing rendering quality and visual reality, and relates to the technical field of image processing. Comprising the following steps: splicing a multi-exposure image, and inputting the image into a convolutional neural network for parameter extraction to obtain a feature value; the feature values are put into a network model for end-to-end unsupervised learning, and a fusion image model is generated; the fusion image model is trained, in the training process, a loss function is used for continuously conducting back propagation iteration, learning content is updated, and the loss function is mean square error loss and structural similarity loss; and performing multi-exposure image fusion by using the trained fusion image model. According to the method, fusion operation can be carried out on an existing public multi-exposure image data set, texture details and brightness information of the multi-exposure image can be effectively extracted from a fused result, information loss and color distortion are effectively avoided after the texture details and the brightness information are combined, and a fused image better conforming to the visual effect of human eyes is formed.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-exposure image fusion method and system for optimizing rendering quality and visual realism. Background Art

[0002] Multi-exposure image fusion is an image enhancement technique that fuses multiple raw input images into an image with high visual quality. It plays a vital role in many fields such as medicine, remote sensing, and autonomous driving. Due to the limitations of imaging devices, a single device can often only capture a limited range of image details, causing the image to experience brightness degradation to varying degrees, forming overexposed or underexposed areas, thereby damaging the image quality. In order to obtain high dynamic range (HDR) images, researchers have tried to develop a variety of complex hardware devices, but these devices are not only expensive to produce, but also suboptimal in efficiency, so they are difficult to be widely promoted in real applications. In order to obtain satisfactory images with good exposure and rich details, multi-exposure image fusion methods have emerged and have been explored and applied. This method generates a high dynamic range image that conforms to human perception by merging low dynamic range (LDR) images captured under different exposure conditions, thereby obtaining a fused image containing complete information of the scene. The goal of multi-exposure image fusion is to merge LDR image sequences under different exposure conditions, extract features from these sequences, and cleverly fuse them into vivid and realistic HDR images.

[0003] The main challenge of multi-exposure image fusion is how to effectively extract the salient features of the two source images and design an appropriate fusion strategy to generate a high-quality fused image that combines the important information of both sources. To solve this problem, many methods have emerged, which can be roughly divided into two branches: one is the traditional fusion method, and the other is the method based on deep learning.

[0004] Traditional image fusion methods cover a variety of techniques such as multi-scale transform (MST), sparse representation (SR), and hybrid models. They usually follow a fixed strategy to extract features in the spatial domain or transform domain and fuse these features according to simple rules. Subsequently, the corresponding inverse transform technology is used to reconstruct the final fused image. However, the effectiveness of these methods is highly dependent on complex metrics or fusion rules carefully designed by humans, which is not only inefficient in time, but also leads to large memory consumption and computational burden. In recent studies, deep learning-based image fusion methods such as DeepFuse, CRMEF, and IFCNN have emerged. These methods make full use of the powerful capabilities of neural networks in feature extraction and nonlinear fitting, and effectively overcome many limitations of traditional methods. However, they also face some challenges. First, in the process of pursuing global image quality improvement, some algorithms may sacrifice the accuracy of local details, resulting in unsatisfactory details in some areas of the image. Second, although some image fusion methods are committed to preserving the color, details, and saturation of the image, they are still insufficient in dealing with the problem of uneven image illumination. Finally, although existing methods can generate images with clear details and good overall visual effects, they often cannot effectively correct the lighting distribution under extreme lighting conditions, causing the details of certain key areas to be obscured by improper light intensity, affecting the overall presentation of the image. Summary of the invention

[0005] The purpose of the present invention is to propose a multi-exposure image fusion method and system for optimizing rendering quality and visual realism, which combines texture feature information and brightness feature information to generate a fused image with good effect.

[0006] According to a first aspect of an embodiment of the present disclosure, a multi-exposure image fusion method for optimizing rendering quality and visual realism is provided, comprising the following steps:

[0007] After the multi-exposure images are stitched together, they are input into a convolutional neural network for parameter extraction to obtain feature values;

[0008] Then put the feature values ​​into the network model for end-to-end unsupervised learning to generate a fused image model;

[0009] Training the fused image model, and continuously performing back propagation iterations using a loss function during the training process to update the learning content, wherein the loss function is a mean square error loss and a structural similarity loss;

[0010] Use the trained fusion image model to perform multi-exposure image fusion.

[0011] According to a second aspect of an embodiment of the present disclosure, a multi-exposure image fusion system for optimizing rendering quality and visual realism is provided, including:

[0012] The feature value acquisition module performs a splicing operation on the multi-exposure images and then inputs them into a convolutional neural network for parameter extraction to obtain feature values;

[0013] The model generation module then puts the feature values ​​into the network model for end-to-end unsupervised learning to generate a fused image model;

[0014] A training module, for training the fused image model, and during the training process, back propagation iteration is continuously performed using a loss function to update the learning content, wherein the loss function is a mean square error loss and a structural similarity loss;

[0015] The fusion module uses the trained fusion image model to perform multi-exposure image fusion.

[0016] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored and running on the memory, wherein when the processor executes the program, the multi-exposure image fusion method for optimizing rendering quality and visual realism is implemented.

[0017] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the multi-exposure image fusion method for optimizing rendering quality and visual realism is implemented.

[0018] Compared with the prior art, the above technical solution adopted by the present invention has the following advantages: the present application can perform fusion operations on the existing public multi-exposure image data set, and the fused result can effectively extract the texture information and brightness details of the multi-exposure images, and the two are combined to form a fused image that is more in line with the visual effect of the human eye. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings in the specification, which constitute a part of the present application, are used to provide further understanding of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application.

[0020] Figure 1 Flowchart of the multi-exposure image fusion method for optimizing rendering quality and visual realism;

[0021] Figure 2 This is the network model architecture diagram;

[0022] Figure 3 It is a qualitative comparison diagram between the present invention and other advanced fusion methods. DETAILED DESCRIPTION

[0023] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0024] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0025] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0026] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the methods and systems according to various embodiments of the present disclosure. It should be noted that each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of a code may include one or more executable instructions for implementing the logical functions specified in each embodiment. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, or they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the flowchart and / or block diagram, and the combination of boxes in the flowchart and / or block diagram can be implemented using a dedicated hardware-based system that performs a specified function or operation, or can be implemented using a combination of dedicated hardware and computer instructions.

[0027] Embodiment 1:

[0028] This embodiment provides a multi-exposure image fusion method for optimizing rendering quality and visual realism, including the following steps:

[0029] S1. After the multi-exposure images are stitched together, they are input into a convolutional neural network for parameter extraction to obtain feature values;

[0030] Specifically, the multi-exposure images are spliced ​​and connected in the direction of the number of channels; the extracted feature map values ​​are input into the convolutional neural network to obtain the shallow features as follows:

[0031] F x =Conv1([I o ,I u ])

[0032] Among them, I o ,Iu denote overexposed and underexposed images respectively, and Conv1 denotes a convolutional layer with a 1×1 kernel.

[0033] S2. Then put the feature values ​​into the network model for end-to-end unsupervised learning to generate a fused image model;

[0034] Specifically, the present invention makes full use of the powerful representation ability of neural networks and establishes a network model, which includes a texture feature extractor, an illumination feature extractor and a feature recombiner. Figure 1 As shown in Figure 1, the fused image is fused through the feature recombiner, so the network proposed in this method is an end-to-end network architecture and does not require a pre-training module. The kernel sizes of the convolution kernels are 1×1 and 3×3, and the step size is set to 1.

[0035] In the texture feature extractor, such as Figure 2 As shown in the figure, it contains detail processing blocks and low-frequency enhancement blocks. In the detail processing block, the upper branch captures long-range dependencies through residual convolution and focuses on extracting and utilizing global context information in the scene, which helps the model understand the overall structure and layout of the image to enhance global information; in the lower branch, two Sobel operators in different directions are used to calculate image gradients to enhance edge details in the image, capture object boundaries and texture information, and ensure the accuracy and richness of local details; the low-frequency enhancement block uses adaptive average pooling of sizes 1×1, 2×2, 3×3, and 6×6, and uses upsampling at the end of each scale to restore the original size of the feature, enriching the semantics of the reconstructed image, and jointly improving image details and reducing high-frequency noise. Specifically, the detail processing block includes two branches, one branch includes a front residual block, a rear residual block, three middle convolutional layers, a Softmax function and an activation function, and the front residual block and the rear residual block each include three convolutional layers and an activation function; the other branch includes a Sobel operation in the horizontal direction, a Sobel operation in the vertical direction, and a convolutional layer after the addition operation; the low-frequency enhancement block includes four adaptive pooling layers of different sizes and a convolutional layer.

[0036] In the illumination feature extractor, the light is usually assumed to be uniform, so global average pooling GAP(·) is used to eliminate unnecessary information in the feature space. One branch includes a global pooling layer, two convolutional layers, an activation function and a sigmoid function; the other branch includes three convolutional layers, an activation function and a sigmoid function.

[0037] The front layer of the feature reorganizer includes two convolutional layers, an activation function and a sigmoid function, the middle layer is a shared layer including three convolutional layers and two activation functions, and the back layer includes one convolutional layer and an activation function.

[0038] S3. training the fused image model, and continuously performing back propagation iterations using a loss function during the training process to update the learning content, wherein the loss function is a mean square error loss and a structural similarity loss;

[0039] Specifically, the loss function consists of two parts, and the overall definition is as follows:

[0040] L=L SSIM +λL MSE

[0041] Among them, SSIM represents structure, MSE represents mean square error; L SSIM Control the overall feeling of the fusion result, L MSE The role of is to highlight visual perception, and λ is a hyperparameter that adjusts the value of the loss function.

[0042] SSIM can measure the structural similarity between the input image and the fused image, and it is very sensitive to lighting, contrast distortion, and correlation loss. SSIM The calculation formula is as follows:

[0043] L SSIM =[1-SSIM(I o ,I f )}+[1-SSIM(I u ,I f )]

[0044] The specific calculation formula of SSIM is:

[0045]

[0046] Among them, I a Represents the original input image, that is, I o and I u ,,μ represents the mean, σ represents the standard deviation, and C1, C2 and C3 are parameters used to keep the matrix stable.

[0047] MSE represents the mean square error between two images and is calculated as:

[0048]

[0049] Where p represents the pixel value of the image. MSE is defined as:

[0050]

[0051] S4. Use the trained fusion image model to perform multi-exposure image fusion.

[0052] like Figure 3As shown, the details of the fused image are marked with boxes in prominent positions, and an enlarged view is provided in the lower right corner for easy observation. Compared with the other twelve methods, the present invention has three core advantages. First, the present invention can retain more detail information, which is mainly due to its unique detail processing module, which can effectively utilize the data of each convolution layer. Therefore, after the technical processing of the present invention, the tree morphology in the image is more realistic, and the outline of the coconut tree is also clearer. Secondly, the present invention can capture more edge and texture information; for example, the wall of the house in the image and the lines of the wall under the light are more prominent due to the use of the structural similarity loss function. Finally, the image provided by the present invention presents a better visual effect as a whole, and can maintain superior performance in terms of image information richness and contrast. This is due to the effective extraction of global and local features by the detail processing module in the network architecture, and the full capture of color information by the illumination feature extractor, and the fusion and reconstruction of the global features and detail features of the image are realized in the final convolution layer.

[0053] In addition to the qualitative analysis, a quantitative analysis was also performed on the SICE dataset, using four image metrics: mutual information, standard deviation, correlation coefficient, and peak signal-to-noise ratio for evaluation, as shown in Table 1.

[0054] Table 1 is the quantitative analysis average value of the present invention

[0055]

[0056] As shown in Table 1, from the perspective of similarity, mutual information can measure the similarity between two images and the amount of information that the fused image has obtained from the original image. The larger the mutual information, the more the fused image retains the source image information and the better the quality. However, as the amount of information increases, the noise information in the fused image also increases, which may cause the overall image effect to be distorted. Therefore, although the mutual information value of the image generated by the present invention is not the best, it does not affect the effect of image performance. The standard deviation is an objective evaluation index for measuring the richness of image information. The larger the value, the more dispersed the grayscale distribution of the image, the more information the image carries, and the better the quality of the fused image. The present invention has a good standard deviation representation, and the fused image generated by the present method has rich texture details. The correlation coefficient measures the linear correlation level between the source image and the fused image. The present invention has a higher correlation coefficient level, which means that the present method can provide a good performance for the fused image. The peak signal-to-noise ratio is used to measure the ratio between the effective information and noise of the image, which can reflect whether the image is distorted. The larger the value, the better the quality of the fused image. The present invention has a good peak signal-to-noise ratio, which can better highlight the performance of the image.

[0057] In the present invention, a new end-to-end deep learning framework is proposed to solve the multi-exposure image fusion task. In order to supplement the missing details in the global information and thus ensure the consistency and integrity of the overall scene, a texture feature extractor is introduced, which cuts in from two directions. The detail processing block focuses on extracting and utilizing the global context information and edge details in the scene. The low-frequency enhancement block predicts key information and enriches the semantics of the reconstructed image. In order to enhance the visual reality, an illumination feature extractor is designed to better enhance the change of color information. The role of the feature recombiner is to fully fuse the information of the two and overcome the details of certain areas being obscured by improper light intensity. After qualitative and quantitative analysis of the SICE dataset, it is shown that the method proposed in the present invention can effectively fuse images of different exposure levels.

[0058] Embodiment 2:

[0059] This embodiment provides a multi-exposure image fusion system for optimizing rendering quality and visual realism, including:

[0060] The feature value acquisition module performs a splicing operation on the multi-exposure images and then inputs them into a convolutional neural network for parameter extraction to obtain feature values;

[0061] The model generation module then puts the feature values ​​into the network model for end-to-end unsupervised learning to generate a fused image model;

[0062] A training module, for training the fused image model, and during the training process, back propagation iteration is continuously performed using a loss function to update the learning content, wherein the loss function is a mean square error loss and a structural similarity loss;

[0063] The fusion module uses the trained fusion image model to perform multi-exposure image fusion.

[0064] Embodiment three:

[0065] An electronic device includes a memory, a processor, and a computer program stored and running on the memory, wherein the processor implements the above-mentioned multi-exposure image fusion method for optimizing rendering quality and visual realism when executing the program, including:

[0066] After the multi-exposure images are stitched together, they are input into a convolutional neural network for parameter extraction to obtain feature values;

[0067] Then put the feature values ​​into the network model for end-to-end unsupervised learning to generate a fused image model;

[0068] Training the fused image model, and continuously performing back propagation iterations using a loss function during the training process to update the learning content, wherein the loss function is a mean square error loss and a structural similarity loss;

[0069] Use the trained fusion image model to perform multi-exposure image fusion.

[0070] Embodiment 4:

[0071] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned multi-exposure image fusion method for optimizing rendering quality and visual realism, including:

[0072] After the multi-exposure images are stitched together, they are input into a convolutional neural network for parameter extraction to obtain feature values;

[0073] Then put the feature values ​​into the network model for end-to-end unsupervised learning to generate a fused image model;

[0074] Training the fused image model, and continuously performing back propagation iterations using a loss function during the training process to update the learning content, wherein the loss function is a mean square error loss and a structural similarity loss;

[0075] Use the trained fusion image model to perform multi-exposure image fusion.

[0076] Those skilled in the art should understand that the modules or steps of the present disclosure can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present disclosure is not limited to any specific combination of hardware and software.

[0077] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0078] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A multi-exposure image fusion method for optimizing rendering quality and visual realism, characterized in that: The following steps are involved: After the multi-exposure images are stitched together, they are input into a convolutional neural network for parameter extraction to obtain feature values; Then put the feature values ​​into the network model for end-to-end unsupervised learning to generate a fused image model; Training the fused image model, and continuously performing back propagation iterations using a loss function during the training process to update the learning content, wherein the loss function is a mean square error loss and a structural similarity loss; Use the trained fusion image model to perform multi-exposure image fusion.

2. The multi-exposure image fusion method for optimizing rendering quality and visual realism according to claim 1, characterized in that: After the multi-exposure images are stitched together, they are input into the convolutional neural network for parameter extraction to obtain feature values, specifically: The multi-exposure images are stitched together and connected in the direction of the number of channels; the extracted feature map values ​​are input into the convolutional neural network to obtain shallow features, as shown below: F x =Conv1([I o ,I u ]) Among them, I o , I u denote overexposed and underexposed images respectively, and Conv1 denotes a convolutional layer with a 1×1 kernel.

3. The multi-exposure image fusion method for optimizing rendering quality and visual realism according to claim 1, characterized in that: The network model includes a texture feature extractor, an illumination feature extractor and a feature recombiner, the texture feature extractor includes a detail processing block and a low-frequency enhancement block, the illumination feature extractor includes two branches, and the feature recombiner includes multiple convolutional layers and activation functions.

4. The multi-exposure image fusion method for optimizing rendering quality and visual realism according to claim 3, characterized in that: The detail processing block includes two branches, one branch includes a front residual block, a rear residual block, three middle convolutional layers, a Softmax function and an activation function, and the front residual block and the rear residual block each include three convolutional layers and an activation function; the other branch includes a Sobel operation in the horizontal direction, a Sobel operation in the vertical direction, and a convolutional layer after the addition operation; the low-frequency enhancement block includes four adaptive pooling layers of different sizes and a convolutional layer.

5. The multi-exposure image fusion method for optimizing rendering quality and visual realism according to claim 3, characterized in that: One branch of the illumination feature extractor includes a global pooling layer, two convolutional layers, an activation function and a sigmoid function; the other branch includes three convolutional layers, an activation function and a sigmoid function.

6. The multi-exposure image fusion method for optimizing rendering quality and visual realism according to claim 3, characterized in that: The front layer of the feature reorganizer includes two convolution layers, an activation function and a sigmoid function, the middle layer is a shared layer including three convolution layers and two activation functions, and the back layer includes one convolution layer and an activation function.

7. The multi-exposure image fusion method for optimizing rendering quality and visual realism according to claim 1, characterized in that: The loss function is defined as follows: L=L SSIM +λL MsE Among them, SSIM represents structure, MSE represents mean square error; L SSIM Controls the overall feeling of the fusion result, L MSE The role of is to highlight visual perception, and λ is a hyperparameter that adjusts the value of the loss function; Loss function L SSIM The calculation formula is as follows: L SSIM =[1-SSIM(I o , I f )]+[1-SSIM(I u , I f )] Among them, the specific calculation formula of SSIM is: Among them, I a Represents the original input image, that is, I o and I u , μ represents the mean, σ represents the standard deviation, C1, C2 and C3 are parameters used to keep the matrix stable; The MSE calculation formula is: Among them, p represents the pixel value of the image; the loss function L MSE is defined as:

8. A multi-exposure image fusion system that optimizes rendering quality and visual realism, characterized by: include: The feature value acquisition module performs a splicing operation on the multi-exposure images and then inputs them into a convolutional neural network for parameter extraction to obtain feature values; The model generation module then puts the feature values ​​into the network model for end-to-end unsupervised learning to generate a fused image model; A training module, for training the fused image model, and during the training process, back propagation iterations are continuously performed using a loss function to update the learning content, wherein the loss function is a mean square error loss and a structural similarity loss; The fusion module uses the trained fusion image model to perform multi-exposure image fusion.

9. An electronic device comprising a memory, a processor and a computer program stored and running on the memory, characterized in that: When the processor executes the program, the multi-exposure image fusion method for optimizing rendering quality and visual realism as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the multi-exposure image fusion method for optimizing rendering quality and visual realism as described in claims 1-7 is implemented.

Images