No-ghosting high-dynamic range image fusion method and system

Abstract

The invention is applicable to the technical field of computer imaging, and provides a ghosting-free high-dynamic-range image fusion method, which comprises the following steps of: based on a low-exposure image, a medium-exposure image and a high-exposure image of an original image, adopting gamma correction to obtain an input image of the low-exposure image and an input image of the high-exposure image; based on the input image of the low-exposure image and the input image of the high-exposure image, adopting a double-domain separation cooperation method to obtain a first double-domain image of the low-exposure image and a first double-domain image of the high-exposure image; splicing the LDR domain image and the HDR domain image of the mid-exposure image to obtain a third double-domain image of the mid-exposure image; and fusing the first double-domain image of the low-exposure image, the first double-domain image of the high-exposure image and the third double-domain image of the medium-exposure image to obtain a target fused image. According to the invention, the method can effectively deal with the ghosting condition caused by the coexistence of the noise and the saturation region in the motion process of the object, thereby generating a ghosting-free high-quality target fusion image.

Description

Technical Field

[0001] The present application belongs to the field of computer imaging technology, and in particular, relates to a ghost-free high dynamic range image fusion method and system. Background Art

[0002] In computer imaging, high dynamic range (HDR) technology can capture a higher range of color, brightness and contrast than ordinary photos, and can show both bright skies and dark shadows in the same photo, avoiding overexposure or underexposure, and presenting a more realistic and delicate picture effect. However, in dynamic scene shooting, due to the movement of the camera or object, low dynamic range (LDR) images are often difficult to align completely, resulting in artifacts and double images when HDR images are merged, the so-called "ghosting", which affects the quality of the final image.

[0003] In the prior art, to address the ghosting problem in HDE imaging, a context-aware visual transformer, CA-ViT, is usually used to eliminate ghosting. CA-ViT effectively captures global and local dependencies by adopting a dual-branch architecture. The global branch uses a window-based Transformer encoder to simulate the motion and intensity changes of distant objects, thereby solving the ghosting problem caused by object movement or different exposures. The local branch designs a local context extractor (LCE) to capture the short-range features of the image, and uses a channel attention mechanism to select informative local details in the extracted features to supplement the global branch, so that the global and local information can work together in a complementary form.

[0004] The above-mentioned method of eliminating ghosting using CA-ViT overcomes the limitations of traditional convolutional neural network (CNN)-based methods that are prone to ghosting artifacts and intensity distortion when processing scenes with large motion and severe saturation. The dual-branch architecture can better model long-distance dependencies, thereby generating high-quality HDR images without ghosting. However, when a set of multi-exposure LDR images simultaneously contain occlusion caused by object motion and brightness oversaturation or darkness, ghosting will also appear in the HDR images fused using the above method. This is because they directly splice the LDR domain of the multi-exposure image with its HDR domain for modeling, which makes it impossible to effectively handle scenes where noise and saturated areas coexist during object motion. Summary of the invention

[0005] The embodiments of the present application provide a ghost-free high dynamic range image fusion method and system, which can solve the problem in the prior art that ghost images are generated by the coexistence of noise and saturation areas during the motion of an object.

[0006] In a first aspect, an embodiment of the present application provides a ghost-free high dynamic range image fusion method, comprising:

[0007] Based on a low-exposure image, a medium-exposure image, and a high-exposure image of the original image, gamma correction is performed to obtain an input image of the low-exposure image and an input image of the high-exposure image, wherein the input image of the low-exposure image and the input image of the high-exposure image both include an LDR domain input image and an HDR domain input image;

[0008] Based on the input image of the low-exposure image and the input image of the high-exposure image, a dual-domain separation and cooperation method is adopted to obtain a first dual-domain image of the low-exposure image and a first dual-domain image of the high-exposure image;

[0009] splicing the LDR domain image and the HDR domain image of the middle exposure image to obtain a third dual-domain image of the middle exposure image;

[0010] The first dual-domain image of the low-exposure image, the first dual-domain image of the high-exposure image, and the third dual-domain image of the medium-exposure image are fused to obtain a target fused image.

[0011] Furthermore, the low-exposure image, the medium-exposure image, and the high-exposure image based on the original image are subjected to gamma correction to obtain the input image of the low-exposure image and the input image of the high-exposure image, including:

[0012] Obtain a low-exposure image, a medium-exposure image, and a high-exposure image of the original image;

[0013] Taking the medium exposure image as a reference image, adopting gamma correction, mapping the low exposure image and the high exposure image to the HDR domain in sequence, obtaining a second dual-domain image of the low exposure image and a second dual-domain image of the high exposure image, wherein the second dual-domain image of the low exposure image and the second dual-domain image of the high exposure image both include an LDR domain image and an HDR domain image;

[0014] Based on the second dual-domain image of the low-exposure image and the second dual-domain image of the high-exposure image, an input image of the low-exposure image and an input image of the high-exposure image are obtained, and the input image of the low-exposure image and the input image of the high-exposure image both include an LDR domain input image and an HDR domain input image.

[0015] Furthermore, obtaining the input image of the low-exposure image and the input image of the high-exposure image based on the second dual-domain image of the low-exposure image and the second dual-domain image of the high-exposure image includes:

[0016] splicing the LDR image of the low exposure image and the LDR image of the high exposure image with the LDR image of the medium exposure image respectively to obtain an LDR domain input image of the low exposure image and an LDR domain input image of the high exposure image;

[0017] The HDR image of the low exposure image and the HDR image of the high exposure image are respectively spliced ​​with the HDR image of the medium exposure image to obtain an HDR domain input image of the low exposure image and an HDR domain input image of the high exposure image.

[0018] Furthermore, the dual-domain separate collaboration method includes: inputting the input image of the low-exposure image and the input image of the high-exposure image into a dual-domain alignment network respectively to obtain a first dual-domain image of the low-exposure image and a first dual-domain image of the high-exposure image.

[0019] Furthermore, the dual-domain separate collaboration method further includes:

[0020] Based on the LDR domain input image and the HDR domain input image, a layer normalization method is adopted to obtain the LDR domain feature map and the HDR domain feature map;

[0021] Inputting the LDR domain feature map and the HDR domain feature map into a dual-channel attention module respectively to obtain a comprehensive attention feature map, wherein the dual-channel attention module includes spatial self-attention and channel self-attention;

[0022] The comprehensive attention feature map is input into a feedforward neural network to obtain a first dual-domain image, wherein the first dual-domain image includes an LDR domain output image and an HDR domain output image.

[0023] Furthermore, the above-mentioned ghost-free high dynamic range image fusion method further includes:

[0024] The target fused image is mapped from the linear domain to the tone mapping domain using a tone mapping loss function, and the loss function of the target fused image is calculated. The loss function includes a dual-domain collaboration loss function, a dual-domain alignment loss function, and an image fusion loss function. The calculation formula of the tone mapping loss function is:

[0025]

[0026] Where x is the input function, u is a fixed value of 5000, and τ(x) represents the tone mapping loss function;

[0027] The calculation formula of the loss function is:

[0028] L=L collaboration +loss aligment +Lfusion

[0029] Among them, L collaboration represents the dual-domain collaborative loss function, loss aligment represents the dual domain alignment loss function, L fusion represents the image fusion loss function, L represents the loss function;

[0030] Further, the dual-domain collaborative loss function calculates the loss after the first dual-domain image is obtained by using the dual-domain separate collaborative method;

[0031] The calculation formula of the dual-domain collaboration loss function is as follows:

[0032]

[0033] Among them, ψ(.) is the feature map output by the pre-trained VGG-16 network, i and j represent the jth feature map after the i-th pooling layer, L1 is the low-exposure image, and L3 is the high-exposure image. and are respectively the LDR domain image and the HDR domain image of the first dual domain image of the low exposure image, and are respectively the LDR domain image and the HDR domain image of the first dual domain image of the high exposure image, The tone mapping loss function of the HDR domain image representing the first dual domain image of the low exposure image, Tone mapping loss function of the HDR domain image representing the first dual domain image of the high exposure image.

[0034] Further, the dual-domain alignment loss function calculates the loss between the LDR domain image and the HDR domain image in the second dual-domain image;

[0035] The calculation formula of the dual-domain alignment loss function is as follows:

[0036] loss alignment =loss 1 +loss 2 +loss 3 +loss 4 ;

[0037] Among them, loss 1 Represents the loss of the LDR domain image of the low exposure image, loss 2 Represents the loss of the HDR domain image of the low-exposure image, loss 3 Represents the loss of the LDR domain image of the high exposure image, loss 4 Represents the loss of HDR domain images for high exposure images.

[0038] Furthermore, the image fusion loss function calculates the loss between the target fused image and the real image;

[0039] The calculation formula of the image fusion loss function is as follows:

[0040] L fusion =||τ(H gt )-τ(H out )|| 1 +λ||ψ i,j (τ(H gt )-ψ i,j (τ(H out )))|| 1

[0041] Among them, ψ(.) is the feature map output by the pre-trained VGG-16 network, i and j represent the jth feature map after the i-th pooling layer, λ is 0.01, H out represents the target fusion image, H gt represents the real image, τ(H gt ) represents the tone mapping loss function of the real image, τ(H out ) represents the tone mapping loss function of the target fused image.

[0042] In a second aspect, an embodiment of the present application provides a ghost-free high dynamic range image fusion system, comprising:

[0043] A first processing module: configured to obtain an input image of the low exposure image and an input image of the high exposure image by gamma correction based on the low exposure image, the medium exposure image and the high exposure image of the original image, wherein the input image of the low exposure image and the input image of the high exposure image both include an LDR domain input image and an HDR domain input image;

[0044] A second processing module: configured to obtain a first dual-domain image of the low-exposure image and a first dual-domain image of the high-exposure image by adopting a dual-domain separation and cooperation method based on the input image of the low-exposure image and the input image of the high-exposure image;

[0045] A third processing module: used for splicing the LDR domain image and the HDR domain image of the medium exposure image to obtain a third dual-domain image of the medium exposure image;

[0046] The fourth processing module is used to fuse the first dual-domain image of the low-exposure image, the first dual-domain image of the high-exposure image, and the third dual-domain image of the medium-exposure image to obtain a target fused image.

[0047] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0048] In the present invention, a dual-domain separate collaboration method is adopted to model the high-exposure image and the low-exposure image in the LDR domain and the HDR domain respectively by combining a dual-channel attention module of spatial self-attention and channel self-attention, so that the features of the high-exposure image and the low-exposure image in the LDR domain and the features of the image in the HDR domain can complement each other, thereby improving the alignment effect of the LDR domain and the HDR domain, and can effectively deal with the situation where ghosting is caused by the simultaneous existence of noise and saturation areas during the movement of the object, thereby generating a high-quality target fusion image without ghosting. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 It is a flowchart of a method for fusion of high dynamic range images without ghosting provided by one embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of the process of the dual-domain separation and cooperation method of the present invention;

[0052] Figure 3 It is a schematic diagram of the structure of the dual-domain alignment network of the present invention;

[0053] Figure 4 It is a structural schematic diagram of a ghost-free high dynamic range image fusion system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0055] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0056] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0057] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0058] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0059] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0060] See also Figure 1 As shown, the present invention is a ghost-free high dynamic range image fusion method, comprising the following steps:

[0061] S100, based on a low-exposure image, a medium-exposure image, and a high-exposure image of an original image, obtaining an input image of the low-exposure image and an input image of the high-exposure image by gamma correction, wherein the input image of the low-exposure image and the input image of the high-exposure image both include an LDR domain input image and an HDR domain input image;

[0062] In this embodiment, before performing dual-domain separate collaboration on the low-exposure image and the high-exposure image, the low-exposure image needs to be aligned in the LDR domain and the HDR domain respectively. Similarly, the high-exposure image needs to be aligned in the LDR domain and the HDR domain respectively, so that the detail information of the saturated or noisy area in the LDR domain and the area occluded by object motion in the HDR domain can be restored simultaneously.

[0063] In some embodiments, the above step S100 includes:

[0064] Obtain a low-exposure image, a medium-exposure image, and a high-exposure image of the original image;

[0065] Taking the medium exposure image as a reference image, adopting gamma correction, mapping the low exposure image and the high exposure image to the HDR domain in sequence, obtaining a second dual-domain image of the low exposure image and a second dual-domain image of the high exposure image, wherein the second dual-domain image of the low exposure image and the second dual-domain image of the high exposure image both include an LDR domain image and an HDR domain image;

[0066] Based on the second dual-domain image of the low-exposure image and the second dual-domain image of the high-exposure image, an input image of the low-exposure image and an input image of the high-exposure image are obtained, and the input image of the low-exposure image and the input image of the high-exposure image both include an LDR domain input image and an HDR domain input image.

[0067] In this embodiment, different exposure parameters are used to obtain a low-exposure image, a medium-exposure image, and a high-exposure image according to the original image, and the medium-exposure image is used as a reference image. Specifically, Figure 2 As shown, based on the medium exposure image as a reference image, a gamma correction algorithm is used to perform nonlinear mapping on the pixel values ​​of the low exposure image and the high exposure image to obtain the mapped low exposure HDR domain image and the high exposure HDR domain image. The second dual-domain image of the low exposure image is obtained by combining the HDR domain image of the mapped low exposure image with the LDR domain image of the original low exposure image; similarly, the second dual-domain image of the high exposure image is obtained by combining the HDR domain image of the mapped high exposure image with the LDR domain image of the original high exposure image.

[0068] In this embodiment, the second dual-domain image includes an LDR domain image and an HDR domain image corresponding to the high-exposure image or the low-exposure image, wherein the LDR domain image retains the low dynamic range details of the original image, and the HDR domain image expands the dynamic range of the image.

[0069] In this embodiment, according to the nonlinear mapping relationship of gamma correction, the pixel value distribution of the low-exposure image and the high-exposure image in the HDR domain is determined to obtain an HDR domain image that is more in line with the visual perception of the human eye. By constructing a second dual-domain image including the LDR domain and the HDR domain, an image representation that simultaneously retains the low dynamic range details and the high dynamic range details is obtained, providing richer image information for subsequent image fusion processing.

[0070] Specifically, let L 1 , L 2 , L 3They represent low-exposure images, medium-exposure images, and over-exposure images respectively. The low-exposure images, medium-exposure images, and high-exposure images are mapped to the HDR domain respectively. The calculation formulas for the low-exposure images, medium-exposure images, and high-exposure images in the HDR domain are as follows: Among them, t i is the exposure time corresponding to each exposure image, and γ is the parameter of gamma correction, which is set to 2.2.

[0071] In some embodiments, obtaining the input image of the low exposure image and the input image of the high exposure image based on the second dual-domain image of the low exposure image and the second dual-domain image of the high exposure image includes:

[0072] splicing the LDR image of the low exposure image and the LDR image of the high exposure image with the LDR image of the medium exposure image respectively to obtain an LDR domain input image of the low exposure image and an LDR domain input image of the high exposure image;

[0073] The HDR image of the low exposure image and the HDR image of the high exposure image are respectively spliced ​​with the HDR image of the medium exposure image to obtain an HDR domain input image of the low exposure image and an HDR domain input image of the high exposure image.

[0074] In this embodiment, the LDR domain image and the HDR domain image obtained after gamma correction are converted into LDR domain input images and HDR domain input images respectively, and then the input image of the low exposure image and the input image of the high exposure image are respectively input into the dual-domain alignment network for alignment.

[0075] In this embodiment, for the LDR domain, based on the LDR image of the low exposure image and the LDR image of the medium exposure image, an image stitching algorithm is used to stitch the LDR image of the low exposure image with the LDR image of the medium exposure image to obtain an LDR domain input image of the low exposure image; based on the LDR image of the high exposure image and the LDR image of the medium exposure image, an image stitching algorithm is used to stitch the LDR image of the high exposure image with the LDR image of the medium exposure image to obtain an LDR domain input image of the high exposure image; through the LDR domain input image of the low exposure image, scene detail information in the low exposure image can be obtained; through the LDR domain input image of the high exposure image, scene texture information in the high exposure image can be obtained; based on the scene detail information of the low exposure image and the scene texture information of the high exposure image, a high dynamic range image containing rich scene details and texture information can be obtained when image fusion is performed subsequently.

[0076] Similarly, in this embodiment, for the HDR domain, by splicing the HDR image of the low-exposure image with the HDR image of the medium-exposure image, an HDR domain input image of the low-exposure image in the HDR domain is obtained, and the image integrates the image details under the low-exposure and medium-exposure environments; by splicing the HDR image of the high-exposure image with the HDR image of the medium-exposure image, an HDR domain input image of the high-exposure image in the HDR domain is obtained, and the image integrates the image details under the high-exposure and medium-exposure environments. The HDR domain input image of the low-exposure image and the HDR domain input image of the high-exposure image are used as inputs for subsequent processing, which can be used for tasks such as image enhancement and color correction to improve the quality of images taken under different exposure environments.

[0077] S200, based on the input image of the low-exposure image and the input image of the high-exposure image, adopting a dual-domain separation and cooperation method to obtain a first dual-domain image of the low-exposure image and a first dual-domain image of the high-exposure image;

[0078] In some of the embodiments, the dual-domain separate collaboration method includes: inputting the input image of the low-exposure image and the input image of the high-exposure image into a dual-domain alignment network respectively to obtain a first dual-domain image of the low-exposure image and a first dual-domain image of the high-exposure image.

[0079] In this embodiment, the low-exposure image and the high-exposure image are separately collaboratively aligned in the LDR domain and the HDR domain and then fused, so that the features of the generated LDR domain output image and the HDR domain output image complement each other, which can improve the alignment effect, thereby reducing the saturation area and noise in the LDR domain and improving the alignment degree of the characteristics of the HDR domain.

[0080] In this embodiment, the LDR domain input image and HDR domain input image of the low exposure image and the LDR domain input image and HDR domain input image of the high exposure image are respectively input into the dual-domain alignment network, so that the alignment of the low exposure image and the high exposure image in the LDR domain and HDR domain can be performed separately and simultaneously, thereby the dual-domain alignment of the low exposure image and the dual-domain alignment of the high exposure image do not interfere with each other, and the alignment efficiency of the low exposure image and the high exposure image is improved.

[0081] In some embodiments, the dual-domain separate collaboration method further includes:

[0082] Based on the LDR domain input image and the HDR domain input image, a layer normalization method is adopted to obtain an LDR domain feature map and an HDR domain feature map;

[0083] Inputting the LDR domain feature map and the HDR domain feature map into a dual-channel attention module respectively to obtain a comprehensive attention feature map, wherein the dual-channel attention module includes spatial self-attention and channel self-attention;

[0084] The comprehensive attention feature map is input into a feedforward neural network to obtain a first dual-domain image, wherein the first dual-domain image includes an LDR domain output image and an HDR domain output image.

[0085] In this embodiment, in order to better complete the alignment and fusion tasks between low-exposure images and high-exposure images, a spatial plus channel self-attention module is set in the dual-domain alignment network to focus on the alignment tasks of the HDR domain and the LDR domain, respectively, and on this basis, dual-domain collaboration is performed to perform the alignment tasks of low-exposure images and high-exposure images, respectively.

[0086] like Figure 3 As shown in the figure, the dual-domain alignment network includes layer normalization (Layer Norm), spatial self-attention (SpatialSelf-Attention), channel self-attention (Channel Self-Attention) and feed-forward neural network (FFN). The LDR domain input image and HDR domain input image are processed by layer normalization, the mean and variance of each feature layer in the LDR domain input image and the HDR domain input image are calculated, and the pixel values ​​of the two are normalized to the same scale range to eliminate the differences in brightness and contrast, so as to obtain the normalized LDR domain feature map and HDR domain feature map. According to the normalized LDR domain feature map and HDR domain feature map, the spatial self-attention mechanism is adopted to extract the spatial features of the image through the convolutional neural network, capture the local information such as texture and edge in the image, and obtain the spatial attention feature map. Then, the channel attention mechanism is adopted. Based on the spatial attention feature map, the weights of different channels are calculated through global average pooling and fully connected layers, the salient area is highlighted, the background interference is suppressed, and the channel attention feature map is obtained. The spatial attention feature map and the channel attention feature map are multiplied to fuse the attention information of the two dimensions of space and channel to obtain a comprehensive attention feature map, highlighting the key areas and significant features in the image, and realizing the adaptive extraction and fusion of cross-domain features. The comprehensive attention feature map is input into the feedforward neural network, and the mapping relationship from the LDR domain to the HDR domain is learned through multiple layers of fully connected layers and nonlinear activation functions to capture the laws of dynamic range changes and detail enhancement, and obtain the prediction results in the HDR domain. According to the predicted HDR domain results, the pixel values ​​are restored to the original HDR scale range and LDR scale range through denormalization processing to obtain the HDR domain output image and the LDR domain output image, and then the first dual-domain image is obtained to realize the dynamic range mapping and detail enhancement from the LDR domain to the HDR domain.

[0087] S300, stitching the LDR domain image and the HDR domain image of the middle exposure image to obtain a third dual-domain image of the middle exposure image;

[0088] In this embodiment, while adopting the gamma correction method to obtain the second dual-domain image of the low exposure image and the second dual-domain image of the high exposure image, the second dual-domain image of the medium exposure image is synchronously obtained, that is, the LDR domain image and the HDR domain image of the medium exposure image are synchronously obtained, and the LDR domain image and the HDR domain image are spliced ​​by using an image stitching algorithm to obtain a third dual-domain image of the medium exposure image. Specifically, by analyzing the pixel value distribution characteristics of the LDR domain image and the HDR domain image of the medium exposure image, the overlapping area of ​​the LDR domain image and the HDR domain image is determined, and the pixel value information of the overlapping area is obtained. According to the pixel value information of the overlapping area, a weighted average algorithm is used to perform weighted average calculation on the pixel values ​​of the LDR domain image and the HDR domain image to obtain the spliced ​​pixel value of the overlapping area. By analyzing the edge features of the LDR domain image and the HDR domain image, the edge position information of the LDR domain image and the HDR domain image is obtained, and the position of the splicing boundary line is determined. According to the position of the splicing boundary line, an image fusion algorithm is used to seamlessly splice the LDR domain image and the HDR domain image to eliminate the splicing traces, and obtain a third dual-domain image of the medium exposure image with a visually natural transition. By evaluating the image quality of the third dual-domain image, including indicators such as clarity, contrast, and color reproduction, it is determined whether the stitching effect meets the preset quality requirements. According to the quality evaluation results, the image enhancement algorithm is used to enhance the local area of ​​the third dual-domain image to improve the visual effect of the image, and finally a high-quality middle-exposure third dual-domain image is obtained.

[0089] S400: Fusing the first dual-domain image of the low-exposure image, the first dual-domain image of the high-exposure image, and the third dual-domain image of the medium-exposure image to obtain a target fused image.

[0090] In this embodiment, a first dual-domain image is obtained by performing dual-domain separation and collaboration of a low-exposure image and a high-exposure image, and then the third dual-domain image is fused with the first dual-domain image, so as to effectively process the scene where object motion and noise or saturated areas coexist, thereby generating a high-quality target fused image without ghosting. In addition, the low-exposure image retains more dark details, the high-exposure image shows more bright details, and the medium-exposure image is located between the two, providing details of intermediate brightness. By fusing these images, it can be ensured that the target fused image contains all important details in the scene, thereby generating a target fused image with richer information and more complete details. At the same time, during the fusion process, the noise and artifacts in each original image can be removed by an algorithm, so that the visual effect of the target fused image is clearer and purer.

[0091] In some embodiments, the above-mentioned ghost-free high dynamic range image fusion method further includes:

[0092] The target fused image is mapped from the linear domain to the tone mapping domain using a tone mapping loss function, and the loss function of the target fused image is calculated. The loss function includes a dual-domain collaboration loss function, a dual-domain alignment loss function, and an image fusion loss function. The calculation formula of the tone mapping loss function is:

[0093]

[0094] Where x is the input function, u is a fixed value of 5000, and τ(x) represents the tone mapping loss function;

[0095] The calculation formula of the loss function is:

[0096] L=L collaboration +loss aligment +L fusion

[0097] Among them, L collaboration represents the dual-domain collaborative loss function, loss aligment represents the dual domain alignment loss function, L fusion represents the image fusion loss function, and L represents the loss function.

[0098] In this embodiment, in order to further improve the quality and visual effect of the fused image, a tone mapping loss function is introduced so that the fused target fused image maintains a high degree of consistency and information integrity in content, while achieving a natural and harmonious effect in color expression.

[0099] Specifically, the method using the tone mapping loss function first converts the target fused image from the original linear domain to the tone mapping domain. The tone mapping domain is a specially processed color space designed to simulate the way the human eye perceives brightness and color, so that the image on the display device is closer to the visual effect of the real world. This conversion is usually achieved by applying a tone mapping algorithm, such as local tone mapping, global tone mapping, or complex models based on visual perception.

[0100] In addition, in order to evaluate and optimize the quality of the fused image, the loss of the target fused image in the tone mapping domain is calculated in this embodiment, so that the quality of the fused image can be improved from different dimensions during the formation of the target fused image. This is an efficient and effective image fusion strategy that can significantly improve the quality and visual effect of the fused image.

[0101] In some of these embodiments, the dual-domain collaborative loss function calculates the loss after the first dual-domain image is obtained using the dual-domain separate collaborative method;

[0102] The calculation formula of the dual-domain collaboration loss function is as follows:

[0103]

[0104] Among them, ψ(.) is the feature map output by the pre-trained VGG-16 network, i and j represent the jth feature map after the i-th pooling layer, and L 1 For low exposure images, L 3 For high exposure images, and are respectively the LDR domain image and the HDR domain image of the first dual domain image of the low exposure image, and are respectively the LDR domain image and the HDR domain image of the first dual domain image of the high exposure image, The tone mapping loss function of the HDR domain image representing the first dual domain image of the low exposure image, Tone mapping loss function of the HDR domain image representing the first dual domain image of the high exposure image.

[0105] In this embodiment, when calculating the loss function, the loss between the HDR domain and the LDR domain in the first dual-domain image obtained after adopting the dual-domain separation and cooperation method is considered. Specifically, the low-exposure image and the high-exposure image are input into the pre-trained VGG-16 network to obtain the corresponding feature map in each layer, and then each feature map is converted from the original linear domain to the tone mapping domain, and the loss of the low-exposure image and the high-exposure image in the HDR domain and the LDR domain is calculated respectively, thereby ensuring that when performing image fusion, the key features in the HDR domain and the LDR domain can be coordinated with each other, which not only maintains the integrity of the original data, but also conforms to the human eye's perception habits of color and brightness, and at the same time enables the fused image to achieve a higher quality level in both the HDR domain and the LDR domain.

[0106] In some of these embodiments, the dual-domain alignment loss function calculates a loss between an LDR domain image and an HDR domain image in a second dual-domain image;

[0107] The calculation formula of the dual-domain alignment loss function is as follows:

[0108] loss aligment =loss 1 +loss 2 +loss 3 +loss 4

[0109] Among them, loss 1 Represents the loss of the LDR domain image of the low exposure image, loss 2 Represents the loss of the HDR domain image of the low-exposure image, loss 3 Represents the loss of the LDR domain image of the high exposure image, loss 4 Represents the loss of HDR domain images for high exposure images.

[0110] In this embodiment, when calculating the loss function, the loss between the HDR domain and the LDR domain in the second dual-domain image obtained after the gamma correction method is adopted is considered. Specifically, the loss of the low-exposure image in the LDR domain and the HDR domain and the loss of the high-exposure image in the LDR domain and the HDR domain are calculated respectively, and the four losses are added together to obtain the loss of dual-domain alignment. Specifically, loss 1 、loss 2 、loss 3 And loss 4 The calculation method of is similar to that of the L1 loss in the image fusion loss function, which is obtained by converting the low-exposure images in the LDR domain and mapped to the HDR domain from the original linear domain to the tone mapping domain, and converting the high-exposure images in the LDR domain and mapped to the HDR domain from the original linear domain to the tone mapping domain.

[0111] In some embodiments, the image fusion loss function calculates the loss between the target fused image and the real image;

[0112] The calculation formula of the image fusion loss function is as follows:

[0113] L fusion =||τ(H gt )-τ(H out )|| 1 +λ||ψ i,j (τ(H gt )-ψ i,j (τ(H out )))|| 1

[0114] Among them, ψ(.) is the feature map output by the pre-trained VGG-16 network, i and j represent the jth feature map after the i-th pooling layer, λ is 0.01, H out represents the target fusion image, H gt represents the real image, τ(H gt ) represents the tone mapping loss function of the real image, τ(H out ) represents the tone mapping loss function of the target fused image.

[0115] In this embodiment, when calculating the loss function, the loss between the target fused image and the real image when the target fused image itself is converted from the original linear domain to the tone mapping domain is considered. Specifically, in order to make the features between the LDR domain and the HDR domain complementary, the target fused image is input into the pre-trained VGG-16 network to obtain the corresponding feature map in each layer of the VGG-16 network, and then each feature map is converted from the original linear domain to the tone mapping domain to obtain the perceptual loss of the target fused image, perceptual loss = λ||ψ i，j (τ(H gt )-ψ i，j (τ(H out )))||, In addition, the image fusion loss function also includes LI loss. L1 loss is the loss between the target fusion function and the real image when the target fusion image and the real image are converted from the original linear domain to the tone mapping domain respectively. L1 loss = ||τ(H gt )-τ(H out )||.

[0116] See also Figure 4 As shown, the present invention also provides a ghost-free high dynamic range image fusion system, the system comprising:

[0117] The first processing module 401 is used to obtain an input image of the low exposure image and an input image of the high exposure image by gamma correction based on the low exposure image, the medium exposure image and the high exposure image of the original image, wherein the input image of the low exposure image and the input image of the high exposure image both include an LDR domain input image and an HDR domain input image;

[0118] The second processing module 402 is configured to obtain a first dual-domain image of the low-exposure image and a first dual-domain image of the high-exposure image by adopting a dual-domain separation and cooperation method based on the input image of the low-exposure image and the input image of the high-exposure image;

[0119] The third processing module 403 is used to stitch the LDR domain image and the HDR domain image of the medium exposure image to obtain a third dual-domain image of the medium exposure image;

[0120] The fourth processing module 404 is configured to fuse the first dual-domain image of the low-exposure image, the first dual-domain image of the high-exposure image, and the third dual-domain image of the medium-exposure image to obtain a target fused image.

[0121] It is understandable that if Figure 1 The contents of the embodiment of the ghost-free high dynamic range image fusion method shown in the figure are all applicable to the embodiment of the ghost-free high dynamic range image fusion system. The functions specifically implemented by the embodiment of the ghost-free high dynamic range image fusion system are similar to those in the embodiment of the figure. Figure 1 The ghost-free high dynamic range image fusion method embodiment shown is the same as that shown in FIG. Figure 1 The beneficial effects achieved by the ghost-free high dynamic range image fusion method embodiment shown are also the same.

[0122] It should be noted that the information interaction, execution process and other contents between the above-mentioned systems are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0123] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0124] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A ghost-free high dynamic range image fusion method, characterized in that: include: Based on a low-exposure image, a medium-exposure image, and a high-exposure image of the original image, gamma correction is performed to obtain an input image of the low-exposure image and an input image of the high-exposure image, wherein the input image of the low-exposure image and the input image of the high-exposure image both include an LDR domain input image and an HDR domain input image; Based on the input image of the low-exposure image and the input image of the high-exposure image, a dual-domain separation and cooperation method is adopted to obtain a first dual-domain image of the low-exposure image and a first dual-domain image of the high-exposure image; splicing the LDR domain image and the HDR domain image of the middle exposure image to obtain a third dual-domain image of the middle exposure image; The first dual-domain image of the low-exposure image, the first dual-domain image of the high-exposure image, and the third dual-domain image of the medium-exposure image are fused to obtain a target fused image.

2. The method according to claim 1, characterized in that The method of obtaining an input image of the low-exposure image and an input image of the high-exposure image by gamma correction based on the low-exposure image, the medium-exposure image, and the high-exposure image of the original image includes: Obtain a low-exposure image, a medium-exposure image, and a high-exposure image of the original image; Taking the medium exposure image as a reference image, using gamma correction, mapping the low exposure image and the high exposure image to the HDR domain in sequence, obtaining a second dual-domain image of the low exposure image and a second dual-domain image of the high exposure image, wherein the second dual-domain image of the low exposure image and the second dual-domain image of the high exposure image both include an LDR domain image and an HDR domain image; Based on the second dual-domain image of the low-exposure image and the second dual-domain image of the high-exposure image, an input image of the low-exposure image and an input image of the high-exposure image are obtained, and the input image of the low-exposure image and the input image of the high-exposure image both include an LDR domain input image and an HDR domain input image.

3. The method according to claim 2, characterized in that The obtaining, based on the second dual-domain image of the low-exposure image and the second dual-domain image of the high-exposure image, the input image of the low-exposure image and the input image of the high-exposure image comprises: splicing the LDR image of the low exposure image and the LDR image of the high exposure image with the LDR image of the medium exposure image respectively to obtain an LDR domain input image of the low exposure image and an LDR domain input image of the high exposure image; The HDR image of the low exposure image and the HDR image of the high exposure image are respectively spliced ​​with the HDR image of the medium exposure image to obtain an HDR domain input image of the low exposure image and an HDR domain input image of the high exposure image.

4. The method according to claim 3, characterized in that The dual-domain separate collaboration method includes: inputting the input image of the low-exposure image and the input image of the high-exposure image into a dual-domain alignment network respectively to obtain a first dual-domain image of the low-exposure image and a first dual-domain image of the high-exposure image.

5. The method according to claim 4, characterized in that The dual-domain separate collaboration method further includes: Based on the LDR domain input image and the HDR domain input image, a layer normalization method is adopted to obtain the LDR domain feature map and the HDR domain feature map; Inputting the LDR domain feature map and the HDR domain feature map into a dual-channel attention module respectively to obtain a comprehensive attention feature map, wherein the dual-channel attention module includes spatial self-attention and channel self-attention; The comprehensive attention feature map is input into a feedforward neural network to obtain a first dual-domain image, wherein the first dual-domain image includes an LDR domain output image and an HDR domain output image.

6. The method according to claim 2, characterized in that Also includes; The target fused image is mapped from the linear domain to the tone mapping domain using a tone mapping loss function, and the loss function of the target fused image is calculated. The loss function includes a dual-domain collaboration loss function, a dual-domain alignment loss function, and an image fusion loss function. The calculation formula of the tone mapping loss function is: Where x is the input function, u is a fixed value of 5000, and τ(x) represents the tone mapping loss function; The calculation formula of the loss function is: L=L collaboration +loss aligmen t +L fusion Among them, L collaboration represents the dual-domain collaborative loss function, loss aligmen t represents the dual domain alignment loss function, L fusion represents the image fusion loss function, and L represents the loss function.

7. The method according to claim 6, characterized in that The dual-domain collaborative loss function calculates the loss after the first dual-domain image is obtained by using the dual-domain separate collaborative method; The calculation formula of the dual-domain collaboration loss function is as follows: Among them, ψ(.) is the feature map output by the pre-trained VGG-16 network, i and j represent the jth feature map after the i-th pooling layer, L1 is the low-exposure image, and L3 is the high-exposure image. and are respectively the LDR domain image and the HDR domain image of the first dual domain image of the low exposure image, and are the LDR domain image and HDR domain image of the first dual domain image of the high exposure image, respectively, τ The tone mapping loss function of the HDR domain image representing the first dual domain image of the low exposure image, Tone mapping loss function for the HDR domain image of the first dual domain image representing the high exposure image.

8. The method according to claim 6, characterized in that The dual-domain alignment loss function calculates the loss between the LDR domain image and the HDR domain image in the second dual-domain image; The calculation formula of the dual-domain alignment loss function is as follows: loss alignment =loss1+loss2+loss3+loss4; Among them, loss1 represents the loss of the LDR domain image of the low exposure image, loss2 represents the loss of the HDR domain image of the low exposure image, loss3 represents the loss of the LDR domain image of the high exposure image, and loss4 represents the loss of the HDR domain image of the high exposure image.

9. The method according to claim 6, characterized in that The image fusion loss function calculates the loss between the target fused image and the real image; The calculation formula of the image fusion loss function is as follows: L fusion =||τ(H gt )-τ(H out )||1+λ||ψ i,j (t(H gt )-ψ i,j (t(H out )))||1 Among them, ψ(.) is the feature map output by the pre-trained VGG-16 network, i and j represent the jth feature map after the i-th pooling layer, λ is 0.01, H out represents the target fusion image, H gt represents the real image, τ(H gt ) represents the tone mapping loss function of the real image, τ(H out ) represents the tone mapping loss function of the target fused image.

10. A ghost-free high dynamic range image fusion system, characterized in that: include: A first processing module: configured to obtain an input image of the low exposure image and an input image of the high exposure image by gamma correction based on the low exposure image, the medium exposure image and the high exposure image of the original image, wherein the input image of the low exposure image and the input image of the high exposure image both include an LDR domain input image and an HDR domain input image; A second processing module: configured to obtain a first dual-domain image of the low-exposure image and a first dual-domain image of the high-exposure image by adopting a dual-domain separation and cooperation method based on the input image of the low-exposure image and the input image of the high-exposure image; A third processing module: used for splicing the LDR domain image and the HDR domain image of the medium exposure image to obtain a third dual-domain image of the medium exposure image; The fourth processing module is used to fuse the first dual-domain image of the low-exposure image, the first dual-domain image of the high-exposure image, and the third dual-domain image of the medium-exposure image to obtain a target fused image.

Images