High dynamic range imaging method and device

Abstract

Disclosed are a high dynamic range imaging method and apparatus that minimize color distortion, preserve texture, and reduce ghosting. 【Solution means】The high dynamic range imaging method of the present invention includes: (a) obtaining a first Low Dynamic Range (LDR) image, a second LDR image, and a third LDR image each having a different exposure value; (b) applying the first, second, and third LDR images to a weighted value map estimation model to extract a weighted value map, and using the extracted weighted value map to generate an input feature map by integrating the first, second, and third LDR images; and (c) applying the input feature map to a Swin Fourier convolution network model for Fourier transform, and after convolution, connecting a reference feature map to generate a High Dynamic Range (HDR) image.

Description

[Technical Field]

[0001] The present invention relates to a high dynamic range imaging method and apparatus. [Background technology]

[0002] The dynamic range of a digital image indicates the range of measurements that can be displayed by the physical sensor in the imaging system. However, if the exposure time is insufficient or excessive, the captured image will have a low dynamic range (LDR) due to the functional limitations of the physical sensor, resulting in a poor subjective image quality. To address this issue, many high dynamic range imaging methods have been researched.

[0003] Conventional imaging systems have some HDR capabilities built in. Motion-sensing based methods utilize weight maps, threshold bitmaps, and similar techniques to synthesize an HDR image from an LDR image with multiple exposure values ​​for the motion occurring in the image.

[0004] Alignment-based methods align LDR images using methods such as twisting around the exposure value image to extend the dynamic range.

[0005] However, although conventional techniques have been successful in obtaining HDR images, they still have problems such as motion and color distortion that occur in LDR images. Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been made in consideration of the above-mentioned conventional problems, and an object of the present invention is to provide a high dynamic range imaging method and apparatus that minimizes color distortion that occurs when converting an LDR image to an HDR image, preserves texture, and reduces ghosting. [Means for solving the problem]

[0007] In order to achieve the above object, one aspect of the present invention provides a high dynamic range (HDR) imaging method, which includes the steps of: (a) acquiring a first LDR (Low Dynamic Range) image, a second LDR image, and a third LDR image, each having different exposure values; (b) applying the first LDR image, the second LDR image, and the third LDR image to a weighting map estimation model to extract a weighting map, and generating an input feature map by integrating the first LDR image, the second LDR image, and the third LDR image using the extracted weighting map; and (c) applying the input feature map to a Sween-Fourier convolutional network model to Fourier transform the input feature map, convolve it, and then connect a reference feature map to generate a high dynamic range (HDR) image.

[0008] The reference feature map may be a feature map of the second LDR image extracted by the weight map estimation model. The step (b) may include applying the first LDR image, the second LDR image, and the third LDR image to a convolutional layer of the weight map estimation model to extract a first feature map, a second feature map, and a third feature map, respectively; applying the first feature map and the second feature map to a first attention module of the weight map estimation model to generate a first weight map, and applying the third feature map and the second feature map to a second attention module of the weight map estimation model to generate a second weight map; and reflecting the first weight map and the second weight map in the first feature map and the third feature map, respectively, and then combining them with the second feature map to generate the input feature map. The Swinn-Fourier convolution network model may include a plurality of Swinn-Fourier convolution blocks and a transpose convolution block, and the Swinn-Fourier convolution block may include: a Swinn transformer block that divides the input feature map into patch units to construct a hierarchical feature map; a residual block that adds a global residual to the feature map of a reference image using a jump connection; a first 3×3 convolution layer and a second 3×3 convolution layer connected to a rear end of the Swinn-Fourier convolution block, and configured to combine output values ​​of the first 3×3 convolution layer and the second 3×3 convolution layer element by element to output a first sub-synthesis result value; a third 3×3 convolution layer connected to a rear end of the residual block, and a connection unit that combines output values ​​of a spectral transform module that performs a Fourier transform on output values ​​of the residual block element by element to output a second sub-synthesis result value; and a connection unit that connects the first sub-synthesis result value and the second sub-synthesis result value. The loss function of the Swin-Fourier convolutional network model can be calculated as follows:

number

number

number

number

[0009] In order to achieve the above object, according to one aspect of the present invention, a high dynamic range (HDR) imaging device includes: an image acquisition unit that acquires a first LDR (Low Dynamic Range) image, a second LDR image, and a third LDR image, each having different exposure values; a weight map estimation unit that applies the first LDR image, the second LDR image, and the third LDR image to a weight map estimation model to extract a weight map, and generates an input feature map that integrates the first LDR image, the second LDR image, and the third LDR image using the extracted weight map; and an HDR generation unit that applies the input feature map to a Sween-Fourier convolutional network model to perform a Fourier transform, convolve, and then connect a reference feature map to generate an HDR (High Dynamic Range) image.

[0010] The weight map estimation unit may apply the first LDR image, the second LDR image, and the third LDR image to a convolutional layer of the weight map estimation model to extract a first feature map, a second feature map, and a third feature map, respectively; apply the first feature map and the second feature map to a first attention module of the weight map estimation model to generate a first weight map; apply the third feature map and the second feature map to a second attention module of the weight map estimation model to generate a second weight map; and reflect the first weight map and the second weight map in the first feature map and the third feature map, respectively, and then combine them with the second feature map to generate the input feature map. [Effects of the Invention]

[0011] According to the high dynamic range synthesis method and apparatus of the present invention, it is possible to minimize color distortion that occurs when converting an LDR image into an HDR image, and to obtain an HDR image that preserves texture and is free of ghosting. [Brief explanation of the drawings]

[0012] [Figure 1]1 is a flowchart illustrating a high dynamic range imaging method according to one embodiment of the present invention. [Figure 2] FIG. 1 is a diagram illustrating a network architecture of an HDR imaging device according to an embodiment of the present invention. [Figure 3] FIG. 1 illustrates the structure of a Swin-Fourier convolution model according to one embodiment of the present invention. [Figure 4] FIG. 1 illustrates the structure of a Swin transformer block according to one embodiment of the present invention. [Figure 5] FIG. 2 illustrates a structure of a residual block according to an embodiment of the present invention. [Figure 6] FIG. 2 illustrates the structure of a spectral transformation block according to an embodiment of the present invention. [Figure 7] FIG. 10 illustrates weight map results according to one embodiment of the present invention. [Figure 8] 1A and 1B are diagrams illustrating activation results with perceptual loss according to the prior art and an embodiment of the present invention. [Figure 9] FIG. 1 is a diagram schematically illustrating the internal configuration of an HDR imaging device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] As used herein, singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "include" should not be interpreted as necessarily including all of the components or steps described in the specification, but may include some of the components or steps, or may include additional components or steps. Furthermore, terms such as "unit," "module," etc. used in the specification refer to a unit that processes at least one function or operation, and may be implemented in hardware or software, or a combination of hardware and software.

[0014] Hereinafter, specific examples of embodiments of the present invention will be described in detail with reference to the drawings.

[0015] FIG. 1 is a flowchart illustrating a high dynamic range imaging method according to an embodiment of the present invention, FIG. 2 is a diagram illustrating the network architecture of an HDR imaging device according to an embodiment of the present invention, FIG. 3 is a diagram illustrating the structure of a Swinn Fourier convolution model according to an embodiment of the present invention, FIG. 4 is a diagram illustrating the structure of a Swinn transformer block according to an embodiment of the present invention, FIG. 5 is a diagram illustrating the structure of a residual block according to an embodiment of the present invention, FIG. 6 is a diagram illustrating the structure of a spectral transformation block according to an embodiment of the present invention, FIG. 7 is a diagram illustrating a weight map result according to an embodiment of the present invention, and FIG. 8 is a diagram illustrating activation results due to perceptual loss in the conventional method and an embodiment of the present invention.

[0016] In step 110, the HDR imaging apparatus 100 captures multiple LDR images each having a different exposure value.

[0017] For example, the HDR imaging device 100 acquires a first LDR image, a second LDR image, and a third LDR image, each having a different exposure value. Here, it is assumed that the first LDR image, the second LDR image, and the third LDR image are consecutive images having different exposure values. For convenience, the following description will be centered on the assumption that the first LDR image has the smallest exposure value and the third LDR image has the largest exposure value.

[0018] For example, to obtain an HDR image, an LDR image is input as i which means images with different exposures.

[0019] Geometrically aligned images The intensity transformed version of TIFF0007795828000005.tif10151 is obtained using gamma correction as in Equation 1.

[0020]

number

[0021] where L i denotes an LDR image, TIFF0007795828000007.tif11151 shows the HDR image, and γ shows the gamma parameter, which in one embodiment of the present invention was set to 2.2. i indicates the exposure time.

[0022] TIFF0007795828000008.tif10128 is obtained by linearizing the nonlinear LDR input image using the camera response function and applying gamma correction. i and Define TIFF0007795828000009.tif10128 as the input image Gi, and obtain the HDR image as the connected LDR input image. This can be expressed mathematically as Equation 2.

[0023]

number

[0024] The overall network model of the high dynamic range imaging device according to an embodiment of the present invention is defined as shown in Equation 3.

[0025]

number

[0026] where: TIFF0007795828000012.tif10128 shows the output (HDR image) of the entire network model, and θ indicates the learning parameter.

[0027] In step 115, the HDR imaging device 100 generates a weighting map using the first LDR image, the second LDR image, and the third LDR image, and generates an input feature map by integrating the first LDR image, the second LDR image, and the third LDR image using the generated weighting map.

[0028] This will be explained in more detail with reference to FIG.

[0029] The HDR imaging device 100 applies a convolution operation to each of the first LDR image, the second LDR image, and the third LDR image to generate a first feature map, a second feature map, and a third feature map, respectively.

[0030] The HDR imaging device 100 applies the first and second feature maps to an attention module to generate a first weight map, and applies the third and second feature maps to an attention module to generate a second weight map, which is used as a reference feature map.

[0031] This can be expressed mathematically as in Equation 4.

[0032]

number

[0033] where G i denotes the LDR image, H1 denotes the extracted feature map layer, and f i indicates the attention module, TIFF0007795828000014.tif11128 shows the weight map, which has values ​​between [0, 1].

[0034] The HDR imaging apparatus 100 reflects the first weight map and the second weight map in the first feature map and the third feature map, respectively, and then combines them with a reference feature map (second feature map) to generate an input feature map. To minimize ghosting, the HDR imaging apparatus 100 reflects the first weight map in the first feature map and the second weight map in the third feature map, and then combines them with the reference feature map (second feature map) to generate an input feature map.

[0035] That is, the HDR imaging apparatus 100 multiplies the first weight map and the first feature map for each element to reflect the first weight map in the first feature map, and multiplies the second weight map and the third feature map for each element to reflect the second weight map in the third feature map. Then, the HDR imaging apparatus 100 combines the first and third feature maps, on which the weight maps are reflected, with a reference feature map (second feature map) to generate an input feature map.

[0036] As a result, as shown in FIG. 7, in areas where ghost artifacts occur due to object movement different from that in the reference image, the weights are lowered, thereby minimizing undesired artifacts. This is an element-by-element multiplication of the first weight map H1(G1) and the second weight map H1(G2), and is expressed as Equation 5.

[0037]

number

[0038] where F 2 denotes the feature map and channel-wise connections, and C(·) denotes the operator that performs the connections after element-wise multiplication.

[0039] F estimated to minimize ghosting 2 is used as the input feature map for learning an improved perceptual loss function.

[0040] In step 120, the HDR imaging device 100 applies the input feature map to a Sween-Fourier convolution network model to perform a Fourier transform, and after performing a convolution operation, connects the reference feature map to generate an HDR (High Dynamic Range) image.

[0041] As shown in Figure 2, the Swin-Fourier convolution network model consists of multiple Swin-Fourier convolution blocks (SCblocks), residual blocks (RBs), and transpose convolution blocks (TCblocks).

[0042] The input feature map is reconstructed with the HDR image via a U-shaped network model (i.e., a Swin-Fourier convolution network model) composed of a Swin-Fourier convolution block and a transpose convolution block.

[0043] The Swinn-Fourier convolution block is based on a transformer structure and consists of a Swinn-Transformer block that divides the feature map into patch units to construct a hierarchical feature map, and a Residual block that adds the global residual to the feature map of the reference image using skip connections.

[0044] As shown in Figure 4, the Sween Transformer block consists of a layer normalization block, a window-based multi-head self-attention module, a multi-layer perceptron module, and a 1x1 convolution structure after the convolution layer.

[0045] The Sween-Fourier convolution block allows the feature map to be called by the U-shaped encoder module, resulting in a lower resolution, so it is downsampled using 2x2 stride convolution, and the transformer is integrated into the U-shaped network block to further utilize hierarchical information. The residual block is shown in Figure 5.

[0046] The Swinn-Fourier convolutional network model also includes a first 3×3 convolutional layer and a second 3×3 convolutional layer connected to a rear end of the Swinn-Transformer block, and combines output values ​​of the first 3×3 convolutional layer and the second 3×3 convolutional layer element by element to output a first sub-synthesis result value; a third 3×3 convolutional layer connected to a rear end of the residual block and output values ​​of a spectral transform module that performs a Fourier transform on the output values ​​of the residual block element by element to output a second sub-synthesis result value; and a connection unit that connects the first sub-synthesis result value and the second sub-synthesis result value.

[0047] The transposed convolution block used in the decoder module consists of a transposed convolution block and a Sween-Fourier convolution block, and uses 2x2 transposed convolution to upsample and increase the resolution in order to learn global information hierarchically.

[0048] Furthermore, according to one embodiment of the present invention, the Sween-Fourier convolutional network model uses jump connections to add feature maps of reference images as global residuals to aid in learning.

[0049] The spectral transform block connected to the Sween-Fourier convolution block performs convolution after FFT (Fast-Fourier Transform), and its structure is shown in Figure 6. Because the spectral transform block uses FFT, it is easy to process periodic patterns expressed in images, and when used together with the Sween-Fourier convolution block, it can enhance not only local information but also global context. The FFT converts the image signal into a periodic frequency signal, which allows the system to learn the periodic patterns in the image.

[0050] According to one embodiment of the present invention, the Sween-Fourier convolutional network model is trained by adding a second feature map for a reference image (i.e., a second LDR image) as a global residual using global jump connections.

[0051] A Swin-Fourier convolutional network model according to one embodiment of the present invention is trained using a log-perceptual loss function.

[0052] This can be expressed mathematically as in Equation 6.

[0053]

number

[0054] where x denotes the target HDR image, y denotes the generated HDR image,

number

[0055]

number

[0056] Here, μ denotes a compression parameter.

[0057] Therefore, the total loss function is expressed as in Equation 8.

[0058]

number

[0059] Here, l1 denotes the MAE loss, and VGG19() denotes the Sween-Fourier convolutional network model as the convolutional neural network.

[0060] As shown in FIG. 8, by applying the log perceptual loss function according to an embodiment of the present invention, activation is increased compared to the conventional method, so that areas with low values ​​have high values ​​and areas with high values ​​maintain their high values.

[0061] FIG. 9 is a diagram schematically illustrating the internal configuration of an HDR imaging device according to an embodiment of the present invention.

[0062] Referring to FIG. 9, the HDR imaging device 100 according to an embodiment of the present invention includes an image acquisition unit 910, a weight map estimation unit 920, an HDR generation unit 930, a memory 940, and a processor 950.

[0063] The image acquisition unit 910 is a means for acquiring a first LDR (Low Dynamic Range) image, a second LDR image, and a third LDR image, each having a different exposure value.

[0064] The weight map estimation unit 920 has a weight map estimation model, and is a means for extracting a weight map by applying the first, second, and third LDR images acquired by the image acquisition unit 910 to the weight map estimation model, and then generating an input feature map that integrates the first, second, and third LDR images using the extracted weight map.

[0065] For example, the weight map estimation unit 920 applies the first, second, and third LDR images to a convolutional layer of a weight map estimation model to extract a first feature map, a second feature map, and a third feature map, respectively. Then, the weight map estimation unit 920 applies the first and second feature maps to a first attention module of the weight map estimation model to generate a first weight map, and applies the third and second feature maps to a second attention module of the weight map estimation model to generate a second weight map. The weight map estimation unit 920 reflects the first and second weight maps in the first and third feature maps, respectively, and then combines them with the second feature map to generate an input feature map.

[0066] The HDR imaging device 930 is a means for applying an input feature map to a Sween-Fourier convolutional network model to perform a Fourier transform, and after the convolution operation, connecting the reference feature map to generate an HDR (High Dynamic Range) image.

[0067] The Swinn-Fourier convolutional network model is a U-shaped network model consisting of multiple Swinn-Fourier convolution blocks and transpose convolution blocks.

[0068] As described above, the Swinn-Fourier convolution block includes a Swinn transformer block that divides an input feature map into patch units to construct a hierarchical feature map; a residual block that adds a global residual to the feature map of a reference image using jump connections; a first 3×3 convolution layer and a second 3×3 convolution layer connected to the rear end of the Swinn transformer block, which combines output values ​​of the first 3×3 convolution layer and the second 3×3 convolution layer element by element to output a first sub-synthesis result value; a connection unit that combines output values ​​of the third 3×3 convolution layer connected to the rear end of the residual block and a spectral transform module that Fourier transforms the output values ​​of the residual block element by element to output a second sub-synthesis result value; and a connection unit that connects the first sub-synthesis result value and the second sub-synthesis result value.

[0069] This type of Swin-Fourier convolutional network model has the advantage of easily performing HDR synthesis by dividing the entire area of ​​the input feature map into patch units, performing a Fourier transform, and learning using periodic patterns.

[0070] Also, as mentioned above, the U-shaped network model is trained using log perceptual loss to increase activation, causing regions with low values ​​to have high values ​​and regions with high values ​​to maintain high values.

[0071] The memory 940 stores various commands for performing an HDR imaging method according to an embodiment of the present invention.

[0072] The processor 950 is a means for controlling the internal components (e.g., the image acquisition unit 910, the weight map estimation unit 920, the HDR generation unit 930, the memory 940, etc.) of the HDR imaging device 100 according to one embodiment of the present invention.

[0073] Apparatus and methods according to embodiments of the present invention may be embodied in the form of program instructions that can be executed by various computer means and recorded on a computer-readable recording medium. Computer-readable recording media may include, alone or in combination, program instructions, data files, data structures, and the like. The program instructions recorded on the computer-readable recording medium may be specially designed and constructed for the present invention, or may be well known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include not only machine language code, such as that produced by a compiler, but also high-level language code that can be executed by a computer using an interpreter, for example.

[0074] The hardware devices described above are configured to operate with at least one software module to perform the operations of the present invention, and vice versa.

[0075] The present invention has been described above with reference to its preferred embodiments. Those skilled in the art will recognize that the present invention can be embodied in various modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the claims, not the above description, and all variations within the scope of equivalents thereto should be construed as being within the scope of the present invention. [Explanation of symbols]

[0076] 100 HDR imaging device 910 Image Acquisition Unit 920 Weight Map Estimation Unit 930 HDR generation section 940 memory 950 processor

Claims

1. 1. A method of high dynamic range imaging performed on a high dynamic range imaging device, comprising: (a) obtaining a first LDR (Low Dynamic Range) image, a second LDR image, and a third LDR image, each having a different exposure value; (b) applying a weight map estimation model to the first LDR image, the second LDR image, and the third LDR image to extract a weight map, and using the extracted weight map to generate an input feature map that integrates the first LDR image, the second LDR image, and the third LDR image; (c) applying the input feature map to a Swin-Fourier convolutional network model to perform a Fourier transform, and then connecting a reference feature map after the convolution operation to generate a High Dynamic Range (HDR) image; The step (b) includes: applying the first LDR image, the second LDR image, and the third LDR image to a convolutional layer of the weight map estimation model to extract a first feature map, a second feature map, and a third feature map, respectively; applying the first feature map and the second feature map to a first attention module of the weight map estimation model to generate a first weight map, and applying the third feature map and the second feature map to a second attention module of the weight map estimation model to generate a second weight map; and reflecting the first weight map and the second weight map in the first feature map and the third feature map, respectively, and then combining the first weight map and the second weight map with the second feature map to generate the input feature map; the reference feature map is the second feature map extracted from the second LDR image; the Swinn-Fourier convolution network model is composed of a plurality of Swinn-Fourier convolution blocks and a transpose convolution block; The Sween-Fourier convolution block a Swine Transformer block that divides the input feature map into patch units to construct a hierarchical feature map; A residual block adds the global residual to the feature map of the reference image using jump connections; a first 3x3 convolution layer and a second 3x3 convolution layer connected to a rear end of the Swinn transformer block, which combine output values ​​of the first 3x3 convolution layer and the second 3x3 convolution layer element by element to output a first sub-synthesis result value; a third 3x3 convolution layer connected to a rear end of the residual block and output values ​​of a spectral transform module that performs a Fourier transform on the output values ​​of the residual block element by element to output a second sub-synthesis result value; and a connection unit that connects the first sub-synthesis result value and the second sub-synthesis result value.

2. The high dynamic range imaging method described in Claim 1, characterized in that the Swinn-Fourier convolutional network model is trained using a log perceptual loss function shown in the following equation 6, and the loss function of the Swinn-Fourier convolutional network model is calculated as shown in the following equation 8. [Equation 8] where: [Equation 6] where x denotes the target HDR image and y denotes the generated HDR image; [Number 61] denotes a Gaussian kernel that generalizes Euclidean distance to the manifold with geodesic distance, and T(·) is the tone mapping operator, [Equation 7] where μ denotes a compression parameter and l １ denotes MAE loss and VGG19() denotes convolutional neural network.

3. 10. A computer-readable recording medium having recorded thereon program code for executing the high dynamic range imaging method performed by the high dynamic range imaging device according to claim 1.

4. an image acquisition unit that acquires a first LDR (Low Dynamic Range) image, a second LDR image, and a third LDR image, each having a different exposure value; a weight map estimator that applies a weight map estimation model to the first LDR image, the second LDR image, and the third LDR image to extract a weight map, and generates an input feature map that integrates the first LDR image, the second LDR image, and the third LDR image using the extracted weight map; and an HDR (High Dynamic Range) generation unit that applies the input feature map to a Swinn-Fourier convolutional network model to perform a Fourier transform, and then connects a reference feature map after the convolution operation to generate an HDR image; The weight map estimation unit applying the first LDR image, the second LDR image, and the third LDR image to a convolutional layer of the weight map estimation model to extract a first feature map, a second feature map, and a third feature map, respectively; applying the first feature map and the second feature map to a first attention module of the weight map estimation model to generate a first weight map, and applying the third feature map and the second feature map to a second attention module of the weight map estimation model to generate a second weight map; reflecting the first weight map and the second weight map in the first feature map and the third feature map, respectively, and then combining the first weight map and the second feature map to generate the input feature map; the reference feature map is the second feature map extracted from the second LDR image; the Swinn-Fourier convolution network model is composed of a plurality of Swinn-Fourier convolution blocks and a transpose convolution block; The Sween-Fourier convolution block a Swine Transformer block that divides the input feature map into patch units to construct a hierarchical feature map; A residual block adds the global residual to the feature map of the reference image using jump connections; a first 3x3 convolution layer and a second 3x3 convolution layer connected to a rear end of the Swinn transformer block, which combine output values ​​of the first 3x3 convolution layer and the second 3x3 convolution layer on an element-by-element basis to output a first sub-synthesis result value; a third 3x3 convolution layer connected to a rear end of the residual block and output values ​​of a spectral transform module that performs a Fourier transform on the output values ​​of the residual block on an element-by-element basis to output a second sub-synthesis result value; and a connection unit that connects the first sub-synthesis result value and the second sub-synthesis result value.

Images