Image processing device, imaging device, control method, and program

Abstract

To determine a composition ratio suitable for generation of a composite image having an extended dynamic range.SOLUTION: An image processing device includes: acquisition means that, for each of a plurality of HDR (High Dynamic Range) images photographed with different amounts of exposure, acquires exposure information of photographing and an upper limit value of a dynamic range expressed in the image; determination means that, for a composition process for composing the plurality of HDR images to generate a composite image in which the dynamic range is further extended, determines a composition ratio of each HDR image for each signal region of the composite image, based on the exposure information and the upper limit value of the dynamic range acquired by the acquisition means.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, a control method, and a program, and in particular to a synthesis technique for a high dynamic range image signal. [Background technology]

[0002] There is a technique for generating a high dynamic range (HDR) composite image with an expanded dynamic range by synthesizing a plurality of standard dynamic range (SDR) images obtained by photographing a scene with different exposure amounts. In Patent Document 1, three types of SDR images, namely, an appropriate image obtained by photographing with a proper exposure, an under image obtained by underexposure, and an over image obtained by overexposure, are synthesized according to a predetermined synthesis ratio to generate a high dynamic range (HDR) composite image. More specifically, synthesis reference luminance thresholds Y1, Y2, Y3, and Y4 (FIG. 10 of Patent Document 1) are determined, and synthesis is controlled so that an over image is used in a luminance range darker than Y1, an appropriate image is used in a luminance range from Y2 to Y3, and an under image is used in a luminance range brighter than Y4. In addition, synthesis is controlled so that the synthesis ratios (weighted addition coefficients) of the over image and the appropriate image and the appropriate image and the under image are gradually changed for the intermediate regions from Y1 to Y2 and from Y3 to Y4. By using this type of synthesis control, it is possible to obtain an HDR synthesized image with an optimally expanded dynamic range from SDR images under three different exposure conditions.

[0003] In recent years, the performance of light-emitting elements such as LEDs has improved, and display devices known as HDR displays, which have a wider dynamic range of display brightness than conventional displays, have appeared. Such display devices can more faithfully display images (HDR images) that have colors and details in the high brightness range. The signal characteristics that represent the relationship between the video signal level and display brightness in HDR images are specified by the EOTF (Electro-Optical Transfer Function), and the following two types of methods are adopted. One is the HLG (Hybrid Log Gamma) method standardized in ARIB STD-B67, which converts the video signal level into a relative value of the display brightness, so that the display brightness corresponds to the maximum brightness that the display device can output. The other is the PQ (Perceptual Quantization) method standardized in SMPTE ST 2084 or ITU-R BT.2100, which converts the video signal level to a maximum of 10,000 nit (or cd / m 2 ) to an absolute value of display luminance. Therefore, when an HDR image obtained by capturing a scene is displayed, in the former method, the scene luminance is converted to a display luminance corresponding to the maximum luminance that the display device can output, while in the latter method, the scene luminance is converted to a display luminance that is absolutely determined regardless of the display device. Therefore, when assuming display on a display device that employs the PQ method, for example, in encoding in the imaging device, it is necessary to convert the image signal of the scene luminance to indicate an absolute luminance value and generate an HDR image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2013-240031 A Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, by acquiring such HDR images with different exposure amounts for a scene and synthesizing them, a synthetic image with a further expanded dynamic range can be generated. That is, as in Patent Document 1, by synthesizing three types of HDR images, namely, an appropriate HDR image obtained by shooting with a proper exposure, an under-HDR image obtained by underexposure, and an over-HDR image obtained by overexposure, a synthetic image expressing more details can be generated.

[0006] On the other hand, in the encoding of the PQ method that expresses the scene luminance absolutely, even if the same scene is shot, the peak luminance (maximum value of display luminance, maximum value of output dynamic range) included in the HDR image may change due to the difference in the exposure amount. This is because the scene luminance at which the sensor output is saturated changes according to the exposure amount, and therefore the gamma curves used for conversion are different in order to assign absolute display luminance to the same scene luminance. For example, as shown in FIG. 1, the input / output characteristics (relationship between the number of input steps and the output luminance) for two shooting modes with different exposure amounts have different peak luminances (maximum value of output luminance). Here, the input / output characteristics 11 in the shooting mode with high exposure amount are represented by a solid line, and the input / output characteristics 12 in the shooting mode with low exposure amount are represented by a dashed line. As shown in the figure, the two shooting modes show common input / output characteristics except for the high luminance range, and are converted to the same display luminance regardless of the exposure amount, whereas in the high luminance range, the peak luminance differs between values ​​13 and 14 according to the difference in the saturation luminance. The value 15 indicates the maximum value (1023) in 10 bits, and corresponds to the maximum display brightness of 10,000 nit in the PQ method.

[0007] Therefore, even if the synthesis method of Patent Document 1 is simply applied, there is a possibility that a suitable synthetic image will not be generated. More specifically, the synthesis method of Patent Document 1 is based on an 8-bit SDR image in which a pixel value with a maximum value of 255 is assigned for each exposure condition, so that in the luminance range in which two types of images are synthesized, it is possible to always refer to the pixel values ​​of both images. However, when synthesizing HDR images with different exposure amounts, in the luminance range in which two types of images are synthesized, a subject exceeding the peak luminance may not be expressed in the high-exposure HDR image, and a suitable synthesis result may not be obtained.

[0008] For example, consider a case where the blending ratio of Patent Document 1 is applied to an aspect where the luminance of the appropriate HDR image is distributed as shown in histogram 24 of FIG. 2 (the peak luminance of the appropriate HDR image is 25). In the figure, a two-dot chain line 21, a solid line 22, and a one-dot chain line 23 indicate the weighted addition coefficient of the over HDR image, the weighted addition coefficient of the appropriate HDR image, and the weighted addition coefficient of the under HDR image, which correspond to the blending ratio of Patent Document 1, respectively, and the sum of the coefficients is 1 in the entire luminance range. In this case, as shown in the figure, if the peak luminance 25 of the appropriate HDR image is included in the luminance range (Y3 to Y4) where the appropriate HDR image and the under HDR image are blended, the appropriate HDR image is not blended in the luminance range from the peak luminance to Y4, and therefore a suitable blending result is not obtained. In other words, since the appropriate HDR image does not include pixels with a peak luminance of 25 or more due to saturation of the sensor output, there is a possibility that an unnatural expression will be made in the area where the subject near the peak luminance of the blended image is distributed.

[0009] As in the above-described example, when a plurality of images with different peak luminance are synthesized, there is a possibility that the resulting image will be unnatural.

[0010] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an image processing device, an imaging device, a control method, and a program that determine an appropriate combination ratio for generating a composite image with an expanded dynamic range. [Means for solving the problem]

[0011] In order to achieve the above-mentioned object, the image processing device of the present invention is an image processing device that synthesizes a plurality of HDR (High Dynamic Range) images captured with different exposure amounts, the plurality of HDR images having scene luminance expressed in an absolute luminance format, and includes: a specifying means that specifies a signal value indicating an upper limit value of an output dynamic range for each of the plurality of HDR images; a determining means that determines a synthesis ratio of the plurality of HDR images based on the signal value specified by the specifying means; and an acquiring means that acquires a reference signal threshold for switching a tendency of the synthesis ratio of each of the plurality of HDR images, and the determining means changes the signal threshold acquired by the acquiring means based on the signal value indicating an upper limit value of the output dynamic range of each of the plurality of HDR images, and determines the synthesis ratio based on the changed signal threshold. Effect of the Invention

[0012] With this configuration, the present invention makes it possible to determine an appropriate blending ratio for generating a blended image with an expanded dynamic range. [Brief description of the drawings]

[0013] [Figure 1] A diagram to explain the input / output characteristics for two shooting modes with different exposure amounts. [Diagram 2] FIG. 13 is a diagram showing a manner in which a weighted addition ratio based on an SDR image is applied to a synthesis process of an HDR image. [Diagram 3] FIG. 1 is a block diagram illustrating a hardware configuration of an image processing device 100 according to an embodiment and a modification of the present invention. [Figure 4] FIG. 1 is a block diagram illustrating a module configuration of a synthesis process according to a first embodiment of the present invention. [Diagram 5] 1 is a flowchart illustrating a synthesis process according to a first embodiment of the present invention. [Figure 6] FIG. 1 is a diagram for explaining the difference in characteristics between an OETF applied according to the amount of exposure for shooting and an OETF applied for generating a composite image. [Figure 7]FIG. 2 is a diagram for explaining a reference mix table and a modified mix table according to an embodiment of the present invention; [Figure 8] FIG. 13 is a diagram illustrating an example of an output file format of a composite image according to an embodiment and a modification of the present invention. [Figure 9] A diagram showing an example of MaxDRL determined according to the shooting mode. [Figure 10] FIG. 11 is a block diagram illustrating a module configuration of a synthesis process according to a second embodiment of the present invention. [Figure 11] FIG. 11 is a block diagram illustrating a module configuration of a development process according to a second embodiment of the present invention. [Figure 12] FIG. 11 is a diagram for explaining a reference Mix table according to a second modified example of the present invention; [Figure 13] FIG. 11 is a diagram for explaining a modified Mix table according to the second modification of the present invention; [Figure 14] FIG. 11 is another diagram for explaining a modified Mix table according to the second modification of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] [Embodiment 1] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0015] In the embodiment described below, an example of the present invention is applied to an image processing device capable of generating a composite image with an expanded dynamic range by synthesizing HDR images encoded in the PQ format, as an example of an image processing device. However, the present invention can be applied to any device capable of deriving the synthesis ratio of each HDR image involved in generating the composite image.

[0016] In addition, in this specification, unless otherwise specified, an "HDR image" is described as an image (PQ signal) to which PQ encoding is applied and configured so that absolute display luminance can be obtained by decoding. Also, an image with a further expanded dynamic range obtained by synthesizing HDR images is described simply as a "synthetic image" to clearly distinguish it from an HDR synthetic image obtained by synthesizing a conventional SDR image and an input (synthetic source) HDR image. In the following description, the synthetic image is a PQ signal like the HDR image.

[0017] Here, a minimum of 10-bit depth is required to express HDR characteristics with a PQ signal, and the HDR image and the composite image are not files that store 8-bit images such as JPEG format. For example, the HDR image and the composite image may be a HEIF file stored using a container of the High Efficiency Image File Format (hereinafter referred to as HEIF), which is an image file format developed by the Moving Picture Experts Group (MPEG) and defined in MPEG-H Part 12 (ISO / IEC 23008-12). HEIF can store not only the original image, but also thumbnails, multiple time-related images, and metadata such as EXIF ​​and XMP in one file. In addition, HEIF can store 10-bit image sequences encoded with HEVC, so it can be said that it is suitable for storing the HDR image and the composite image in the present invention. Note that the implementation of the present invention is not limited to this, and the HDR image and the composite image may be in any format as long as it can store image data of 10 bits or more, such as RAW data and TIFF data after development processing.

[0018] Hardware configuration of image processing device 3 is a block diagram showing a hardware configuration of the image processing device 100 according to this embodiment. As shown in the figure, the image processing device 100 according to this embodiment has a CPU 101, a ROM 102, a RAM 103, a recording device 104, an operation I / F 105, a display device 106, and a system bus 107.

[0019] The CPU 101 performs overall control of the image processing device 100. The ROM 102 is a storage device that stores control programs such as a BIOS required to start up the image processing device 100, programs that do not require modification, parameters, and data. The RAM 103 is a storage device that has a work area for the CPU 101, a primary storage area for temporarily storing various data, an area for loading various programs, etc.

[0020] The recording device 104 is a recording device that stores various data such as an OS program, various control programs, various software programs executable on the OS program, an HDR image used in a synthesis process described below, and a synthesis image obtained as a result of the synthesis process. The recording device 104 includes, for example, a hard disk or flash memory built into or detachably connected to the image processing device 100, a detachably connected flexible disk, optical disk, magneto-optical disk, IC card, memory card, etc. Therefore, the CPU 101 can control the image processing device 100 by expanding various programs stored in the ROM 102 or the recording device 104 into the RAM 103 and executing them.

[0021] The operation I / F 105 is a user interface provided in the image processing device 100, such as a keyboard, a mouse, a touch panel, etc. When the operation I / F 105 detects that a user has input an operation, it sends a control signal related to the operation to the CPU 101. The display device 106 is, for example, a liquid crystal display, and displays various information, including a graphical user interface related to the OS and software operating on the image processing device 100. The system bus 107 connects each block constituting the image processing device 100 so that they can communicate with each other.

[0022] The image processing device 100 may be a dedicated device for developing and editing captured RAW images, in which case a compositing program can be stored in the ROM 102. A general-purpose personal computer (PC) can function as the image processing device 100 by the CPU 101 expanding the compositing program stored in the recording device 104 into the RAM 103. An imaging device capable of acquiring a RAW image by imaging can also function as the image processing device 100 by its control unit executing the compositing program or by an image processing unit that performs the compositing process performing a specified operation.

[0023] <<Composition Processing>> Next, a compositing process executed in the image processing device 100 according to this embodiment, which combines a plurality of HDR images captured with different exposure amounts to generate a composite image, will be described in detail with reference to the drawings. This compositing process is realized by the CPU 101 reading out a compositing process program from the ROM 102 or the recording device 104, and expanding and executing the program in the RAM 103. FIG. 4 illustrates an example of a module configuration realized during execution of the compositing process, and FIG. 5 illustrates an example of a flowchart for explaining the flow of the compositing process.

[0024] Hereinafter, a description will be given assuming that a plurality of HDR images taken with different exposure amounts, which are used as inputs for the synthesis process, are all HDR PQ images (hereinafter referred to as PQ images) encoded in the PQ format. In addition, in the synthesis process of this embodiment, the input PQ images are three types: an appropriate PQ image 402 obtained by shooting with proper exposure, an over PQ image 401 obtained by shooting with overexposure, and an under PQ image 403 obtained by shooting with underexposure. The over PQ image 401, the appropriate PQ image 402, and the under PQ image 403 may be images obtained by sequentially shooting a common scene in a shooting mode with different exposure settings for synthesis of a synthetic image. In this embodiment, the shooting with different exposure settings will be described as shooting with different shooting modes for convenience, but it will be understood that the implementation of the present invention is not limited to this.

[0025] In this embodiment, each input PQ image is assigned with exposure information indicating the exposure setting of the imaging device when the PQ image was captured, and information on the OETF (Opt-Electronic Transfer Function) used to encode the PQ image as metadata. The exposure information may be, for example, information on the aperture, shutter time, and ISO sensitivity set at the time of capture. The OETF is a product of an OOTF (Opt-Optical Transfer Function) based on the production intention of the output image set in advance for each exposure setting or set by the user, and an inverse function of the reference EOTF defined in ITU-R BT.2100. Furthermore, each PQ image is assigned with MaxDRL (Maximum Dynamic Range Level) described in JP2020-039118A as metadata. MaxDRL is the maximum value of the PQ signal output value after applying the corresponding OETF in developing and encoding the captured image signal, and is the peak luminance value (upper limit value) of the dynamic range that can be expressed by each PQ image. In this embodiment, the maximum DRL is a peak luminance value, but the maximum DRL may be a value indicating a nit value corresponding to the peak luminance value. The maximum DRL is used to know the dynamic range of an input image when tone mapping an HDR image to an HDR or SDR signal with a narrower dynamic range. In this specification, the three types of PQ images that are input may be input in different HEIF files, or may be stored in a single HEIF file.

[0026] In S501, the over PQ image 401, the appropriate PQ image 402, and the under PQ image 403 are read from the recording device 104 and developed in the RAM 103. By the processing in this step, the PQ images to be composited into the composite image are input.

[0027] In S502, the exposure of the three types of PQ images is adjusted. Since the accuracy of the exposure adjustment may decrease if the PQ images are used as nonlinear signals, the PQ images are linearized before the exposure adjustment. More specifically, the exposure adjustment of the three types of PQ images is realized by linearization by the PQ degamma module 404, exposure adjustment by the exposure adjustment module 405, and nonlinearization by the PQ gamma module 406.

[0028] The linearization by the PQ degamma module 404 may be performed by referencing the information of the OETF used for encoding, which is attached to each PQ image, and applying the inverse function (degamma) of the OETF.

[0029] The exposure adjustment by the exposure adjustment module 405 is performed by applying a gain based on the exposure step to each linearized image. In this embodiment, the exposure adjustment is performed based on the appropriate exposure (the exposure used to obtain the appropriate PQ image 402), and a gain is applied to the linearized overexposed image and the underexposed image, and the exposure is corrected to the same as the appropriate exposure. For example, if the appropriate PQ image 402 is taken with an aperture of F5.6, a shutter speed of 1 / 60, and an ISO sensitivity of 200, and the under-PQ image 403 is taken with an aperture of F5.6, a shutter speed of 1 / 500, and an ISO sensitivity of 200, the exposure step is three steps. Therefore, the exposure adjustment module 405 corrects the exposure to the same as the appropriate PQ image 402 by increasing the gain of the image obtained by linearizing the under-PQ image 403 by three steps.

[0030] After that, the PQ gamma module 406 applies a common OETF to each image after exposure correction to make it nonlinear. The OETF applied by the PQ gamma module 406 is an OETF (dashed line 603) that corresponds to a wider dynamic range than the OETF (solid line 601) applied to the input PQ image, as shown in FIG. 6. As shown in the figure, the OETF applied by the PQ gamma module 406 may correspond to a higher luminance 604 so as to obtain a composite image with an expanded dynamic range in the subsequent composition process. Here, the OETF applied by the PQ gamma module 406 may be a fixed one that is set in advance, such as an inverse function of the reference EOTF defined in ITU-R BT.2100. Alternatively, the OETF may be adaptively changed so that the maximum luminance (luminance 604) is set according to the under-PQ image 403'.

[0031] By performing the process of S502, an over PQ image, an appropriate PQ image, and an under PQ image with consistent exposure are obtained. For convenience, the PQ images after the process of S502 are referred to below as an over PQ image 401', an appropriate PQ image 402', and an under PQ image 403'. Here, the over PQ image 401' has less noise in dark areas than the appropriate PQ image 402', and the under PQ image 403' has less blown-out highlights in bright areas than the appropriate PQ image 402', and is an image with gradation.

[0032] In S503, the misalignment correction module 407 detects misalignment between the over PQ image 401' and the appropriate PQ image 402', and between the under PQ image 403' and the appropriate PQ image 402', and performs a process of correcting the misalignment if any misalignment occurs. Any known method may be used for detecting and correcting the misalignment. For example, the image may be divided into blocks, edges may be extracted, and a motion vector that minimizes the sum of absolute values ​​of pixel value differences (SAD: Sum of Absolute Difference) is derived for each block. Then, the image to be corrected may be affine-transformed using an affine coefficient obtained based on the motion vector, thereby correcting the misalignment. When correcting the misalignment based on the appropriate PQ image 402', the affine transformation may be performed on the other images, that is, the over PQ image 401' and the under PQ image 403'. In this embodiment, because detection accuracy is improved by reducing the difference in brightness between images, the description will be given of performing position shift detection and correction after the processing of S502; however, since there is no dependency between these, it will be understood that the order of processing may be reversed.

[0033] In S504, the ratio change module 408 determines the composition ratio (weighted addition ratio, weighted addition coefficient) of each image. In this embodiment, the weighted addition ratio of each image is determined by configuring a modified Mix table by modifying the reference Mix table 409 as shown in FIG. 7(a) and referring to the modified Mix table. As shown in the figure, the reference Mix table 409 distributes the weighted addition ratios of the over PQ image 401', the appropriate PQ image 402', and the under PQ image 403' differently according to the luminance value, as in Patent Document 1. The reference Mix table 409 includes an over image ratio 701 (two-dot chain line), an appropriate image ratio 702 (solid line), and an under image ratio 703 (one-dot chain line) that indicate the weighted addition ratio of each image for the display luminance range of the PQ method (corresponding to an upper limit of 10,000 nit). Here, the weighted addition ratios of each image in the reference Mix table 409 are configured so that the sum is 1 (100%) for each luminance value. Therefore, each luminance range is divided into a range in which only the over PQ image 401' is used, a range in which the over PQ image 401' and the appropriate PQ image 402' are synthesized, a range in which only the appropriate PQ image 402' is used, a range in which the appropriate PQ image 402' and the under PQ image 403' are synthesized, and a range in which only the under PQ image 403' is used. These range divisions are defined based on a reference point (luminance threshold) that is set for each image and that switches the tendency of synthesis.

[0034] The ratio change module 408 reads the reference mix table 409 from, for example, the ROM 102 or the recording device 104, and then acquires the exposure information and MaxDRL attached to each of the over PQ image 401, the appropriate PQ image 402, and the under PQ image 403. The ratio change module 408 then changes a predetermined brightness threshold provided in the reference mix table 409 based on the acquired exposure information and MaxDRL, thereby forming a table indicating weighted addition ratios that make the synthesis result of the subsequent synthesis module 410 favorable. In this embodiment, as the predetermined brightness thresholds, a brightness threshold 704 at which the weighted addition ratio of the over PQ image 401' is 0% and a brightness threshold 705 at which the weighted addition ratio of the appropriate PQ image 402' is 0% are provided and are subject to change.

[0035] As described above, the luminance range in which the over PQ image 401' and the appropriate PQ image 402' are combined, and the luminance range in which the appropriate PQ image 402' and the under PQ image 403' are combined may not appear as a signal of an appropriate luminance value in an image taken with a high exposure. In other words, applying a weighted addition ratio exceeding 0% to an image that does not have a corresponding luminance range, or applying a weighted addition ratio less than 100% to the other image, may result in an unsuitable combination result. Therefore, the ratio change module 408 of this embodiment changes the luminance threshold 704 and the luminance threshold 705 to values ​​that do not exceed the maximum display luminance that can be represented by the corresponding images.

[0036] In this embodiment, the exposure of the over PQ image 401 and the under PQ image 403 is adjusted based on the appropriate PQ image 402, so the maximum display luminance of the appropriate PQ image 402' is the MaxDRL of the appropriate PQ image 402. On the other hand, the maximum display luminance of the over PQ image 401' is different from the MaxDRL of the over PQ image 401 due to the exposure adjustment in S502, so the ratio change module 408 derives it using the following formula. JPEG2025074260000002.jpg13148 That is, the ratio change module 408 applies the EOTF (OETF -1 ) to linearize the result, and then apply a gain related to the exposure difference between the over-PQ image 401 and the appropriate PQ image 402. The ratio change module 408 then applies the OETF to the obtained value to nonlinearize it, and derives the maximum display luminance for the over-PQ image 401′.

[0037] Therefore, the ratio change module 408 changes the brightness threshold 704 and the brightness threshold 705 to the maximum display brightness of the over PQ image 401' and the appropriate PQ image 402' obtained in this way, respectively, to obtain a new changed Mix table as shown in FIG. 7(b). More specifically, the ratio change module 408 linearly scales the over image ratio 701 so that the brightness threshold 704 becomes the brightness threshold 714 (the maximum display brightness of the over PQ image 401') to obtain a changed over image ratio 711. The ratio change module 408 linearly scales the appropriate image ratio 702 so that the brightness threshold 705 becomes the brightness threshold 715 (the maximum display brightness of the appropriate PQ image 402') to obtain a changed appropriate image ratio 712. The ratio change module 408 also linearly scales the under image ratio 703 at the same ratio as the appropriate image ratio 712 to obtain a changed under image ratio 713.

[0038] In this case, since the ratio of scaling applied to the over image ratio 711 and the appropriate image ratio 712 may differ, the luminance range in which the over PQ image 401' and the appropriate PQ image 402' are synthesized may be determined based on the over image ratio 711. That is, the luminance range in the appropriate image ratio 712 may be changed so that the weighted addition ratio of the appropriate PQ image 402' becomes 100% at the luminance threshold value 714. Also, the luminance range may be changed so that the weighted addition ratio of the appropriate PQ image 402' starts to increase from 0% at the luminance value 716 where the weighted addition ratio in the over image ratio 711 starts to decrease. That is, the modified Mix table may be configured by adjusting the image on the high exposure side in particular so that the sum of the weighted addition ratios of the images to be synthesized becomes 100% at any luminance value.

[0039] In S505, the synthesis module 410 performs a luminance-specific synthesis process using the over PQ image 401', the appropriate PQ image 402', and the under PQ image 403' after the alignment based on the changed Mix table constructed in S504, to generate a synthesis image. That is, the synthesis module 410 stores pixel values ​​at the same positions of the over PQ image 401' after the alignment for the dark area (luminance range where the over image ratio 711 is 100%). Also, the synthesis module 410 stores pixel values ​​at the same positions of the appropriate PQ image 402' for the midtone area (luminance range where the appropriate image ratio 712 is 100%). Also, the synthesis module 410 stores pixel values ​​at the same positions of the under PQ image 403' for the bright area (luminance range where the under image ratio 713 is 100%). Furthermore, for the dark to midtone regions, the synthesis module 410 derives and stores pixel values ​​by multiplying the pixel values ​​at the same positions of the appropriate PQ image 402' and the over PQ image 401' after alignment by their respective weighted addition coefficients and adding them together. For the midtone to light regions, the synthesis module 410 derives and stores pixel values ​​by multiplying the pixel values ​​at the same positions of the appropriate PQ image 402' and the under PQ image 403' after alignment by their respective weighted addition coefficients and adding them together. This makes it possible to obtain a synthesis image in which the expression of the luminance range where switching between images occurs is smooth and the scene is expressed in a suitable manner from dark to light.

[0040] The generated composite image may be stored in the recording device 104 as a new PQ image in the form of an image file in the HEIF format. The HEIF format image file has a file structure as shown in FIG. 8(a). That is, the HEIF format file is composed of an ftyp box 801 indicating the file format, a meta box 802 storing metadata including a thumbnail image 811, and an mdat box 803 storing encoded data (main image 812). Therefore, the synthesis module 410 stores data obtained by encoding the generated composite image in the PQ format in the mdat box 803, stores a thumbnail image of the composite image and the MaxDRL related to the composite image in the meta box 802, and generates a file.

[0041] Here, the MaxDRL of the composite image may be, for example, the maximum display luminance of the under-PQ image 403'. That is, since the upper limit of the dynamic range of the composite image is determined mainly based on the under-PQ image 403 that is synthesized by gaining up, the synthesis module 410 may derive information on the maximum display luminance by the following formula and use it as MaxDRL. JPEG2025074260000003.jpg14135 In other words, the MaxDRL of the under-PQ image 403 is converted into a linear value by applying the inverse characteristics of the OETF used for encoding, and this value is then gained up and made nonlinear by applying the common OETF, and the resulting value is stored as the MaxDRL of the composite image.

[0042] Alternatively, the MaxDRL does not need to be determined based on the MaxDRL of the image, and for example, for the shooting mode used to acquire the input PQ image, information on the MaxDRL may be stored in advance as shown in Fig. 9, and the corresponding value may be adopted. For example, when three PQ images shot in a normal shooting mode with ±3 stops are input, the synthesis module 410 can determine the MaxDRL of the composite image to be 888 (2906 nit) based on the information (maximum signal PQ code value) stored in the recording device 104. This allows the output composite image to be a file in an easily usable form.

[0043] [Embodiment 2] In the above-described embodiment, the input HDR image to be subjected to the synthesis process is a PQ image, but the present invention is not limited to this. That is, the input image may be an HDR image in which the scene luminance is expressed in a manner different from the manner of allocating a fixed range regardless of the exposure condition as in the case of an SDR image, and may be, for example, a RAW image (12 to 14 bits) having a higher resolution than a PQ image (10 bits). In this case, for example, a module configuration as shown in FIG. 10 may be used to realize the synthesis process by inputting a plurality of RAW images obtained by shooting with different exposure amounts. Here, the input RAW images are three types: an appropriate RAW image 1002 obtained by shooting with appropriate exposure, an over RAW image 1001 obtained by shooting with overexposure, and an under RAW image 1003 obtained by shooting with underexposure. In FIG. 9, the same reference numbers are used for the components common to the module configuration related to the synthesis process of the first embodiment, and detailed description thereof will be omitted below.

[0044] The synthesis process of this embodiment includes conversion (development process) of three types of inputted RAW images before development into PQ images by the development process module 1004. The development process executed by the development process module 1004 is divided into processes by the functional modules shown in FIG. 11. Specifically, the white balance module 1101 executes white balance processing for making white white on the inputted RAW image, and a gain is applied to each of R, G, and B so that R, G, and B in the area that should be white have the same signal value. Then, the noise reduction module 1102 executes noise reduction processing for the inputted image to reduce noise caused by the sensor that is not derived from the subject image. Then, the color interpolation module 1103 applies color interpolation processing to the inputted color mosaic image to generate a color image in which R, G, and B color information is complete for all pixels. The generated color image is subjected to matrix conversion processing by the matrix conversion module 1104 and gamma conversion processing by the gamma conversion module 1105 to generate a basic color image. Here, for example, an OETF corresponding to the shooting mode attached to each RAW image is used for the gamma conversion process by the gamma conversion module 1105. After that, the color adjustment module 1106 applies image correction processes to the color image to improve the appearance of the image, such as saturation emphasis, hue correction, edge emphasis, etc., and outputs a PQ image.

[0045] In this way, even in a case where a linear RAW image is input, by first applying the OETF to convert it to a PQ image, a similarly suitable composite image can be generated using the functional module related to the synthesis processing of embodiment 1.

[0046] In this embodiment, the functional module related to the compositing process of the first embodiment is used so that it can be used in combination with the compositing process of the first embodiment, but it will be easily understood that the implementation of the present invention is not limited to this. That is, since the information on the luminance range of each image related to the determination of the compositing ratio is obtained by adjusting the exposure of the linearized images, the development process by the development process module 1004 and the linearization process by the PQ degamma module 404 do not have to be performed.

[0047] [Variation 1] In the above-described embodiment, an aspect has been described in which the OETF used / to be used for development and encoding of an HDR image that is an input for synthesis processing can be obtained. On the other hand, since including information on the OETF (or information on the EOTF corresponding to the inverse characteristic thereof) as metadata in an image file can cause an increase in file size, the information may not be included. In this case, the inverse function of the OETF used for linearization by the PQ degamma module 404 may be the reference EOTF defined in ITU-R BT.2100, although the accuracy may decrease. In this case, the inverse function of the reference EOTF may be used for nonlinearization by the PQ gamma module 406.

[0048] [Variation 2] In the above-described embodiment and modified example, the HDR image, which is the input for the compositing process, is obtained by shooting with different exposure amounts, and the magnitude relationship of the peak luminance values ​​in the dynamic range is distributed in a manner suitable for the compositing process. In other words, the weighted addition ratio that can favorably combine the input images is derived by scaling the reference Mix table 409 based on the MaxDRL of the HDR image with the adjusted exposure.

[0049] However, for example, if the brightness conversion characteristics of the input HDR image have been changed by retouching applied after shooting (MaxDRL changes), the brightness range in which the two types of images are synthesized may be suppressed, and the transition between the images may not be expressed smoothly. In other words, if the brightness range allocated to the transition is narrow, the change in the weighted addition ratio in that brightness range becomes steep, and this may cause an unnatural expression (hereinafter referred to as a transition step) in the generated composite image.

[0050] Furthermore, for example, when the OETF cannot be acquired for the input HDR image as in Modification 1, or when retouching is applied that changes the luminance conversion characteristics (changing the EOTF required for linearization), linearization and exposure adjustment may be insufficient. In other words, since it is not possible to assign an appropriate luminance to a subject near the saturation luminance of each HDR image, color bending and the like may occur in the composite image.

[0051] In this modification, a method for generating a modified Mix table related to determining the weighted addition ratio of each HDR image in order to prevent such switching step and color bending from occurring in a composite image will be described with reference to the drawings. That is, in this modification, a mode will be described in which the weighted addition ratio of each image can be suitably set according to not only the MaxDRL of the input HDR image but also the combination thereof.

[0052] In this embodiment, unlike the above-described embodiment, a reference mix table as shown in FIG. 12 is provided for Max(R,G,B), which is the maximum value of any one of the R, G, and B color signals of the reference pixel of the input image. In the reference mix table of FIG. 12, like the reference mix table 409 shown in FIG. 7(a), a two-dot chain line 1201 indicates the weighted addition ratio for an over HDR image, a solid line 1202 indicates the weighted addition ratio for an appropriate HDR image, and a one-dot chain line 1203 indicates the weighted addition ratio for an under HDR image. The reference mix table of FIG. 12 differs from the reference mix table 409 shown in FIG. 7(a) in that the reference mix table of FIG. 12 specifies the maximum signal value (Max(R,G,B)) among R, G, and B, rather than the luminance value. In the following, for ease of explanation, a signal threshold value 1206 at which the weighted addition ratio of an over HDR image starts to decrease from 100% and the weighted addition ratio of an appropriate HDR image starts to increase from 0%, or a threshold value corresponding thereto, is referred to as Over. 1End In addition, the signal threshold 1204 at which the weighted addition ratio of the over HDR image reaches 0% and the weighted addition ratio of the suitable HDR image reaches 100%, or a threshold corresponding thereto, is referred to as Suit 1StartIn addition, the signal threshold 1207 at which the weighted addition ratio of the suitable HDR image starts to decrease from 100% and the weighted addition ratio of the under HDR image starts to increase from 0%, or a threshold corresponding thereto, is referred to as Suit 1End In addition, the signal threshold 1205 at which the weighted addition ratio of the suitable HDR image reaches 0% and the weighted addition ratio of the under HDR image reaches 100%, or a threshold corresponding thereto, is referred to as the Under HDR image. 1Start The ratio change module 408 of this modified example configures a changed Mix table for synthesis processing in the synthesis module 410 by adjusting these specified signal thresholds 1204 to 1207.

[0053] <<Overview of Changed Mix Table Configuration>> Here, an overview will be given of the configuration of the modified Mix table that determines the weighted addition ratio of each HDR image using the reference Mix table in this modified example.

[0054] As shown in FIG. 6, the occurrence of color bending as described above is mainly caused by nonlinear conversion (change in the degree of slant of the output curve) near the saturation luminance value. Therefore, if the conversion characteristics applied at the time of development or the conversion characteristics taking into account retouching cannot be obtained, the luminance value of the input HDR image cannot be accurately linearized, and consistency between images is not guaranteed in the composition of the corresponding value range. On the other hand, if the vicinity of the saturation luminance value is excluded, it is highly likely that linear conversion has been performed. For this reason, the ratio change module 408 of this modified example determines the number of steps (composite avoidance steps) at which nonlinear conversion may be performed as a constant, and derives the maximum display luminance value (converted MaxDRL) for the over HDR image and the appropriate HDR image by taking into account the composite avoidance steps. More specifically, the ratio change module 408 derives the converted MaxDRL by reducing the value linearized by applying, for example, the reference EOTF to the MaxDRL of the input image by the value of the composite avoidance steps, and changes the reference Mix table based on this.

[0055] The occurrence of the switching step as described above may occur due to suppression of the signal range defined by the signal threshold 1206 to the signal threshold 1204 and the signal range defined by the signal threshold 1207 to the signal threshold 1205. In particular, as in this modification, when the reference Mix table is changed using the converted MaxDRL set to exclude the luminance range near the saturation luminance, suppression of the signal range occurs. Since there is a margin before the signal value saturates for the under-HDR image, in this modification, the ratio change module 408 assigns the weighted addition ratio of the over-HDR image and the appropriate HDR image based on the signal value corresponding to the converted MaxDRL of the appropriate HDR image. Therefore, the range from the signal value 0 to the signal value corresponding to the converted MaxDRL of the appropriate HDR image includes the following two types of ranges. One is a range where the weighted addition ratio of the over-HDR image is 100% and a range of 0% to 100% (hereinafter referred to as an over use range). The other is a range where the weighted addition ratio of the appropriate HDR image is 100% and a range from 0% to 100% (hereinafter referred to as appropriate use range). In this modified example, for the over use range and appropriate use range, a threshold W is set for the width of the signal range that should be secured to perform smooth synthesis, and each signal threshold is adjusted so that the appropriate use range has a range width of at least the threshold W, thereby configuring a modified Mix table.

[0056] As a process performed by the ratio change module 408, first, a converted MaxDRL is derived for the over HDR image and the appropriate HDR image by the following formula. JPEG2025074260000004.jpg14131Specifically, the converted MaxDRL of each image is derived by applying a reference EOTF to the MaxDRL of each image, for example, and linearizing the value, and then applying a common OETF to perform gain down according to the difference from the appropriate exposure and the number of synthesis avoidance steps to make it nonlinear, thereby deriving the converted MaxDRL of each image. That is, in order to avoid inconsistencies that occur due to differences between the inverse characteristics of the OETF used when developing the input image and the characteristics of the EOTF applied to linearization, the ratio change module 408 obtains a converted MaxDRL that excludes the vicinity of saturation luminance by reducing the gain by the number of synthesis avoidance steps.

[0057] Then, the proportion change module 408 scales each signal threshold in the reference Mix table based on the converted MaxDRL obtained for each of the over HDR image and the suitable HDR image, and derives each signal threshold for the modified Mix table. More specifically, the proportion change module 408 normalizes the range of signal value 0 to signal threshold 1204 in the reference Mix table, and multiplies this by a signal value corresponding to the converted MaxDRL of the over HDR image. As a result, for the over HDR image, the Over 1End (corresponding to signal threshold 1206) and Suit 1Start (corresponding to the signal threshold value 1204). The ratio change module 408 also normalizes the range of signal values ​​0 to 1205 in the reference Mix table, and multiplies this range by a signal value corresponding to the converted MaxDRL of the suitable HDR image. As a result, a Suit 1End (corresponding to signal threshold 1207) and Under 1Start (corresponding to a signal threshold of 1205) is obtained.

[0058] In this way, four types of signal thresholds (Over 1End Suit 1Start Suit 1End and Under 1Start ) does not need to be adjusted if the over use range and the appropriate use range have a range width of the threshold W. In other words, even if a changed Mix table is constructed based on the four signal thresholds, a composite image in which the occurrence of switching gaps is avoided can be obtained. On the other hand, if the range width conditions are not met for each use range, switching gaps may occur, so the ratio change module 408 uses the four signal thresholds to determine whether or not adjustment is necessary and makes the adjustment, for example, as follows:

[0059] As an extreme example, Suit 1Start and Under 1StartIn this case, if the ratio change module 408 simply creates a modified Mix table, it will be as shown in FIG. 13(a). According to the modified Mix table, the value range (Suit 1End ~Under 1Start ) is not sufficiently secured, and a switching step may occur. In order to avoid such a situation, the ratio change module 408 of this modified example changes Suit so that the suitable use range has a range width of the threshold W, as shown in FIG. 1Start Adjust the Suit 1Start The signal value corresponding to the conversion MaxDRL of the appropriate HDR image (Under 1Start ) minus the threshold W, the appropriate range is secured. As a result, the range in which the appropriate HDR image and the under-HDR image are combined is secured, and the occurrence of a step at the switching is avoided.

[0060] On the other hand, it is also possible to suppress the over-use region by giving priority to the appropriate use region. 1Start If the threshold value W is more than twice as large as the threshold value W, the range of the threshold value W can be secured for the appropriate use range and the overuse range even if the adjustment as shown in FIG. 13(b) is performed. However, as shown in FIG. 1Start When the threshold value W is less than twice the threshold value W, the over-use range can be suppressed by securing the range of the threshold value W for the appropriate use range (FIG. 14(b)). 1Start is less than twice the threshold W, and Suit 1Start Under 1Start , the over-use range is suppressed by adjusting the appropriate use range to have the width of the threshold W. As a result, a value range for synthesizing the over HDR image and the appropriate HDR image is not secured, and a switching step may occur. Therefore, in such a case, the ratio change module 408 adjusts the signal value from 0 to Under as shown in FIG. 14(c), for example, to avoid the occurrence of a switching step in the over-use range and the appropriate use range as much as possible. 1StartSpecifically, the ratio change module 408 adjusts the range of Suit 1Start Under 1Start Adjust to half the value.

[0061] In other words, the ratio change module 408 of this modified example derives four types of signal thresholds based on the converted MaxDRL of the over HDR image and the appropriate HDR image, and then configures a changed Mix table as follows. (1) Under 1Start > Threshold W × 2 and Under 1Start -Suit 1Start <If the threshold is W, then the suitability range is set to W. 1Start (Fig. 13(b)) to construct the modified Mix table. (2) Under 1Start ≦ Threshold W × 2 and Suit 1Start ≧Under 1Start If the ratio is ×1 / 2, adjust each signal threshold so that the over-usage range and the appropriate usage range are of equal width (Fig. 14(c)) and configure the modified Mix table. (3) Otherwise, construct a modified Mix table without adjusting each signal range. In constructing the modified Mix table, the ratio change module 408 adjusts the ratio of the Over 1End and Suit 1End In this way, it is possible to configure a modified Mix table that ensures the width of the threshold W or an equal width for each signal range in which the same combination of HDR images is synthesized.

[0062] By using the weighted addition ratio related to the modified Mix table configured in this manner, the synthesis module 410 can generate a synthesis image in which the occurrence of switching steps and color bending is reduced. Note that, in this modified example, a threshold value W is set for each use range to adjust the signal threshold value and configure the modified Mix table, but the implementation of the present invention is not limited to this. For example, when the change in the weighted addition ratio for the signal range in which the HDR image is synthesized is determined linearly, a threshold value may be set for the slope of the linear function (the rate of change of the weighted addition ratio) to perform the adjustment and configure the modified Mix table.

[0063] As described in the above embodiment and modified examples, the image processing device according to the present invention can determine a suitable combination ratio for generating a composite image with a further expanded dynamic range based on an HDR image.

[0064] In this specification, the input for the synthesis process is described as three types of HDR images taken with different exposure amounts, but the present invention is not limited to this embodiment, and it goes without saying that the present invention can be applied to synthesis processing of any number of HDR images. In addition, in generating a synthetic image, the maximum number of HDR images to be synthesized for one signal range is not limited to two, and may be two or more as long as the sum of the weighted addition ratios falls within 100%.

[0065] In addition, in this specification, a mode in which the weighted addition ratio of each HDR image can be obtained by configuring a modified Mix table by modifying the reference Mix table has been described, but the implementation of the present invention is not limited to this. That is, when implementing the present invention, the weighted addition ratio of each HDR image does not need to be defined in the form of a table, and may be defined by a function that derives the weighted addition ratio by performing a calculation each time.

[0066] In addition, in this specification, in order to synthesize HDR images of the PQ method to generate a composite image, the PQ gamma module 406 has been described as applying a common gamma related to the same method to perform nonlinearization. However, it should be understood that the nonlinearization is not essential when determining the weighted addition ratio. In other words, the weighted addition ratio of each HDR image may be determined according to the format of the HDR image to be synthesized in the synthesis process or the format of the synthesized image to be generated.

[0067] The file format of the composite image output after the composition process may be such that it is possible to store at least the upper limit value of the dynamic range of the composite image, for example, the MP4 format as shown in Fig. 8(b). In this case, the upper limit value may be stored in the meta data 822 in the mdat box 821.

[0068] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0069] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0070] 100: image processing device, 101: CPU, 102: ROM, 103: RAM, 104: recording device, 404: PQ degamma module, 405: exposure matching module, 406: PQ gamma module, 408: ratio change module, 410: composition module

Claims

1. An image processing device that synthesizes a plurality of HDR (High Dynamic Range) images that are taken with different exposure amounts, and in which scene luminance is expressed in an absolute luminance format, a signal value indicating an upper limit of an output dynamic range for each of the plurality of HDR images; a determination means for determining a synthesis ratio of the plurality of HDR images based on the signal value identified by the identification means; an acquisition means for acquiring a reference signal threshold value for switching a tendency of the synthesis ratio for each of the plurality of HDR images; having The image processing device characterized in that the determination means changes the signal threshold acquired by the acquisition means based on the signal value indicating an upper limit value of the output dynamic range of each of the plurality of HDR images, and determines the combining ratio based on the changed signal threshold.

2. The image processing apparatus according to claim 1 , wherein the determining unit determines a blending ratio of the plurality of HDR images based on the exposure amounts of the plurality of HDR images.

3. The image processing apparatus according to claim 2 , wherein the specifying unit specifies the signal value indicating the upper limit of the output dynamic range by adjusting exposure of the plurality of HDR images.

4. 4. The image processing device according to claim 3, wherein the determining means corrects the signal value indicating an upper limit value of the output dynamic range of each of the plurality of HDR images based on an exposure amount of the image, and determines a combining ratio of the plurality of HDR images based on the signal value indicating an upper limit value of the output dynamic range of the plurality of HDR images after the correction.

5. The image processing apparatus according to claim 4 , wherein the correction is a process of adjusting a gain to match an exposure amount of one of the plurality of HDR images to an exposure amount of another HDR image.

6. The image processing device according to claim 4 or 5, wherein the determining means determines the combining ratio so as not to combine images in a signal range exceeding an upper limit value of the output dynamic range after the correction of each of the plurality of HDR images.

7. The method further includes a means for acquiring a reference Mix table that defines a reference signal threshold value for switching a tendency of a synthesis ratio of each of the plurality of HDR images for a signal range in which the plurality of HDR images are synthesized, The determination means changes the signal threshold value of the reference mix table based on a signal value indicating an upper limit value of an output dynamic range after the correction of the plurality of HDR images to configure a modified mix table, and determines the blending ratio based on the modified mix table.

7. The image processing device according to claim 4, wherein the first and second inputs are input to the image processing apparatus.

8. Each of the plurality of HDR images is an image in which a signal value is nonlinearized by applying gamma; The determining means performs the correction after linearizing the signal value indicating the upper limit value of the output dynamic range of each of the plurality of HDR images by applying de-gamma.

8. The image processing device according to claim 4, wherein the first and second inputs are input to the image processing apparatus.

9. 9. The image processing device according to claim 1 , wherein the specifying means specifies the signal value indicating an upper limit value of the output dynamic range of each of the plurality of HDR images according to an imaging mode that was set when the image was captured.

10. 10. The image processing device according to claim 1, wherein the signal value indicating the upper limit of the output dynamic range identified by the identification means for each of the plurality of HDR images is lower than a maximum signal value at a bit depth of the image.

11. 11. The image processing apparatus according to claim 1, wherein the signal region for which the combining ratio is determined by the determining means is a luminance region.

12. 12. The image processing apparatus according to claim 1, wherein the signal range for which the blending ratio is determined by the determining means is a signal range of any one of color signals.

13. a synthesis means for synthesizing the plurality of HDR images based on the synthesis ratio determined by the determination means to generate a synthesis image; an output means for outputting the composite image generated by the composition means as an image file; and The output means outputs the image file including information on a signal value indicating an upper limit of an output dynamic range of the composite image.

13. The image processing device according to claim 1,

14. the specifying means specifies a signal value indicating an upper limit value of the output dynamic range of an HDR image captured with the lowest exposure among the plurality of HDR images; 14. The image processing device according to claim 13, wherein an upper limit value of an output dynamic range of the composite image is determined based on a signal value indicating an upper limit value of the output dynamic range of an HDR image captured at the lowest exposure among the plurality of HDR images and an exposure amount of the image.

15. 15. The image processing apparatus according to claim 1, wherein the amount of exposure is determined based on an aperture, a shutter speed, and an ISO sensitivity when the image is captured.

16. 16. The image processing device according to claim 1, wherein each of the plurality of HDR images is an image expressed using a perceptual quantization (PQ) method.

17. An image processing device according to any one of claims 1 to 16, an imaging means for capturing the plurality of HDR images with different exposure amounts; An imaging device comprising:

18. A control method for an image processing device that synthesizes a plurality of HDR (High Dynamic Range) images captured with different exposure amounts, the HDR images having scene luminance expressed in an absolute luminance format, comprising: identifying a signal value indicating an upper limit of an output dynamic range for each of the plurality of HDR images; a determination step of determining a synthesis ratio of the plurality of HDR images based on the signal value identified in the identification step; acquiring a reference signal threshold value for switching a tendency of the synthesis ratio for each of the plurality of HDR images; having a signal threshold acquired in the acquisition step being changed based on the signal value indicating an upper limit value of the output dynamic range of each of the plurality of HDR images, and the combining ratio being determined based on the changed signal threshold,

19. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 16.

Images