Image processing device, control method, and program

Abstract

To provide an image processing system that generates a composite image obtained by suitably combining images with different dynamic ranges.SOLUTION: An image processing system includes: first acquisition means that acquires a first image and a second image having a wider dynamic range than the first image; transformation means that generates a transformed image by transforming the dynamic range of the first image; and generation means that generates a composite image by combining the transformation image generated by the transformation means with the second image. The transformation means determines, on the basis of the second image, a transformation range, and transforms the dynamic range of the first image into the transformation range.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a control method, and a program, and particularly to a technique for synthesizing images with different dynamic ranges.

Background Art

[0002] There is a technique for generating a composite image with a multiple exposure expression in which images appearing in each captured image are superimposed by synthesizing a plurality of captured images. Composite methods related to multiple exposure expressions include addition synthesis in which pixel values at the same position of a plurality of images to be synthesized are added, comparison bright synthesis in which the pixel value of the higher brightness is used as the image value after synthesis, and comparison dark synthesis in which the pixel value of the lower brightness is used as the pixel value after synthesis.

[0003] General sRGB 8-bit images such as JPEG are standard dynamic range (SDR) images, and the luminance of the imaging scene (scene luminance) is represented by pixel values in the range of 0 to 255. When such SDR images are synthesized to obtain a composite image with a multiple exposure expression, the output composite image is also an SDR image in which each pixel value is represented in the range of 0 to 255.

[0004] In recent years, the performance of light-emitting elements such as LEDs has improved, and a display device called an HDR display with a wider dynamic range of display luminance than before has appeared, and it is possible to display an image with a tone expression corresponding to a wider dynamic range than an SDR image. Therefore, among imaging devices, there are also those that can record high dynamic range (HDR) images so that the colors and details in each luminance range can be confirmed on such a display device. Such an HDR image generally has a scene luminance converted to a display luminance in the range of 10 bits, that is, 0 to 1023, as a pixel value.

[0005] By the way, because HDR images and SDR images have different pixel resolutions, they cannot be simply combined when generating composite images such as multiple exposure representations. Patent Document 1 discloses a technology for a display device capable of displaying SDR and HDR images that generates and outputs an HDR composite image by mapping the brightness range of the SDR image to the brightness range of the HDR image, in order to display the combined image in a suitable state. Specifically, Patent Document 1 employs a method in which the image processing device acquires information indicating the brightness range, such as the maximum brightness unique to the display device, converts the SDR image to the gradation of the HDR brightness range, and combines it with the HDR image. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2019-103041 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] By the way, the signal characteristics representing the relationship between the video signal level and display brightness in HDR images are defined by the EOTF (Electro-Optical Transfer Function), and the following two methods are employed. One is the HLG (Hybrid Log Gamma) method, which is standardized in ARIB STD-B67, and converts the video signal level to a relative value of the display brightness, resulting in a display brightness corresponding to the maximum brightness that the display device can output. The other is the PQ (Perceptual Quantization) method, which is standardized in SMPTE ST 2084 or ITU-R BT.2100, and converts the video signal level to a maximum of 10,000 nits (or cd / m²). 2The display brightness is converted to an absolute value within the range of ). Therefore, when displaying an HDR image obtained by capturing a scene, in the former method, the scene brightness is converted to a display brightness corresponding to the maximum brightness that the display device can output, while in the latter method, the scene brightness is converted to a display brightness that is absolutely determined regardless of the display device. Consequently, when considering display on a display device employing the PQ method, for example, in the encoding of the imaging device, it is necessary to convert the image signal of the scene brightness to show an absolute brightness value and generate an HDR image.

[0008] Therefore, in PQ encoding, which absolutely represents scene brightness, even if the same scene is captured, the peak brightness (maximum display brightness, maximum output dynamic range) included in the HDR image may change depending on the exposure amount. This is because the scene brightness at which the sensor output saturates changes according to the exposure amount, and therefore, different gamma curves are used for conversion in order to assign absolute display brightness to the same scene brightness. For example, as shown in Figure 1, the input / output characteristics (relationship between the number of input stages and output brightness) for two different imaging modes with different exposure amounts have different peak brightness (maximum output brightness). Here, the input / output characteristics 11 in the high exposure imaging mode are shown by a solid line, and the input / output characteristics 12 in the low exposure imaging mode are shown by a dashed line. As shown in the figure, the two imaging modes show common input / output characteristics except in the high brightness region, and are converted to the same display brightness regardless of the exposure amount, whereas in the high brightness region, the peak brightness differs between values ​​13 and 14 according to the difference in brightness at which saturation occurs. The value 15 represents the maximum value in 10 bits (1023), which corresponds to the maximum display brightness of 10,000 nits in the PQ method.

[0009] In other words, the peak brightness of a PQ-type HDR image can change depending on the imaging mode used for imaging, so its saturation level does not necessarily coincide with the maximum display brightness of the PQ method or the maximum brightness of the display device. Therefore, even if an SDR image and an HDR image are combined using the method described in Patent Document 1, there is a possibility that a suitable combined image may not be generated, for example, the high-brightness side of the SDR image may appear unnaturally bright.

[0010] As illustrated above, when performing image synthesis using images whose peak brightness may vary depending on the imaging mode used, there is a problem that a suitably synthesized image may not be generated.

[0011] The present invention has been made in view of the above-mentioned problems, and aims to provide an image processing apparatus, a control method, and a program for generating a composite image by suitably combining images with different dynamic ranges. [Means for solving the problem]

[0012] To achieve the aforementioned objectives, the image processing apparatus of the present invention is: The first form is The first image and the first format Wider dynamic range The second form is A first acquisition means for acquiring a second image and the dynamic range of the first image In the composite range conversion did The system comprises a conversion means for generating a converted image, and a generation means for generating a composite image by combining the converted image generated by the conversion means with a second image, wherein the conversion means generates a second image The peak luminance value of the dynamic range that can be expressed in the format is set as the maximum value of the composite range, and the maximum signal level of the dynamic range that can be expressed in the converted image becomes the signal level corresponding to the maximum value of the composite range. The method is characterized by converting the dynamic range of the first image to the composite range. [Effects of the Invention]

[0013] With this configuration, the present invention makes it possible to generate a composite image by suitably combining images with different dynamic ranges. [Brief explanation of the drawing]

[0014] [Figure 1] A diagram illustrating the input / output characteristics of two imaging modes with different exposure levels. [Figure 2] Block diagram illustrating the hardware configuration of the image processing apparatus 100 according to embodiments and modified examples of the present invention. [Figure 3] Block diagram illustrating the modular configuration of the synthesis process according to embodiments and modified examples of the present invention. [Figure 4] Flowchart illustrating the synthesis process according to an embodiment of the present invention [Figure 5] Diagram illustrating various transfer functions employed in the synthesis process according to an embodiment of the present invention [Figure 6] Diagram illustrating the input / output characteristics in the range conversion module 305 according to an embodiment and a modified example of the present invention [Figure 7] Diagram illustrating the output file format of the synthesized image according to an embodiment and a modified example of the present invention [Figure 8] Diagram illustrating the MaxDRL determined according to the imaging mode

Embodiments for Carrying Out the Invention

[0015] [Embodiment] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and duplicate explanations are omitted.

[0016] One embodiment described below is an example of applying the present invention to an image processing apparatus that can synthesize one HDR image encoded in the PQ format and one SDR image to generate a synthesized image with multiple exposure expressions as an example of an image processing apparatus. However, the present invention is applicable to any device capable of synthesizing images with different dynamic ranges to generate a synthesized image.

[0017] In this specification, unless otherwise specified, the "HDR image" is described as an image (PQ signal) to which the encoding of the PQ (Perceptual Quantization) method standardized by ITU-R BT.2100 is applied and the absolute display luminance can be obtained by decoding. On the other hand, the "SDR image" is described as an image encoded according to the ITU-R BT.709 standard. In the present embodiment, both the HDR image and the SDR image used for generating the composite image are described as actual captured images obtained by capturing an imaging scene. Also, in the following description, the composite image is assumed to be a PQ signal similar to the HDR image.

[0018] Here, at least 10-bit depth is required to represent the HDR characteristics by a PQ signal, and it is assumed that the HDR image and the composite image are not files storing 8-bit images such as the JPEG format. For example, the HDR image and the composite image may be HEIF files stored using a container of the High Efficiency Image File Format (hereinafter referred to as HEIF), which is an image file format developed by the Moving Picture Experts Group (MPEG) and defined in MPEG-H Part 12 (ISO / IEC 23008-12). HEIF can store not only the main image but also metadata such as thumbnails, a plurality of temporally related images, EXIF, and XMP in one file. Also, since HEIF can store a 10-bit image sequence encoded by HEVC, it can be said that it is suitable for storing the HDR image and the composite image in the present invention.

[0019] 《Hardware Configuration of Image Processing Apparatus》 FIG. 2 is a block diagram showing the hardware configuration of the image processing apparatus 100 according to the present embodiment. As shown in the figure, the image processing apparatus 100 of the present embodiment includes a CPU 101, a ROM 102, a RAM 103, a recording device 104, an operation I / F 105, a display device 106, and a system bus 107.

[0020] The CPU 101 performs overall control of the image processing unit 100. The ROM 102 is a storage device that stores control programs such as the BIOS necessary for starting the image processing unit 100, as well as programs, parameters, and data that do not require modification. The RAM 103 is a storage device that has a work area for the CPU 101, a primary storage area for temporarily storing various data, and a load area for various programs.

[0021] The recording device 104 is a recording device that stores various data such as the OS program, various control programs, various software programs that can be executed on the OS program, HDR images used in the synthesis process described later, and synthesized images obtained as a result of the synthesis process. The recording device 104 includes, for example, a hard disk or flash memory that is built into or detachably connected to the image processing device 100, a detachably connected flexible disk or optical disk, magneto-optical disk, IC card, memory card, etc. Therefore, the CPU 101 can control the image processing device 100 by loading the various programs stored in the ROM 102 or the recording device 104 into the RAM 103 and executing them.

[0022] The operation I / F 105 is a user interface provided by the image processing device 100, such as a keyboard, mouse, or touch panel. When the operation I / F 105 detects that a user has made an operation input, it sends a control signal related to that operation to the CPU 101. The display device 106 is, for example, a liquid crystal display, and displays various information, including a graphical user interface related to the OS and software running on the image processing device 100. The system bus 107 connects each block that makes up the image processing device 100 so that they can communicate with each other.

[0023] The image processing device 100 may be a dedicated device for developing and editing captured RAW images, in which case the synthesis processing program can be stored in the ROM 102. Furthermore, a general-purpose personal computer (PC) can function as the image processing device 100 by having the CPU 101 load the synthesis processing program stored in the recording device 104 into the RAM 103. Additionally, an imaging device capable of acquiring RAW images through imaging can also function as the image processing device 100 by having its control unit execute the synthesis processing program, or by having an image processing unit that performs the synthesis processing perform a predetermined operation.

[0024] Synthesis process Next, the synthesis process performed in the image processing apparatus 100 according to this embodiment, which generates a composite image of multiple exposure representation by combining one HDR image and one SDR image, will be described in detail with reference to the figures. This synthesis process is realized by the CPU 101 reading the synthesis process program from the ROM 102 or recording device 104, expanding it in the RAM 103, and executing it. Figure 3(a) illustrates the module configuration of various functions realized by the CPU 101 during the execution of the synthesis process in this embodiment, and Figure 4 is a diagram illustrating a flowchart for explaining the flow of the synthesis process.

[0025] As described above, the SDR image and HDR image used as input for the composite processing are images captured using an imaging device, and the latter has a wider dynamic range so that it can express the gradation of scene brightness in more detail. Each image is developed in the imaging device where it was captured by encoding the brightness of the subject into a nonlinear signal using OETF (Opt-Electronic Transfer Function). Furthermore, the HDR image will be described as a PQ image encoded using the PQ method (hereinafter sometimes referred to as an HDRPQ image). Therefore, in this embodiment, the input for the composite processing is the SDR image 301 and the HDRPQ image 302 as shown in Figure 3(a). Here, the OETF is defined in the imaging device for development, and separate OETFs may be provided for SDR and HDR.

[0026] In step S401, the CPU 101 reads the SDR image 301 and HDRPQ image 302 from the recording device 104 and expands them into RAM 103. This step enables the input of the images to be synthesized.

[0027] As described above, SDR image 301 and HDRPQ image 302 are images that represent scene brightness with different dynamic ranges, so it is preferable to match the dynamic range when combining the images. In other words, SDR image 301 and HDRPQ image 302 have different ranges of possible signal levels and cannot be simply used for combining, so it is necessary to match the dynamic range of both. More specifically, SDR image 301 and HDRPQ image 302 are images that differ in how they are generated, using either an OETF for SDR or an OETF for HDR to convert the brightness of the subject into a nonlinear signal.

[0028] In the synthesis process of this embodiment, the SDR image 301 is converted into an image (converted image) that is represented with a wider dynamic range of the HDRPQ image 302, and synthesis is performed using the converted image and the HDRPQ image 302. For this reason, in the processes S402 to S406 that follow, the SDR image 301 is converted into an image (PQ format image (HDRPQ image)) with the same dynamic range as the HDRPQ image 302.

[0029] In S402, the SDR de-gamma module 303 linearizes the SDR image 301 to be synthesized. As described above, the SDR image 301 is a nonlinear signal obtained by applying an OETF for SDR. Therefore, the SDR de-gamma module 303 converts the signal level of each pixel in the SDR image 301 into a linear luminance value of the display light by applying an OETF for SDR, for example, as shown in Figure 5(a). That is, the SDR de-gamma module 303 transforms the SDR image 301 and generates a first intermediate image in which each pixel represents the display luminance.

[0030] Here, the EOTF for SDR may be the reference EOTF defined in ITU-R BT.709. Alternatively, the inverse function of the OETF for SDR used in the imaging device that captured the SDR image 301 may be used. The OETF is the product of the OOTF (Opt-Optical Transfer Function) based on the intended output image, which is set in advance for each exposure setting or set by the user, and the inverse function of the reference EOTF. The EOTF for SDR used for linearization may be recorded in the recording device 104 of the image processing device 100, for example, or it may be acquired from the imaging device in relation to the SDR image 301.

[0031] In S403, the color space conversion module 304 converts the color space of the first intermediate image using the SDR degamma module 303 as needed. In this embodiment, since the HDRPQ image 302 is an HDR image in PQ format, the color space conversion module 304 performs a color space conversion from BT.709 to BT.2020. Details of the color space conversion are described in ITU-R BT.2087.

[0032] In S404, the CPU 101 determines the maximum value of the dynamic range to which the SDR image 301 is converted. In the synthesis process of this embodiment, the SDR image 301 is converted to a dynamic range common to the HDRPQ image 302, so in this step, the maximum value of the converted dynamic range (hereinafter referred to as the synthesis range) is determined. As described above with respect to Figure 1, the peak brightness value of the PQ format HDR image may differ depending on the exposure amount, so in this step, the CPU 101 obtains the peak brightness value of the HDRPQ image 302 and determines this as the maximum value of the synthesis range. That is, the CPU 101 adopts the maximum signal level of the HDRPQ image 302 as the signal level corresponding to the maximum value of the synthesis range so that the synthesized image does not become unnatural due to the synthesis of an image with a signal level exceeding the maximum signal level of the HDRPQ image 302.

[0033] Here, the peak brightness of the HDRPQ image 302 may be included in the image file (HEIF file, etc.) related to the HDRPQ image 302, or it may be associated with the HDRPQ image 302 as a separate file. In this embodiment, the image file related to the HDRPQ image 302 has MaxDRL (Maximum Dynamic Range Level) as metadata as described in Japanese Patent Application Publication No. 2020-039118, and the CPU 101 acquires this as the peak brightness value. MaxDRL is the maximum value of the PQ signal output value after applying the corresponding OETF in the development and encoding of the captured image signal, and it indicates the peak brightness value (upper limit) of the dynamic range that can be represented by the HDRPQ image 302. Therefore, in this embodiment it is described as the peak brightness value, but MaxDRL may also be a value that indicates the nit value corresponding to the peak brightness value.

[0034] In S405, the range conversion module 305 generates a second intermediate image by converting the dynamic range of the first intermediate image to a composite range with the maximum value determined in S404. In this embodiment, when generating the second intermediate image, the range conversion module 305 converts the dynamic range to a composite range by converting the luminance values ​​of the input / output characteristics shown in Figure 6(a). In the example in Figure 6(a), the luminance value of the input (first intermediate image) is set on the horizontal axis and the luminance value of the output (second intermediate image) is set on the vertical axis, showing the input / output relationship. As shown in the figure, the input / output characteristics in the range conversion module 305 linearly scale the value range of the luminance value of the first intermediate image to a range where the maximum value is the MaxDRL (peak luminance value) of the HDRPQ image 302. Here, the value range of the luminance value of the first intermediate image is defined as the maximum value, i.e., the peak luminance value, based on the information of the pixels indicating white in the SDR image 301. In other words, the range conversion module 305 linearly transforms the luminance value of the first intermediate image so that its peak luminance value becomes the MaxDRL of the HDRPQ image 302, thereby deriving the luminance value of each pixel in the second intermediate image and generating a second intermediate image with a transformed dynamic range. More specifically, the range conversion module 305 derives a conversion coefficient related to the transformation of the dynamic range based on the peak luminance value of the first intermediate image and the MaxDRL of the HDRPQ image 302. Then, the range conversion module 305 generates the second intermediate image by storing the luminance values ​​obtained by multiplying the luminance value of each pixel in the first intermediate image by the said conversion coefficient as each pixel in the second intermediate image.

[0035] In S406, the HDRPQ gamma module 306 applies gamma to the second intermediate image to convert (encode) the luminance values ​​into a nonlinear signal, thereby generating a converted image. In this embodiment, since the HDRPQ image 302 that is combined with the converted image is a PQ format image, the gamma applied in the HDRPQ gamma module 306 is an HDR OETF corresponding to the PQ format, as shown in Figure 5(b). Here, the HDR OETF can be obtained and used, for example, from the OETF used to encode the HDRPQ image 302, which is included in the metadata of the image file related to the HDRPQ image 302. Alternatively, the HDR OETF may be derived, for example, by multiplying the OOTF defined for the exposure settings related to the imaging of the HDRPQ image 302 by the inverse function of the reference EOTF defined in ITU-R BT.2100.

[0036] In S407, the synthesis module 307 generates a composite image 309 of the multiple exposure representation by combining the converted image generated in S406 with the HDRPQ image 302, thereby completing the synthesis process. Since there are multiple synthesis methods for multiple exposures, such as addition, lightening, and darkening, the synthesis process in this embodiment accepts a synthesis method 308 that uniquely identifies the synthesis method as another input to the SDR image 301 and the HDRPQ image 302. The input synthesis method 308 may be specified based on user operation input made via the operation I / F 105, or it may be specified by reading pre-set information. Accordingly, the synthesis module 307 generates the composite image 309 by executing processing based on the input synthesis method 308.

[0037] The generated composite image 309 may be stored in the recording device 104 as an HEIF format image file, for example, as a new PQ image. An HEIF format image file has a file structure as shown in Figure 7(a). That is, an HEIF format file consists of an ftyp box 701 indicating the file format, a meta box 702 storing metadata including a thumbnail image 711, and an mdat box 703 storing encoded data (main image 712). Therefore, the synthesis module 307 stores the data of the generated composite image 309 in the mdat box 703, stores the thumbnail image of the composite image 309 and the MaxDRL related to the composite image 309 in the meta box 702, and generates the file.

[0038] Here, the MaxDRL for the composite image 309 included in the metadata may be, for example, the MaxDRL of the input image, the HDRPQ image 302. Alternatively, the MaxDRL for the composite image 309 may be read from the recording device 104, for example, the maximum signal PQ code value corresponding to the imaging mode set when the HDRPQ image 302 was captured, as shown in the table in Figure 8, and included in the metadata as the MaxDRL. In addition, the metadata may include OETF and other information used for linearizing the composite image 309.

[0039] Furthermore, the file format of the composite image 309 output after the synthesis process only needs to be able to store at least the MaxDRL related to the composite image 309, and for example, the MP4 format as shown in Figure 7(b) may be adopted. In this case, the MaxDRL should be stored in the meta data 722 in the mdat box 721.

[0040] Furthermore, although this embodiment describes the composite image, which is the output of the synthesis process, as a PQ format HDR image, the implementation of the present invention is not limited to this, and the output may also be an image converted to an SDR image.

[0041] As described above, the image processing apparatus of this embodiment can generate a composite image by suitably combining images with different dynamic ranges. More specifically, when combining an SDR image and an HDRPQ image, the SDR image is converted to a dynamic range based on the peak brightness of the HDRPQ image. Therefore, even if the SDR image does not have a defined absolute brightness, it can be combined in a manner that is suitable for the HDRPQ image.

[0042] [Example 1] In the embodiments described above, the input / output characteristics of the range conversion module 305 were described in which the peak luminance value of the first intermediate image is simply linearly assigned to the value range up to MaxDRL of the HDRPQ image 302, as shown in Figure 6(a). However, the implementation of the present invention is not limited to this, and the input / output characteristics may be set with respect to a predetermined knee point for the luminance value of the first intermediate image, as shown in Figure 6(b), and different characteristics may be set for the value range of the luminance value before and after that point.

[0043] In the example shown in Figure 6(b), the knee point is set to the luminance value corresponding to the properly exposed subject included in the SDR image 301. More specifically, the input / output characteristics in Figure 6(b) show a characteristic represented by a straight line with a slope of 1 for inputs of luminance values ​​in the value range from 0, which is the lower limit of luminance values, to the luminance value corresponding to proper exposure (proper luminance value) (first value range). That is, when a luminance value in the first value range is input, the same value is output. Furthermore, for inputs of luminance values ​​in the value range from the proper luminance value to the peak luminance value of the first intermediate image (second value range), the input / output characteristics output a value linearly assigned to the value range from the proper luminance value to the MaxDRL of the HDRPQ image 302. That is, the slope of the input / output characteristics related to the second value range is adaptively changed according to the proper luminance value related to the SDR image 301. By employing such input / output characteristics, the range conversion module 305 can generate a second intermediate image in a manner that maintains the tonal expression up to proper exposure in the first intermediate image.

[0044] Therefore, in the module configuration for the synthesis process related to this modified example, as shown in Figure 3(b), information on the appropriate brightness value acquired in association with the SDR image 301 is input to the range conversion module 305.

[0045] Incidentally, when a transformation is performed such that the input / output characteristics change at the knee point as shown in Figure 6(b), there is a range of values ​​in which the contrast becomes stronger depending on the luminance value at the knee point, and low-amplitude noise that was difficult to see before the transformation may be emphasized after the transformation. The composite image of the multiple exposure representation obtained by combining the transformed image based on the second intermediate image obtained in this way may not be a suitable representation because the noise level and resolution differ between the transformed image and the HDRPQ image 302.

[0046] Therefore, in the synthesis process of this modified example, as shown in Figure 3(b), the image quality adjustment module 310 applies image quality adjustment processing according to the brightness value of the knee point to the second intermediate image to which gamma by the HDRPQ gamma module 306 has been applied, and then generates a converted image. The image quality adjustment processing may be, for example, noise reduction processing, and its intensity is controlled according to the brightness value of the knee point. For example, if the brightness value of the knee point is close to the peak brightness value of the first intermediate image, that is, if the slope of the input / output characteristics related to the second value range is steep, the intensity of noise reduction is increased. Conversely, if the slope of the input / output characteristics related to the second value range is gentle, the intensity of noise reduction is decreased.

[0047] By doing so, the impact of image quality degradation that may occur during dynamic range conversion can be reduced, making it possible to match the noise level and resolution of the converted image and the HDRPQ image 302 to generate a composite image with a suitable representation.

[0048] Furthermore, the image quality adjustment processing applied by the image quality adjustment module 310 is not limited to noise reduction processing inputs. For example, if the SDR image 301 is a so-called noiseless image such as computer graphics, the image quality adjustment processing may be a process that adds random noise to match the noise level of the HDRPQ image 302, rather than noise reduction processing.

[0049] Furthermore, although the input / output characteristics of the range conversion module 305 were described as being in the form of a broken line as shown in Figure 6(b) in this modified example, the implementation of the present invention is not limited to this, and may also be in the form of a curve. In this case, the image quality adjustment processing performed by the image quality adjustment module 310 should also be configured to be adjusted according to the input / output characteristics.

[0050] [Differentiation 2] In the embodiments and modifications described above, a method for generating a composite image of multiple exposure representation by combining one HDRPQ image and one SDR image has been explained. However, there may be multiple images of each dynamic range used as input for the compositing process. In this case, if multiple HDRPQ images are input as the target for compositing, the peak luminance value set as the maximum value of the compositing range may be the largest value among the MaxDRL of the multiple HDRPQ images.

[0051] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0052] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0053] 101: CPU, 102: ROM, 103: RAM, 104: Recording device, 303: SDR de-gamma module, 304: Color space conversion module, 305: Range conversion module, 306: HDRPQ gamma module, 307: Compositing module, 310: Image quality adjustment module

Claims

1. A first acquisition means for acquiring a first image which is a first format and a second image which is a second format having a wider dynamic range than the first format, A conversion means for generating a converted image obtained by converting the dynamic range of the first image to a composite range, A generation means that generates a composite image by combining the converted image generated by the conversion means with the second image, It has, The conversion means converts the dynamic range of the first image to the composite range such that the peak luminance value of the dynamic range expressible in the second format is the maximum value of the composite range, and the maximum signal level of the dynamic range expressible in the converted image corresponds to the signal level of the maximum value of the composite range. An image processing apparatus characterized by the following:

2. The image processing apparatus according to claim 1, characterized in that the peak brightness value of the dynamic range that can be expressed in the second form differs depending on the exposure amount at the time of imaging.

3. The image processing apparatus according to claim 1 or 2, characterized in that the second image is an HDR (High Dynamic Range) image encoded using the PQ (Perceptual Quantization) method standardized by ITU-R BT. 2100.

4. The first image and the second image are images obtained by converting the brightness of the subject into a nonlinear signal by applying a gamma corresponding to their respective dynamic ranges. The conversion means is By applying a degamma corresponding to the dynamic range expressible in the first format, the signal of the first image is converted into a linear luminance value. The linear luminance value is converted such that its maximum value becomes the peak luminance value of the dynamic range that can be expressed in the second format, The converted image is generated by applying a gamma corresponding to the dynamic range expressible in the second format and converting it into a nonlinear signal. The image processing apparatus according to feature 1.

5. The conversion means is A first intermediate image is generated by applying a degamma corresponding to the dynamic range expressible in the first format to convert the signal of the first image into linear luminance values. A second intermediate image is generated by assigning the luminance values ​​of the first intermediate image to a value range whose maximum value is the peak luminance value of a dynamic range that can be expressed in the second format. The converted image is generated by applying a gamma corresponding to the dynamic range expressible in the second format to convert the luminance values ​​of the second intermediate image into a nonlinear signal. The image processing apparatus according to feature 4.

6. The image processing apparatus according to claim 5, characterized in that the input / output characteristics for generating the second intermediate image are such that the range of luminance values ​​of the first intermediate image is scaled to a range where the maximum value is the peak luminance value of the second image.

7. The image processing apparatus according to claim 5, characterized in that the input / output characteristics for generating the second intermediate image exhibit different characteristics for inputs to a first value range and a second value range, respectively, which are determined with respect to a predetermined knee point for the luminance value of the first intermediate image.

8. The system further includes a second acquisition means for acquiring a brightness value corresponding to the proper exposure for the first image, The predetermined knee point is a luminance value corresponding to the appropriate exposure obtained by the second acquisition means, The first value range is a range of values ​​from the lower limit of the luminance value to the luminance value corresponding to the appropriate exposure. The second value range is the range from the luminance value corresponding to the correct exposure to the peak luminance value of the first intermediate image. The image processing apparatus according to feature 7.

9. The image processing apparatus according to claim 8, characterized in that the input / output characteristics for generating the second intermediate image output the same value for input of a luminance value in the first value range, and for input of a luminance value in the second value range, output a value scaled to a value range from the luminance value corresponding to the appropriate exposure to the peak luminance value of the second image.

10. The image processing apparatus according to any one of claims 7 to 9, characterized in that the conversion means converts the luminance value of the second intermediate image into a nonlinear signal, and then performs image quality adjustment processing according to the luminance value of the predetermined knee point to generate the converted image.

11. The image processing apparatus according to any one of claims 3 to 10, characterized in that the conversion means obtains the peak brightness value of the second image from the metadata associated with the second image.

12. The second images used to generate the composite image are multiple, The conversion means determines the maximum value of the composite range to be the maximum value among the peak brightness values ​​of the plurality of second images. The image processing apparatus according to any one of claims 3 to 11.

13. The image processing apparatus according to any one of claims 1 to 12, characterized in that the first image is an SDR (Standard Dynamic Range) image encoded in the ITU-R BT. 709 standard.

14. A method for controlling an image processing device, A step of acquiring a first image in a first format and a second image in a second format having a wider dynamic range than the first format, A conversion step that generates a converted image by converting the dynamic range of the first image to a composite range, A step of generating a composite image by combining the converted image generated in the conversion step with the second image, It has, In the conversion step, the peak luminance value of the dynamic range expressible in the second format is set as the maximum value of the composite range, and the dynamic range of the first image is converted to the composite range such that the maximum signal level of the dynamic range expressible in the converted image corresponds to the signal level of the maximum value of the composite range. A control method characterized by the following:

15. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 13.

Images