Image processing device, control method, and program

Abstract

To generate a composite image indicating a preferable brightness expression when performing additive composition of a plurality of images including one or more HDR images.SOLUTION: An image processing apparatus for generating a composite image, includes: acquisition means for acquiring a plurality of images including one or more High Dynamic Range (HDR) images; determination means for determining a peak luminance value of the composite image; and generation means for generating the composite image by executing additive composition processing using the plurality of images. The generation means controls the additive composition processing so that a signal level of each pixel of the composite image falls within an output dynamic range whose maximum value is set to a signal level corresponding to the peak luminance value determined by the determination means.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a control method, and a program, and particularly to a technique for generating a composite image using an HDR image.

Background Art

[0002] There is an addition synthesis method for generating a composite image with multiple exposure expressions. In addition synthesis, the pixel value of each pixel of a plurality of images to be synthesized is added together, and the pixel value of the corresponding pixel of the composite image is determined.

[0003] A general sRGB 8-bit image such as JPEG is an image with a standard dynamic range (SDR: Standard Dynamic Range), and the luminance of the imaging scene (scene luminance) is represented by pixel values in the range of 0 to 255. When such SDR images are added and synthesized to obtain a composite image with multiple exposure expressions, the output composite image is also an SDR image in which each pixel value is represented in the range of 0 to 255. Since an SDR image sometimes relatively represents the brightness of a subject, when the SDR image obtained by addition synthesis is displayed on a display device, the brightness is not likely to show a significant difference compared to the images to be synthesized.

[0004] On the other hand, in recent years, the performance of light-emitting elements such as LEDs has improved, and a display device called an HDR display with a wider dynamic range of display luminance than before has emerged, and it is possible to display an image with a gradation expression corresponding to a wider dynamic range than an SDR image. Therefore, among imaging devices, there are also those that can record high dynamic range (HDR: High Dynamic Range) images so that the colors and details in each luminance range can be confirmed in such a display device. Such an HDR image generally has a scene luminance converted to a display luminance in the range of 10 bits, that is, 0 to 1023, as a pixel value.

[0005] By the way, the signal characteristics representing the relationship between the video signal level and display brightness in HDR images are defined by the EOTF (Electro-Optical Transfer Function), and the following two methods are employed. One is the HLG (Hybrid Log Gamma) method, which is standardized in ARIB STD-B67, and converts the video signal level to a relative value of the display brightness, resulting in a display brightness corresponding to the maximum brightness that the display device can output. The other is the PQ (Perceptual Quantization) method, which is standardized in SMPTE ST 2084 or ITU-R BT.2100, and converts the video signal level to a maximum of 10,000 nits (or cd / m²). 2 The display brightness is converted to an absolute value within the range of ). Therefore, when displaying an HDR image obtained by capturing a scene, in the former method, the scene brightness is converted to a display brightness corresponding to the maximum brightness that the display device can output, while in the latter method, the scene brightness is converted to a display brightness that is absolutely determined regardless of the display device. Consequently, when considering display on a display device employing the PQ method, for example, in the encoding of the imaging device, it is necessary to convert the image signal of the scene brightness to show an absolute brightness value and generate an HDR image.

[0006] Therefore, in PQ encoding, which absolutely represents scene brightness, the peak brightness value (maximum display brightness, maximum output dynamic range) included in the HDR image may change even when capturing the same scene. This is because the scene brightness at which the sensor output saturates changes depending on the imaging mode, and therefore the gamma curve used for conversion differs in order to assign absolute display brightness to the same scene brightness. For example, as shown in Figure 1, the input / output characteristics (relationship between input stage and output brightness) for two imaging modes with different exposure levels show different peak brightness values ​​(maximum output brightness). Here, the input / output characteristics 11 in the high exposure imaging mode are shown by a solid line, and the input / output characteristics 12 in the low exposure imaging mode are shown by a dashed line. As shown in the figure, the two imaging modes show common input / output characteristics except in the high brightness region, and are converted to the same display brightness regardless of the exposure level, whereas in the high brightness region, the peak brightness values ​​differ between values ​​13 and 14 depending on the difference in brightness at which the sensor saturates. The value 15 represents the maximum value in 10 bits (1023), which corresponds to the maximum display brightness of 10,000 nits in the PQ method.

[0007] Therefore, the images to be combined in multiple exposure representations are not limited to SDR images, but can also include HDR images with different dynamic ranges. Furthermore, as mentioned above, PQ-type HDR images (hereinafter referred to as HDRPQ images) with different peak luminance values, i.e., maximum values ​​of the output dynamic range, can also be combined. Patent Document 2 discloses a method for generating a display image by combining images with different dynamic ranges, in which the dynamic range of the overlay image is assigned to a range that perceptually matches the overlaid image. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Special Publication No. 2016-515327 [Overview of the Initiative] [Problems that the invention aims to solve]

[0009] However, the method disclosed in Patent Document 1 primarily assumes that one of the images with adjusted dynamic range is predominantly displayed, and does not consider the possibility that the output dynamic range will change between the image being combined and the combined image due to additive blending. Therefore, for example, when adding together images with different assumed maximum output brightness values, or images that do not show 1023 in 10 bits, there was a possibility that some of the subject images would appear with a brightness different from what was expected when displayed.

[0010] The present invention has been made in view of the above-mentioned problems, and aims to provide an image processing apparatus, a control method, and a program that generate a composite image exhibiting suitable brightness representation when adding and combining multiple images including one or more HDR images. [Means for solving the problem]

[0011] To achieve the aforementioned objective, the present invention provides an image processing apparatus for generating a composite image, comprising: acquisition means for acquiring a plurality of images including one or more HDR (High Dynamic Range) images; determination means for determining the peak brightness value of the composite image; and generation means for generating a composite image by performing an additive synthesis process using the plurality of images. Additive synthesis is a process that simply adds the signal levels of pixels at corresponding positions in multiple images. The generation means ensures that the signal level of each pixel in the composite image falls within an output dynamic range whose maximum value is the signal level corresponding to the peak luminance value determined by the determination means. Generate a composite image It is characterized by doing so. [Effects of the Invention]

[0012] With this configuration, according to the present invention, when adding and combining multiple images including one or more HDR images, it is possible to generate a composite image that exhibits a suitable brightness representation. [Brief explanation of the drawing]

[0013] [Figure 1]A diagram for explaining input / output characteristics of two imaging modes with different exposure amounts [Figure 2] A block diagram illustrating the hardware configuration of the image processing apparatus 100 according to embodiments and modified examples of the present invention [Figure 3] A flowchart exemplifying the synthesis process executed by the image processing apparatus 100 according to Embodiment 1 of the present invention [Figure 4] A diagram exemplifying HDR images to be synthesized according to embodiments and modified examples of the present invention [Figure 5] A diagram exemplifying signal characteristics of HDR images to be synthesized according to embodiments and modified examples of the present invention [Figure 6] A diagram for explaining a synthesized image according to an embodiment of the present invention [Figure 7] A flowchart diagram showing an example of image processing corresponding to Example 2 of the present invention [Figure 8] A diagram exemplifying conversion of the dynamic range of an HDR image to be synthesized according to Embodiment 2 of the present invention [Figure 9] A diagram exemplifying an image with a converted dynamic range according to Embodiment 2 of the present invention [Figure 10] A diagram exemplifying conversion of the dynamic range of an HDR image to be synthesized according to Modified Example 1 of the present invention

Embodiments for Carrying Out the Invention

[0014] [Embodiment 1] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0015] The following describes an example of applying the present invention to an image processing apparatus capable of generating a composite image with multiple exposure expressions by adding and synthesizing two HDR images encoded in the PQ method, as an example of an image processing apparatus. However, the present invention is applicable to any device capable of generating a composite image by performing an addition and synthesis process using a plurality of images including one or more HDR images.

[0016] Also, in this specification, unless otherwise specified, the "HDR image" is described as an image (PQ signal) to which encoding in the PQ (Perceptual Quantization) method standardized by ITU-R BT.2100 is applied and absolute display luminance can be obtained by decoding. In this embodiment, the images used for generating the composite image are all described as actual captured images obtained by capturing an imaging scene. Also, in the following description, the composite image is assumed to be a PQ signal like the HDR image.

[0017] Here, at least 10-bit depth is required to represent HDR characteristics with a PQ signal, and HDR images and composite images are not files that store 8-bit images such as the JPEG format. For example, HDR images and composite images may be HEIF files stored using a container of the High Efficiency Image File Format (hereinafter referred to as HEIF), an image file format developed by the Moving Picture Experts Group (MPEG) and defined in MPEG-H Part 12 (ISO / IEC 23008-12).

[0018] 《Hardware Configuration of Image Processing Apparatus》 FIG. 2 is a block diagram showing the hardware configuration of the image processing apparatus 100 according to this embodiment. As shown in the figure, the image processing apparatus 100 of this embodiment includes a CPU 101, a ROM 102, a RAM 103, a recording device 104, an operation I / F 105, a display device 106, and a system bus 107.

[0019] The CPU 101 performs overall control of the image processing unit 100. The ROM 102 is a storage device that stores control programs such as the BIOS necessary for starting the image processing unit 100, as well as programs, parameters, and data that do not require modification. The RAM 103 is a storage device that has a work area for the CPU 101, a primary storage area for temporarily storing various data, and a load area for various programs.

[0020] The recording device 104 is a recording device that stores various data such as the OS program, various control programs, various software programs that can be executed on the OS program, HDR images used in the synthesis process described later, and synthesized images obtained as a result of the synthesis process. The recording device 104 includes, for example, a hard disk or flash memory that is built into or detachably connected to the image processing device 100, a detachably connected flexible disk or optical disk, magneto-optical disk, IC card, memory card, etc. Therefore, the CPU 101 can control the image processing device 100 by loading the various programs stored in the ROM 102 or the recording device 104 into the RAM 103 and executing them.

[0021] The operation I / F 105 is a user interface provided by the image processing device 100, such as a keyboard, mouse, or touch panel. When the operation I / F 105 detects that a user has made an operation input, it sends a control signal related to that operation to the CPU 101. The display device 106 is, for example, a liquid crystal display, and displays various information, including a graphical user interface related to the OS and software running on the image processing device 100. The system bus 107 connects each block that makes up the image processing device 100 so that they can communicate with each other.

[0022] The image processing device 100 may be a dedicated device for developing and editing captured RAW images, in which case the synthesis processing program can be stored in the ROM 102. Furthermore, a general-purpose personal computer (PC) can function as the image processing device 100 by having the CPU 101 load the synthesis processing program stored in the recording device 104 into the RAM 103. Additionally, an imaging device capable of acquiring RAW images through imaging can also function as the image processing device 100 by having its control unit execute the synthesis processing program, or by having an image processing unit that performs the synthesis processing perform a predetermined operation.

[0023] Synthesis Process In the image processing apparatus 100 of this embodiment, which has the above configuration, the synthesis process that generates a composite image by adding two HDR images to be synthesized will be explained using the flowchart in Figure 3. The process corresponding to the flowchart can be realized by the CPU 101 reading the corresponding processing program stored in, for example, the ROM 102, loading it into the RAM 103, and executing it. This synthesis process will be explained assuming that it is started, for example, when an operation input related to the generation of a composite image is received via the operation I / F 105.

[0024] In S301, the CPU 101 acquires two different HDR images to be combined. The HDR images to be combined are recorded, for example, in the recording device 104, and the CPU 101 acquires them by reading them for the combination process and loading them into the RAM 103. In the following description, the two HDR images to be combined may be referred to as the first target image and the second target image, respectively.

[0025] In S302, the CPU 101 determines the peak luminance value of the composite image to be generated in this synthesis process. As described above, the composite image generated in the synthesis process is an HDR image in PQ format, and its dynamic range defines the signal level corresponding to the absolute display luminance. The peak luminance value is the maximum value of the display luminance shown by the image appearing in the composite image, that is, the maximum value of the output dynamic range. In the synthesis process of this embodiment, the CPU 101 determines the peak luminance value of the composite image based on the peak luminance values ​​of the HDR images to be synthesized (the first target image and the second target image).

[0026] Here, the peak luminance value of the HDR image to be synthesized may be included in the image file (HEIF file, etc.) related to the HDR image, or it may be associated with the HDR image as a separate file. In this embodiment, each image file related to the HDR image to be synthesized is assumed to have MaxDRL (Maximum Dynamic Range Level) as metadata as described in Japanese Patent Application Publication No. 2020-039118. MaxDRL is the maximum value of the PQ signal output value after applying the corresponding OETF in the development and encoding of the captured image signal, and indicates the peak luminance value (upper limit) of the PQ signal. In this embodiment, MaxDRL is described as indicating the signal level corresponding to the peak luminance of each HDR image, but the implementation of the present invention is not limited to this, and for example, it may indicate the nit value corresponding to the peak luminance.

[0027] The CPU 101 obtains the MaxDRL for both the first and second target images, and determines the larger of the two MaxDRL values ​​as the peak luminance value of the composite image to be generated. Hereafter, the peak luminance value determined in this step will be referred to as the "peak signal level".

[0028] In S303, the CPU 101 generates an image (hereinafter referred to as an intermediate image) by adding the first target image and the second target image together. More specifically, the CPU 101 obtains the signal levels of pixels at the same positions from the first target image and the second target image, and derives the signal level of the pixel relating to the intermediate image by adding these together (summing the signal values). The CPU 101 performs this signal level addition process for all pixels, deriving the signal levels of all pixels in the intermediate image and constructing the intermediate image. In this embodiment, in order to simplify the understanding of the invention, the first target image and the second target image are described as having the same number of pixels in the horizontal and vertical directions, but the implementation of the present invention is not limited to this. If the number of pixels in the first target image and the second target image are different, for example, one image can be scaled so that the number of pixels in the horizontal or vertical direction is the same as that of the other image, and then the additive composition can be performed. Furthermore, since intermediate images are a different concept from the images output for display, even if the signal level exceeds the maximum dynamic range of the PQ format (corresponding to 10,000 nits) due to additive blending, it is possible to maintain and store that value. In other words, the signal level of each pixel in the intermediate image simply represents the sum of the signal level of the pixel at the same position in the first target image and the signal level of the pixel at the same position in the second target image.

[0029] In S304, the CPU 101 generates a composite image by clipping the signal level of the intermediate image to the peak signal level determined in S302. In other words, the CPU 101 identifies pixels in the intermediate image whose signal level exceeds the peak signal level, and generates a composite image by changing (replacing) the signal level of those pixels to the peak signal level.

[0030] Here, the synthesis process of this embodiment will be described in general terms with reference to the figures. In the following examples, the first target image will be described as being in the manner shown in Figure 4(a), and the second target image will be described as being in the manner shown in Figure 4(b). The two images are real HDR images captured in different imaging modes, and therefore have different output dynamic ranges. Specifically, the first target image is an image of a darker subject than the second target image, and the output dynamic range of the first target image is narrower than that of the second target image, and its maximum value, i.e., the peak brightness value, is also lower.

[0031] Figures 5(a) and 5(b) show the relationship between scene brightness and display brightness (signal characteristics) for each image, illustrating the difference in peak brightness values ​​(MaxDRL). Specifically, Figure 5(a) shows the signal characteristics of the first target image, where the peak brightness value is Y1. In the first target image shown in Figure 4(a), the signal level of pixels included in region 401 is assumed to be Y1. Figure 5(b) shows the signal characteristics of the second target image, where the peak brightness value is Y2, which is higher than Y1. In the second target image shown in Figure 4(b), the signal level of pixels included in region 411 is assumed to be Y2. Note that both the vertical and horizontal axes represent signal values ​​(e.g., PQValueCode). The Ymax shown for each signal characteristic in Figure 5 represents the maximum value of the dynamic range in PQ format, i.e., the maximum display brightness (signal value corresponding to 10000 nits). The same applies to Figures 8 and 10, which will be discussed later.

[0032] In this case, the intermediate image obtained by simply adding the first target image and the second target image together will have the appearance shown in Figure 6(a). As shown in the figure, the signal level of pixels included in region 601, where region 401 and region 402 of the first target image overlap, becomes brighter through additive blending. For example, if Y1 is 712 (equivalent to 600 nits) and Y2 is 769 (1000 nits), then 712 + 769 = 1481, which exceeds the maximum display brightness of 1023. In other words, if the intermediate image is output as a composite image, when displayed using a display device capable of displaying brightness higher than 1000 nits, the brightness of region 601 will be brighter than the peak brightness value Y2 of the second target image and will be displayed at the maximum brightness, which is undesirable. That is, the composite image will not maintain the brightness and gradation expression that was present in the first and second target images before blending.

[0033] Therefore, in the synthesis process of this embodiment, by clipping the signal level at S304 to the peak signal level, the signal level of the pixels related to region 601 of the intermediate image is saturated to the peak signal level, resulting in the representation shown in Figure 6(b). More specifically, in the embodiment of Figure 4, where the peak luminance value of the second target image is higher, the peak signal level is determined to be Y2, which is the peak luminance value of the second target image. As a result, a composite image can be created in which the signal level of the pixels at the location of region 601 is set to Y2, as shown in Figure 6(b).

[0034] In S305, the CPU 101 constructs an image file associated with the peak signal level of the composite image generated in S304, records it to the recording device 104, and completes the synthesis process. Here, the image file may be in a format with a file structure capable of storing metadata, such as the HEIF format. In this case, the peak signal level information is stored in the metadata and associated with the composite image. By recording the image file of the composite image with the peak signal level information associated in this way, it becomes possible to display an image converted according to the expected dynamic range even in an SDR environment.

[0035] As described above, the image processing apparatus of this embodiment can generate a composite image that exhibits a suitable brightness representation when adding and combining multiple images, including one or more HDR images.

[0036] [Embodiment 2] In the above-described embodiment, a method was explained in which a composite image is generated in which the maximum value of the output dynamic range is limited to the peak signal level by clipping the signal level of the intermediate image, which is simply obtained by additively combining the images to be combined, at the peak signal level. On the other hand, in such a method of clipping at the peak signal level after additive combining, the gradation expression that was represented in the intermediate image in the level range from the peak signal level to the maximum value of the added signal level is lost. In this embodiment, a method is described in which a composite image is generated in which the signal level obtained as a result of additive combining is limited to the peak signal level by converting the dynamic range of each image to be combined before performing additive combining.

[0037] Synthesis Process The synthesis process performed in the image processing apparatus 100 of this embodiment will be described below using the flowchart in Figure 7. The process corresponding to the flowchart can be realized by the CPU 101 reading the corresponding processing program stored in the ROM 102, for example, loading it into the RAM 103 and executing it. This synthesis process will be described as starting, for example, when an operation input related to the generation of a synthesized image is received via the operation I / F 105. In the description of the synthesis process of this embodiment, steps that perform the same processing as the synthesis process of Embodiment 1 will be given the same reference numerals and their descriptions will be omitted, and only steps that perform processing unique to this embodiment will be described below.

[0038] In S302, once the peak signal level is determined, the CPU 101 in S701 converts the dynamic range of the first target image and the second target image so that the signal level of the composite image obtained after additive synthesis does not exceed the peak signal level.

[0039] Here, the dynamic range conversion performed in this step will be explained with reference to the figures. If the first target image and the second target image are as shown in Figure 4 and their signal characteristics are as shown in Figure 5, the dynamic range of each target image is converted as shown in Figure 8, for example. More specifically, in this step, the dynamic range of each target image is converted so that the maximum signal level (display brightness) that the composite image obtained by adding the converted first target image and the second target image can take becomes the peak signal level acquired in S302. Therefore, since there are two images to be composited in this embodiment, the maximum display brightness of each converted target image is set to the value obtained by dividing the peak signal level by 2 (Y2 / 2). That is, in the embodiment where two HDR images are composited, the higher of these peak brightness values ​​is divided equally by the number of images to be composited, and this value is set as the maximum display brightness of each converted target image. In other words, in the processing of this step, the maximum display brightness of both the first target image and the second target image after conversion is the maximum value Y2 / 2.

[0040] The dynamic range conversion is performed using a knee characteristic, as shown in Figure 8, which compresses the dynamic range so that the maximum value falls within Y2 / 2 for level ranges exceeding a predetermined knee point defined for scene brightness, while preserving the tonal representation in each target image.

[0041] Figure 8(a) shows the signal characteristics of the scene brightness of the subject and the display brightness of the image obtained by transforming the first target image with this knee characteristic (first transformed image). As shown in the figure, the signal characteristics of the first transformed image maintain linear gradation for scene brightness (dark areas) up to a predetermined knee point, and compress the display brightness for scene brightness (bright areas) above the knee point. More specifically, for the scene brightness of the bright areas, compression is performed by assigning the scene brightness from the predetermined knee point to the saturated scene brightness in the first target image (the lowest scene brightness that becomes the peak brightness value in the first target image) to the Y2 / 2 level range from the predetermined knee point.

[0042] Figure 8(b) similarly shows the signal characteristics of the scene brightness of the subject and the display brightness of the image obtained by transforming the second target image using the knee characteristics described above (second transformed image). As shown in the figure, the signal characteristics related to the second transformed image are the same as those related to the first transformed image: the scene brightness (dark areas) up to a predetermined knee point is linearly maintained, while the display brightness of the scene brightness (bright areas) above the knee point is compressed. More specifically, for the scene brightness of the bright areas, compression is performed by assigning the range from the predetermined knee point to the saturated scene brightness in the second target image (the lowest scene brightness that becomes the peak brightness value in the second target image) to the Y2 / 2 level range from the predetermined knee point.

[0043] Furthermore, the predetermined knee point for the conversion of each composite image may be fixed to a brightness level obtained by a predetermined number of stops above or below the brightness of the properly exposed image, or it may be shifted towards the darker areas as the amount of dynamic range compression increases. In addition, maintaining gradation by making the dark areas linear during dynamic range conversion is based on the fact that PQ format encoding allocates more bits to the dark areas to match the characteristics of human vision.

[0044] The first and second transformed images obtained in this way are illustrated in Figures 9(a) and (b). In each case, the brightness of the region showing the peak brightness value is reduced compared to the first and second target images shown in Figure 4.

[0045] In S701, a first converted image and a second converted image are generated by converting the dynamic range. In S702, the CPU 101 adds the first converted image and the second converted image together to generate a composite image. As described above, in the compositing process of this embodiment, the peak luminance value of each converted image is adjusted to half the peak signal level. Therefore, unlike Embodiment 1, no pixels with signal levels exceeding the peak signal level are generated in the image obtained by additive compositing. Consequently, signal level clipping processing is unnecessary after additive compositing, and the obtained image can be treated as a composite image. The composite image obtained at this time, as shown in Figure 9(c), has a maximum output dynamic range within the level range of Y2 and has a gradation representation for both the first target image and the second target image. More specifically, the composite image generated by the compositing process of this embodiment differs from Embodiment 1 in that it includes the difference in gradation between region 401 in the first target image and region 402 in the second target image.

[0046] Thus, according to the image processing apparatus of this embodiment, when additively combining multiple images including one or more HDR images, it is possible to generate a composite image that exhibits a suitable brightness representation while ensuring the gradation representation of the images to be combined.

[0047] [Example 1] In the above-described Embodiment 2, the dynamic range of the HDR image to be synthesized is converted so that the maximum value of its display brightness remains constant. However, the implementation of the present invention is not limited to this. In the method of Embodiment 2, since target images with different output dynamic ranges are compressed to a common output dynamic range, depending on the difference in peak brightness values ​​between the HDR images to be synthesized, the scene brightness and the signal level in each converted image may be reversed after conversion. That is, for a subject with the same scene brightness, the first converted image may be shown with a higher display brightness than the second converted image. Also, a subject in the second target image, which should be brighter than the subject in the first target image, may become darker in the second converted image than the subject in the first converted image. In this modified example, the manner of converting the dynamic range of each target image in S702 is made different so that such a reversal of gradation between converted images does not occur.

[0048] The dynamic range conversion in this modified example is performed by converting scene brightness to display brightness by referring to a common conversion characteristic regardless of the target image, while making the maximum value of the converted display brightness for each target image different according to the peak brightness value of that target image. More specifically, if the first and second target images are as shown in Figure 4 and their signal characteristics are as shown in Figure 5, the dynamic range of each target image is converted as shown in Figure 10, for example.

[0049] Here, the dynamic range conversion of both target images remains the same as in Embodiment 2, in that the peak signal level is the maximum signal level (display brightness) that the composite image obtained by adding the converted images can take. That is, the conversion is performed so that the sum of the maximum display brightness Y1' of the first converted image according to this modified example shown in Figure 10(a) and the maximum display brightness Y2' of the second converted image according to this modified example shown in Figure 10(b) is equal to the peak signal level, Y2.

[0050] On the other hand, as shown in the figure, the signal characteristics for both converted images are common up to the maximum display brightness of each image. More specifically, the signal characteristics are a knee characteristic that maintains linear gradation for scene brightness up to a common knee point, and compresses scene brightness (input) above the knee point so that the output is Y1' when it is Y1 and Y2' when it is Y2. By using a common conversion characteristic for each target image in this way and making the maximum display brightness of each different according to the peak brightness value, the dynamic range can be converted without the signal level exceeding the peak signal level after additive synthesis, and without causing gradation reversal between converted images.

[0051] Thus, according to this modified image processing apparatus, when additively combining multiple images including one or more HDR images, it is possible to generate a composite image that exhibits a suitable brightness representation while ensuring absolute gradation representation of scene brightness.

[0052] [Differentiation 2] In Embodiment 2 and Modification 1 described above, the dynamic range conversion of the target image was explained as being performed based on a knee characteristic in which the conversion characteristics change at a predetermined knee point, as shown in Figures 8 and 10. However, the present invention is not limited to this. Any conversion method, such as gamma characteristics or scale conversion, may be employed for the dynamic range conversion of the target image.

[0053] [Difference 3] In the embodiments and modifications described above, a MaxDRL indicating the peak brightness value is associated with one image to be synthesized, and the MaxDRL is read out and determined as the peak signal level. However, the present invention is not limited to this. For example, in an embodiment in which the maximum signal PQ code value corresponding to the image imaging conditions such as the imaging mode and exposure amount is stored in the recording device 104 in advance, the corresponding maximum signal PQ code value may be obtained based on the imaging condition information of the target image and determined as the peak signal level.

[0054] [Differentiation Example 4] In the embodiments and modifications described above, the images to be combined were assumed to be two HDR images. However, the present invention is not limited to this and can also be applied to embodiments that combine three or more HDR images. In this case, the peak signal level can be determined to be the largest of the peak luminance values ​​of the multiple HDR images to be combined. Alternatively, any one of the peak luminance values ​​of the multiple HDR images may be adopted as the peak signal level.

[0055] In this embodiment, as in Embodiment 2, when the dynamic range is converted to equalize the maximum display brightness of each target image, the maximum display brightness of each target image after conversion should be set to the value obtained by dividing the peak signal level by the number of images to be combined. Also, as in Modification 1, when the dynamic range conversion is performed so that the maximum display brightness of each target image differs according to the peak brightness value of that target image, the sum of the maximum display brightness values ​​of each target image after conversion should be adjusted to equal the peak signal level.

[0056] [Difference 5] In the embodiments and modifications described above, it was explained that all images to be synthesized were HDR images. However, the implementation of the present invention is not limited to this, and as long as one or more HDR images are included, SDR images may also be included as other images. In this case, since SDR images are images that relatively represent scene brightness, in order to suitably synthesize them with HDR images, the SDR images may be made nonlinear, scaled to an arbitrary peak brightness, linearized to the HDR image, and then subjected to the same processing. In this case, the arbitrary peak brightness used for scaling may be a fixed value such as 100 nits, or it may be set based on user input.

[0057] Furthermore, although the HDR image to be synthesized has been described as a PQ-type HDR image, the present invention is also applicable to embodiments that use an HLG-type HDR image that does not utilize the maximum signal value.

[0058] [Modification 6] In the embodiments and modifications described above, the peak signal level, which is the maximum value of the output dynamic range of the composite image, is determined based on the peak brightness value of any of the HDR images to be composited. However, the present invention is not limited to this. The peak signal level may be determined to any value received based on user input received via the operation I / F 105, for example, within a range up to 1023, which is the maximum display brightness.

[0059] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0060] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0061] 100: Image processing device, 101: CPU, 102: ROM, 103: RAM, 104: Recording device, 105: Operation I / F

Claims

1. An image processing apparatus for generating a composite image, Acquisition means for acquiring multiple images, including one or more HDR (High Dynamic Range) images, A determination means for determining the peak brightness value of the composite image, A generation means that generates the composite image by performing an additive synthesis process using the aforementioned multiple images, It has, The additive synthesis process is a process of simply adding the signal levels of pixels at corresponding positions in the plurality of images, The generation means generates the composite image such that the signal level of each pixel in the composite image falls within an output dynamic range whose maximum value is the signal level corresponding to the peak brightness value determined by the determination means. An image processing apparatus characterized by the following:

2. An image processing apparatus for generating a composite image, Acquisition means for acquiring multiple images, including one or more HDR (High Dynamic Range) images, A determination means for determining the peak brightness value of the composite image, A generation means that generates the composite image by performing an additive synthesis process using the aforementioned multiple images, It has, The generation means generates the composite image after the additive synthesis process is executed by changing the signal level of each pixel in the composite image so that it fits within an output dynamic range where the signal level corresponding to the peak brightness value determined by the determination means is the maximum value. An image processing apparatus characterized by the following:

3. The image processing apparatus according to claim 1 or 2, characterized in that the generation means generates an intermediate image by adding and combining the plurality of images, and generates the composite image by changing the signal level of pixels in the intermediate image whose signal level exceeds the maximum value of the output dynamic range to the maximum value.

4. An image processing apparatus for generating a composite image, Acquisition means for acquiring multiple images, including one or more HDR (High Dynamic Range) images, A determination means for determining the peak brightness value of the composite image, A generation means that generates the composite image by performing an additive synthesis process using the aforementioned multiple images, It has, The generation means converts the dynamic range of each of the multiple images so that the signal level of each pixel after the additive synthesis process does not exceed the maximum value of the output dynamic range, which is the signal level corresponding to the peak brightness value determined by the determination means, and generates the composite image by adding and combining the converted multiple images. An image processing apparatus characterized by the following:

5. The image processing apparatus according to claim 4, characterized in that the dynamic range conversion is performed by setting the maximum value of the converted display brightness of each of the plurality of images to a value obtained by dividing the maximum value of the output dynamic range by the number of the plurality of images.

6. The image processing apparatus according to claim 4, characterized in that the dynamic range conversion causes the maximum value of the converted display brightness of each of the plurality of images to differ according to the peak brightness value of each image.

7. The image processing apparatus according to claim 6, characterized in that the dynamic range conversion is performed by referring to a common conversion characteristic for the plurality of images.

8. The aforementioned HDR image is an HDR image encoded using the PQ (Perceptual Quantization) method standardized by ITU-R BT. 2100. The determination means determines the peak brightness value of the HDR image as the peak brightness value of the composite image. The image processing apparatus according to any one of claims 1 to 7.

9. The image processing apparatus according to claim 8, characterized in that, when there are multiple HDR images included in the multiple images, the determination means determines the maximum value among the multiple HDR images as the peak brightness value of the composite image.

10. The aforementioned HDR image is an HDR image encoded using the PQ (Perceptual Quantization) method standardized by ITU-R BT. 2100. The determination means determines the peak brightness value corresponding to the imaging conditions of the HDR image as the peak brightness value of the composite image. The image processing apparatus according to any one of claims 1 to 7.

11. The system further includes an input means for receiving the peak brightness value of the composite image, The determination means determines the peak brightness value of the composite image based on the input received by the input means. The image processing apparatus according to any one of claims 1 to 7.

12. The image processing apparatus according to any one of claims 1 to 11, further comprising an output means for outputting an image file relating the composite image generated by the generation means to the maximum value of the output dynamic range.

13. The image processing apparatus according to any one of claims 1 to 12, characterized in that the additive synthesis process is a process of adding the signal levels of pixels at corresponding positions in the plurality of images without multiplying them by a coefficient different from 1.

14. The image processing apparatus according to claim 1, characterized in that the generation means generates the composite image after performing the additive synthesis process by fitting the signal level of each pixel of the composite image into an output dynamic range whose maximum value is the signal level corresponding to the peak brightness value determined by the determination means.

15. The image processing apparatus according to claim 1, characterized in that the generating means converts the dynamic range of each of the plurality of images so that the signal level of each pixel after the additive synthesis process does not exceed the maximum value of the output dynamic range, and generates the composite image by adding and combining the plurality of converted images.

16. The image processing apparatus according to any one of claims 1 to 15, characterized in that the peak brightness value determined by the determination means is the peak brightness value of the dynamic range that can be expressed in the format of the composite image.

17. A method for controlling an image processing device that generates a composite image, A process of acquiring multiple images, including one or more HDR (High Dynamic Range) images, A determination step for determining the peak brightness value of the composite image, A generation step in which an additive synthesis process is performed using the aforementioned multiple images to generate the composite image, It has, The additive synthesis process is a process of simply adding the signal levels of pixels at corresponding positions in the plurality of images, In the generation step, the composite image is generated such that the signal level of each pixel in the composite image falls within an output dynamic range whose maximum value is the signal level corresponding to the peak brightness value determined in the determination step. A control method characterized by the following:

18. A method for controlling an image processing apparatus that generates a composite image, A process of acquiring multiple images, including one or more HDR (High Dynamic Range) images, A determination step for determining the peak brightness value of the composite image, A generation step in which an additive synthesis process is performed using the aforementioned multiple images to generate the composite image, It has, In the generation step, after the additive synthesis process is executed, the composite image is generated by changing the signal level of each pixel in the composite image so that it fits within an output dynamic range where the signal level corresponding to the peak brightness value determined in the determination step is the maximum value. A control method characterized by the following:

19. A control method for an image processing apparatus that generates a composite image, A process of acquiring multiple images, including one or more HDR (High Dynamic Range) images, A determination step for determining the peak brightness value of the composite image, A generation step in which an additive synthesis process is performed using the aforementioned multiple images to generate the composite image, It has, In the generation step, the dynamic range of each of the multiple images is converted so that the signal level of each pixel after the additive synthesis process does not exceed the maximum value of the output dynamic range, which is the signal level corresponding to the peak brightness value determined in the determination step. The composite image is then generated by adding and combining the converted multiple images. A control method characterized by the following:

20. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 16.

Images