Image processing device, imaging device, control method, and program

Abstract

To determine a composition ratio suitable for generation of a composite image having an extended dynamic range.SOLUTION: An image processing device includes: acquisition means that, for each of a plurality of HDR (High Dynamic Range) images photographed with different amounts of exposure, acquires exposure information of photographing and an upper limit value of a dynamic range expressed in the image; determination means that, for a composition process for composing the plurality of HDR images to generate a composite image in which the dynamic range is further extended, determines a composition ratio of each HDR image for each signal region of the composite image, based on the exposure information and the upper limit value of the dynamic range acquired by the acquisition means.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, a control method, and a program, and more particularly to a technique for synthesizing high dynamic range image signals. [Background technology]

[0002] There is a technique for generating a high dynamic range (HDR) composite image by combining multiple standard dynamic range (SDR) images taken at different exposure levels. Patent Document 1 describes how to generate an HDR (High Dynamic Range) composite image by combining three types of SDR images—an appropriate image obtained from properly exposed shooting, an underexposed image obtained from underexposed shooting, and an overexposed image obtained from overexposed shooting—according to a predetermined combination ratio. More specifically, the luminance thresholds Y1, Y2, Y3, and Y4 (Figure 10 in Patent Document 1) are defined as the basis for combination, and the combination is controlled so that an overexposed image is used for luminance ranges darker than Y1, an appropriate image is used for luminance ranges from Y2 to Y3, and an underexposed image is used for luminance ranges brighter than Y4. Furthermore, for the intermediate regions between Y1 to Y2 and Y3 to Y4, the combination is controlled so that the combination ratio (weighted addition coefficient) of the overexposed image and the appropriate image, and the appropriate image and the underexposed image, are gradually changed. This type of composite control makes it possible to obtain an HDR composite image with a suitably expanded dynamic range from SDR images under three different exposure conditions.

[0003] In recent years, the performance of light-emitting elements such as LEDs has improved, leading to the emergence of HDR displays, which offer a wider dynamic range of display brightness than conventional displays. Such displays can more faithfully display images with high-brightness colors and details (HDR images). The signal characteristics representing the relationship between the video signal level and display brightness in HDR images are defined by the EOTF (Electro-Optical Transfer Function), and the following two methods are employed. One is the HLG (Hybrid Log Gamma) method, standardized in ARIB STD-B67, which converts the video signal level to a relative value of the display brightness, resulting in a display brightness corresponding to the maximum brightness that the display device can output. The other is the PQ (Perceptual Quantization) method, standardized in SMPTE ST 2084 or ITU-R BT.2100, which converts the video signal level to a maximum of 10,000 nits (or cd / m²). 2 The display brightness is converted to an absolute value within the range of ). Therefore, when displaying an HDR image obtained by capturing a scene, in the former method, the scene brightness is converted to a display brightness corresponding to the maximum brightness that the display device can output, while in the latter method, the scene brightness is converted to a display brightness that is absolutely determined regardless of the display device. Consequently, when considering display on a display device employing the PQ method, for example, in encoding in the imaging device, it is necessary to convert the image signal of the scene brightness to show an absolute brightness value and generate an HDR image. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2013-240031 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] Incidentally, by acquiring and combining HDR images with different exposure levels for the same scene, it is also possible to generate a composite image with an even wider dynamic range. That is, similar to Patent Document 1, by combining three types of HDR images—a properly exposed HDR image obtained with proper exposure, an underexposed HDR image obtained with underexposure, and an overexposed HDR image obtained with overexposure—a composite image that expresses more detail can be generated.

[0006] On the other hand, in PQ encoding, which absolutely represents scene brightness, even when shooting the same scene, the peak brightness (maximum display brightness, maximum output dynamic range) included in the HDR image may change depending on the exposure amount. This is because the scene brightness at which the sensor output saturates changes according to the exposure amount, and therefore, different gamma curves are used for conversion in order to assign absolute display brightness to the same scene brightness. For example, as shown in Figure 1, the input / output characteristics (relationship between input stage and output brightness) for two shooting modes with different exposure amounts have different peak brightness (maximum output brightness). Here, the input / output characteristics 11 in the high exposure shooting mode are shown by a solid line, and the input / output characteristics 12 in the low exposure shooting mode are shown by a dashed line. As shown in the figure, the two shooting modes show common input / output characteristics except in the high brightness range, and are converted to the same display brightness regardless of the exposure amount, whereas in the high brightness range, the peak brightness differs between values ​​13 and 14 according to the difference in brightness at which saturation occurs. The value 15 represents the maximum value in 10 bits (1023), which corresponds to the maximum display brightness of 10,000 nits in the PQ method.

[0007] Therefore, simply applying a synthesis method like that described in Patent Document 1 may not produce a suitable composite image. More specifically, the synthesis method in Patent Document 1 assumes an 8-bit SDR image with a maximum pixel value of 255 assigned to each exposure condition, so in the brightness range where two types of images are synthesized, the pixel values ​​of both images could always be referenced. However, when synthesizing HDR images with different exposure levels, in the brightness range where the two types of images are synthesized, subjects exceeding the peak brightness may not be represented in the high-exposure HDR image, which may result in unsatisfactory synthesis results.

[0008] For example, consider the case where the luminance of the appropriate HDR image is distributed as shown in histogram 24 of Figure 2 (the peak luminance of the appropriate HDR image is 25), and the blending ratio of Patent Document 1 is applied. In the figure, the dashed line 21, the solid line 22, and the single dashed line 23 represent the weighted addition coefficients for the over-HDR image, the appropriate HDR image, and the under-HDR image, respectively, corresponding to the blending ratio of Patent Document 1, and the sum of the coefficients is 1 across the entire luminance range. In this case, as shown in the figure, if the peak luminance of the appropriate HDR image of 25 is included in the luminance range (Y3~Y4) where the appropriate HDR image and the under-HDR image are blended, the appropriate HDR image will not be blended in the luminance range from that peak luminance to Y4, and therefore a suitable blending result will not be obtained. In other words, because the appropriate HDR image does not contain pixels with a peak luminance of 25 or higher due to sensor output saturation, there was a possibility that an unnatural representation would occur in the region of the blended image where subjects near that peak luminance are distributed.

[0009] As illustrated above, combining multiple images with different peak brightness levels can result in unnatural-looking images.

[0010] The present invention has been made in view of the above-mentioned problems, and aims to provide an image processing apparatus, an imaging apparatus, a control method, and a program for determining a suitable synthesis ratio for generating a composite image with an expanded dynamic range. [Means for solving the problem]

[0011] In order to achieve the above object, an image processing apparatus according to the present invention is an image processing apparatus that synthesizes a plurality of HDR (High Dynamic Range) images taken with different exposure amounts, and the scene luminance is expressed in an absolute luminance method. For each of the plurality of HDR images, specifying means for specifying a signal value indicating the upper limit value of the output dynamic range, determining means for determining the synthesis ratio of the plurality of HDR images based on the signal value specified by the specifying means, and acquisition means for acquiring a signal threshold value serving as a reference for switching the tendency of the synthesis ratio of each of the plurality of HDR images, The identification means identifies the signal value from metadata corresponding to the upper limit of the output dynamic range attached to each of the multiple HDR images, The determining means is characterized in that it changes the signal threshold value acquired by the acquisition means based on the signal value indicating the upper limit value of the output dynamic range of each of the plurality of HDR images, and determines the synthesis ratio based on the changed signal threshold value.

Effect of the Invention

[0012] According to the present invention with such a configuration, it is possible to determine a suitable synthesis ratio for generating a synthesized image with an extended dynamic range.

Brief Description of the Drawings

[0013] [Figure 1] A diagram for explaining input / output characteristics for two types of shooting modes with different exposure amounts [Figure 2] A diagram showing an aspect of applying the weighted addition ratio based on the SDR image standard to the synthesis process of HDR images [Figure 3] A block diagram illustrating the hardware configuration of an image processing apparatus 100 according to an embodiment and a modified example of the present invention [Figure 4] A block diagram illustrating the module configuration of the synthesis process according to Embodiment 1 of the present invention [Figure 5] A flowchart illustrating the synthesis process according to Embodiment 1 of the present invention [Figure 6] A diagram for explaining the difference in characteristics between the OETF applied according to the exposure amount of shooting and the OETF applied in relation to the generation of the synthesized image [Figure 7] Figure illustrating the reference Mix table and modified Mix table according to embodiments of the present invention. [Figure 8] Figure illustrating the output file format of composite images according to embodiments and modifications of the present invention. [Figure 9] A diagram illustrating the maximum DRL values ​​determined according to the shooting mode. [Figure 10] Block diagram illustrating the modular configuration of the synthesis process according to Embodiment 2 of the present invention. [Figure 11] Block diagram illustrating the modular configuration of the developing process according to Embodiment 2 of the present invention. [Figure 12] A diagram illustrating a standard Mix table according to Modification 2 of the present invention. [Figure 13] A diagram illustrating a modified Mix table according to Modification 2 of the present invention. [Figure 14] Another diagram illustrating a modified Mix table according to Modification 2 of the present invention. [Modes for carrying out the invention]

[0014] [Embodiment 1] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0015] One embodiment described below illustrates an example of an image processing apparatus in which the present invention is applied to an image processing apparatus capable of generating a composite image with an expanded dynamic range by combining HDR images encoded in the PQ method. However, the present invention is applicable to any device capable of deriving the composition ratio of each HDR image involved in the generation of the composite image.

[0016] Furthermore, in this specification, unless otherwise specified, "HDR image" refers to an image (PQ signal) to which PQ encoding is applied and configured so that absolute display luminance can be obtained by decoding. In addition, to distinguish from HDR images obtained by combining HDR images, which have an even wider dynamic range, such as HDR composite images obtained by combining conventional SDR images, and from the input (source) HDR images, these will be simply referred to as "composite images." In the following description, composite images will be assumed to be PQ signals, just like HDR images.

[0017] Here, representing HDR characteristics with a PQ signal requires a minimum depth of 10 bits, and HDR images and composite images are not files that store 8-bit images like the JPEG format. For example, HDR images and composite images may be HEIF files stored using a High Efficiency Image File Format (hereinafter referred to as HEIF), an image file format developed by the Moving Picture Experts Group (MPEG) and defined in MPEG-H Part 12 (ISO / IEC 23008-12). HEIF can store not only the main image but also thumbnails, multiple time-related images, and metadata such as EXIF ​​and XMP in a single file. Furthermore, since HEIF can also store 10-bit image sequences encoded with HEVC, it can be said to be suitable for storing HDR images and composite images in this invention. However, the implementation of this invention is not limited to this, and HDR images and composite images may be in any format that can store image data of 10 bits or more, such as RAW data or TIFF data after development processing.

[0018] Hardware configuration of the image processing unit Figure 3 is a block diagram showing the hardware configuration of the image processing device 100 according to this embodiment. As shown in the figure, the image processing device 100 of this embodiment includes a CPU 101, ROM 102, RAM 103, recording device 104, operation I / F 105, display device 106, and system bus 107.

[0019] The CPU 101 performs overall control of the image processing unit 100. The ROM 102 is a storage device that stores control programs such as the BIOS necessary for starting the image processing unit 100, as well as programs, parameters, and data that do not require modification. The RAM 103 is a storage device that has a work area for the CPU 101, a primary storage area for temporarily storing various data, and a load area for various programs.

[0020] The recording device 104 is a recording device that stores various data such as the OS program, various control programs, various software programs that can be executed on the OS program, HDR images used in the synthesis process described later, and synthesized images obtained as a result of the synthesis process. The recording device 104 includes, for example, a hard disk or flash memory that is built into or detachably connected to the image processing device 100, a detachably connected flexible disk or optical disk, magneto-optical disk, IC card, memory card, etc. Therefore, the CPU 101 can control the image processing device 100 by loading the various programs stored in the ROM 102 or the recording device 104 into the RAM 103 and executing them.

[0021] The operation I / F 105 is a user interface provided by the image processing device 100, such as a keyboard, mouse, or touch panel. When the operation I / F 105 detects that a user has made an operation input, it sends a control signal related to that operation to the CPU 101. The display device 106 is, for example, a liquid crystal display, and displays various information, including a graphical user interface related to the OS and software running on the image processing device 100. The system bus 107 connects each block that makes up the image processing device 100 so that they can communicate with each other.

[0022] The image processing device 100 may be a dedicated device for developing and editing captured RAW images, in which case the synthesis processing program can be stored in the ROM 102. Furthermore, a general-purpose personal computer (PC) can function as the image processing device 100 by having the CPU 101 load the synthesis processing program stored in the recording device 104 into the RAM 103. Additionally, an imaging device capable of acquiring RAW images through imaging can also function as the image processing device 100 by having its control unit execute the synthesis processing program, or by having an image processing unit that performs the synthesis processing perform a predetermined operation.

[0023] Synthesis process Next, the image synthesis process performed in the image processing apparatus 100 according to this embodiment, which generates a composite image by combining multiple HDR images taken with different exposure levels, will be described in detail with reference to the figures. This synthesis process is realized when the CPU 101 reads the synthesis process program from the ROM 102 or recording device 104, loads it into the RAM 103, and executes it. Figure 4 illustrates the module configuration realized during the execution of the synthesis process, and Figure 5 is a flowchart illustrating the flow of the synthesis process.

[0024] In the following description, the multiple HDR images taken with different exposure levels and used as input for the image synthesis process will all be HDRPQ images encoded using the PQ method (hereinafter referred to as PQ images). Furthermore, in the image synthesis process of this embodiment, the input PQ images will be of three types: a proper PQ image 402 obtained by shooting with proper exposure, an overexposed PQ image 401 obtained by shooting with overexposure, and an underexposed PQ image 403 obtained by shooting with underexposure. The overexposed PQ image 401, the proper PQ image 402, and the underexposed PQ image 403 may be images obtained by sequentially shooting a common scene in shooting modes with different exposure settings for the purpose of synthesizing the composite image. In this embodiment, shooting with different exposure settings will be described as shooting in different shooting modes for convenience, but it will be understood that the implementation of the present invention is not limited to this.

[0025] In this embodiment, each input PQ image is assumed to have metadata including exposure information indicating the exposure settings of the imaging device at the time of capturing the PQ image, and OETF (Opt-Electronic Transfer Function) information used to encode the PQ image. The exposure information may be, for example, information on the aperture, shutter speed, and ISO sensitivity set at the time of capturing. The OETF is the product of the OOTF (Opt-Optical Transfer Function) based on the user's intent to create the output image, which is pre-set for each exposure setting, and the inverse function of the reference EOTF defined in ITU-R BT.2100. Furthermore, each PQ image is assumed to have MaxDRL (Maximum Dynamic Range Level) as described in Japanese Patent Application Publication No. 2020-039118 attached as metadata. MaxDRL is the maximum value of the PQ signal output value after applying the corresponding OETF during the development and encoding of the captured image signal, and is the peak brightness value (upper limit) of the dynamic range that can be represented by each PQ image. In this embodiment, MaxDRL is described as the peak luminance value, but MaxDRL may also be a value indicating the nit value corresponding to the peak luminance value. MaxDRL is used to determine the dynamic range of the input image, for example, when tone mapping an HDR image to an HDR or SDR signal with a narrower dynamic range. The three types of PQ images that are input may be input as different HEIF files, or they may be stored in a single HEIF file.

[0026] In S501, the over-PQ image 401, the appropriate PQ image 402, and the under-PQ image 403 are read from the recording device 104 and loaded into the RAM 103. This step provides the input for the PQ images to be combined into the composite image.

[0027] In S502, exposure matching is performed on three types of PQ images. Here, since the accuracy of exposure matching may decrease if the PQ image is left as a nonlinear signal, the PQ image is linearized before exposure matching is performed. More specifically, exposure matching of the three types of PQ images is achieved by linearization by the PQ de-gamma module 404, exposure matching by the exposure matching module 405, and nonlinearization by the PQ gamma module 406.

[0028] Linearization using the PQ degamma module 404 can be performed by referring to the OETF information used for encoding attached to each PQ image and applying its inverse function (degamma).

[0029] Furthermore, exposure adjustment by the exposure adjustment module 405 is performed by applying a gain based on the exposure step difference to each linearized image. In this embodiment, exposure adjustment is performed based on the appropriate exposure (the exposure used to acquire the appropriate PQ image 402), and gain is applied to the linearized overexposed and underexposed images, correcting them to the same exposure as the appropriate exposure. For example, if the appropriate PQ image 402 was taken with an aperture of F5.6, a shutter speed of 1 / 60 seconds, and an ISO sensitivity of 200, and the underexposed PQ image 403 was taken with an aperture of F5.6, a shutter speed of 1 / 500 seconds, and an ISO sensitivity of 200, the exposure step difference is 3 stops. Therefore, the exposure adjustment module 405 corrects the underexposed PQ image 403 to the same exposure as the appropriate PQ image 402 by increasing the gain of the linearized image by 3 stops.

[0030] Subsequently, the PQ gamma module 406 nonlinearizes the image by applying a common OETF to each image after exposure correction. The OETF applied by the PQ gamma module 406 is an OETF (dotted line 603) that corresponds to a wider dynamic range than the OETF (solid line 601) applied to the input PQ image, as shown in Figure 6. As illustrated, the OETF applied by the PQ gamma module 406 may correspond to a higher luminance 604 in order to obtain a composite image with an expanded dynamic range in the subsequent synthesis process. Here, the OETF applied by the PQ gamma module 406 may be predetermined and fixed, for example, as the inverse function of the reference EOTF defined in ITU-R BT.2100. Alternatively, the OETF may be adaptively changed so that the maximum luminance (luminance 604) is set according to the under-PQ image 403'.

[0031] The S502 process yields overexposed PQ images, correct PQ images, and underexposed PQ images with uniform exposure. For convenience, the PQ images after processing in S502 will be referred to as overexposed PQ image 401', correct PQ image 402', and underexposed PQ image 403'. Here, overexposed PQ image 401' has less noise in the dark areas than correct PQ image 402', and underexposed PQ image 403' has less blown-out highlights in the bright areas than correct PQ image 402', resulting in an image with tonal gradation.

[0032] In S503, the misalignment correction module 407 detects the misalignment between the over-PQ image 401' and the appropriate PQ image 402', and between the under-PQ image 403' and the appropriate PQ image 402', and corrects the misalignment if it occurs. Any known method may be used for detecting and correcting the misalignment. For example, the image may be divided into blocks, edges may be extracted, and a motion vector may be derived that minimizes the sum of absolute differences in pixel values ​​(SAD) for each block. Then, the misalignment correction may be performed by affine transforming the image to be corrected using the affine coefficients obtained based on this motion vector. When correcting the misalignment based on the appropriate PQ image 402', the affine transform may be performed on the other images, i.e., the over-PQ image 401' and the under-PQ image 403'. In this embodiment, detection accuracy is improved by reducing the difference in brightness between images, so the positional shift detection and correction are described as being performed after the processing in S502. However, since there is no dependency between these processes, it should be understood that the order of processing can be changed.

[0033] In S504, the ratio change module 408 determines the composite ratio (weighted addition ratio, weighted addition coefficient) for each image. In this embodiment, the weighted addition ratio for each image is determined by modifying the reference Mix table 409 as shown in Figure 7(a) to form a modified Mix table, and by referring to the modified Mix table. As shown in the figure, the reference Mix table 409 is distributed such that the weighted addition ratios for over-PQ image 401', appropriate PQ image 402', and under-PQ image 403' differ according to the luminance value, similar to Patent Document 1. The reference Mix table 409 includes the over-image ratio 701 (dotted line), appropriate image ratio 702 (solid line), and under-image ratio 703 (dotted line), which show the weighted addition ratio for each image in the display luminance range of the PQ method (corresponding to an upper limit of 10000 nits). Here, the weighted addition ratios for each image in the reference Mix table 409 are configured so that the sum is 1 (100%) for each luminance value. Therefore, each luminance range is divided into a range that uses only the over-PQ image 401', a range that combines the over-PQ image 401' and the appropriate PQ image 402', a range that uses only the appropriate PQ image 402', a range that combines the appropriate PQ image 402' and the under-PQ image 403', and a range that uses only the under-PQ image 403'. These range divisions are defined based on a reference point (luminance threshold) set for each image that switches the blending tendency.

[0034] The ratio change module 408 reads a reference Mix table 409 from, for example, the ROM 102 or recording device 104, and then acquires exposure information and MaxDRL for each of the over-PQ image 401, the appropriate PQ image 402, and the under-PQ image 403. The ratio change module 408 then modifies predetermined luminance thresholds provided in the reference Mix table 409 based on the acquired exposure information and MaxDRL to construct a table showing weighting ratios that favorably optimize the synthesis result of the subsequent synthesis module 410. In this embodiment, predetermined luminance thresholds are provided, including a luminance threshold 704 where the weighting ratio of the over-PQ image 401' becomes 0%, and a luminance threshold 705 where the weighting ratio of the appropriate PQ image 402' becomes 0%, and these are subject to modification.

[0035] As described above, the luminance ranges in which the over-PQ image 401' and the appropriate PQ image 402' are combined, and the luminance ranges in which the appropriate PQ image 402' and the under-PQ image 403' are combined, may not appear as signals of appropriate luminance values ​​in images taken with high exposure. In other words, applying a weighting ratio greater than 0% to an image that does not represent the corresponding luminance range, or applying a weighting ratio of less than 100% to the other image, may result in undesirable synthesis results. Therefore, the ratio changing module 408 of this embodiment changes the luminance thresholds 704 and 705 to values ​​that do not exceed the maximum display luminance that can be represented in the corresponding images.

[0036] In this embodiment, the exposure of the over-PQ image 401 and under-PQ image 403 is adjusted based on the appropriate PQ image 402. Therefore, the maximum display brightness for the appropriate PQ image 402' is the MaxDRL of the appropriate PQ image 402. On the other hand, the maximum display brightness for the over-PQ image 401' differs from the MaxDRL of the over-PQ image 401 due to the exposure adjustment in S502. Therefore, the ratio change module 408 is derived using the following formula. JPEG0007829765000001.jpg13148 That is, the ratio change module 408, like S502, sets the MaxDRL of the over-PQ image 401 to EOTF (OETF -1 The OETF is applied to linearize the result, and the gain related to the exposure step between the over-PQ image 401 and the appropriate PQ image 402 is applied. Subsequently, the ratio change module 408 applies OETF to the obtained value to make it nonlinear, and derives the maximum display brightness related to the over-PQ image 401'.

[0037] Accordingly, the ratio change module 408 changes the luminance thresholds 704 and 705 respectively to the maximum display luminance for the over-PQ image 401' and the appropriate PQ image 402' obtained in this way, and obtains a new modified Mix table as shown in Figure 7(b). More specifically, the ratio change module 408 linearly scales the over-image ratio 701 so that the luminance threshold 704 becomes the luminance threshold 714 (the maximum display luminance for the over-PQ image 401'), thereby obtaining the modified over-image ratio 711. The ratio change module 408 linearly scales the appropriate image ratio 702 so that the luminance threshold 705 becomes the luminance threshold 715 (the maximum display luminance for the appropriate PQ image 402'), thereby obtaining the modified appropriate image ratio 712. The ratio change module 408 also linearly scales the under-image ratio 703 by the same ratio as the appropriate image ratio 712, thereby obtaining the modified under-image ratio 713.

[0038] In this case, since the scaling ratio applied to the over-image ratio 711 and the appropriate image ratio 712 may differ, the luminance range for combining the over-PQ image 401' and the appropriate PQ image 402' may be determined based on the over-image ratio 711. That is, the luminance range in the appropriate image ratio 712 should be changed so that the weighted addition ratio of the appropriate PQ image 402' becomes 100% at the luminance threshold 714. Furthermore, the luminance range should be changed so that the weighted addition ratio of the appropriate PQ image 402' starts to increase from 0% at the luminance value 716 where the weighted addition ratio in the over-image ratio 711 begins to decrease. In other words, the modified Mix table should be configured so that the sum of the weighted addition ratios of the images to be combined becomes 100% at any luminance value, with particular emphasis on the high-exposure image.

[0039] In S505, the synthesis module 410 performs brightness-based synthesis processing using the aligned over-PQ image 401', the appropriate PQ image 402', and the under-PQ image 403' based on the modified Mix table configured in S504, and generates a composite image. Specifically, the synthesis module 410 stores the pixel values ​​at the same positions in the aligned over-PQ image 401' for the dark areas (brightness range where the over-image ratio 711 is 100%). The synthesis module 410 also stores the pixel values ​​at the same positions in the appropriate PQ image 402' for the midtone areas (brightness range where the appropriate image ratio 712 is 100%). The synthesis module 410 also stores the pixel values ​​at the same positions in the under-PQ image 403' for the bright areas (brightness range where the under-image ratio 713 is 100%). Furthermore, for the dark to midtone region, the synthesis module 410 derives and stores pixel values ​​by adding the pixel values ​​at the same positions in the appropriate PQ image 402' and the aligned over-PQ image 401' multiplied by their respective weighted addition coefficients. Similarly, for the midtone to bright region, the synthesis module 410 derives and stores pixel values ​​by adding the pixel values ​​at the same positions in the appropriate PQ image 402' and the aligned under-PQ image 403' multiplied by their respective weighted addition coefficients. As a result, a composite image can be obtained in which the representation of the scene is smoothly expressed from dark to bright areas, with smoother representation of the brightness range where transitions between images occur.

[0040] The generated composite image may be stored in the recording device 104 as an HEIF format image file, for example, as a new PQ image. The HEIF format image file has a file structure as shown in Figure 8(a). That is, the HEIF format file consists of an ftyp box 801 indicating the file format, a meta box 802 storing metadata including a thumbnail image 811, and an mdat box 803 storing encoded data (main image 812). Therefore, the synthesis module 410 stores the data of the generated composite image encoded in the PQ format in the mdat box 803, stores the thumbnail image of the composite image and the MaxDRL related to the composite image in the meta box 802, and generates the file.

[0041] Here, the MaxDRL for the composite image may be, for example, the maximum display brightness for the under-PQ image 403'. That is, since the upper limit of the dynamic range of the composite image is mainly determined based on the under-PQ image 403 which has been composited with increased gain, the composite module 410 may derive the information of the maximum display brightness using the following formula and use it as MaxDRL. JPEG0007829765000002.jpg14135 In other words, the MaxDRL of the under-PQ image 403 is converted to a linear value by applying the inverse characteristics of the OETF used for encoding, and this value is then gain-up and a common OETF is applied to convert it to a nonlinear value, which is then stored as the MaxDRL of the composite image.

[0042] Alternatively, MaxDRL does not need to be determined based on the MaxDRL of the image. For example, as shown in Figure 9, MaxDRL information may be stored in advance for the shooting mode used to acquire the input PQ image, and the corresponding value may be adopted. For example, if three PQ images taken in a normal shooting mode of ±3 stops are input, the synthesis module 410 can determine the MaxDRL of the synthesized image to be 888 (2906 nits) based on the information (maximum signal PQ code value) stored in the recording device 104. This makes it possible to create a file of the output synthesized image that can be easily used.

[0043] [Embodiment 2] In the embodiments described above, the embodiment in which the input HDR image to be processed for synthesis is a PQ image was explained, but the present invention is not limited to this. That is, the input image can be any HDR image in which scene brightness is expressed in a manner different from that of an SDR image, which assigns a fixed range of values ​​regardless of exposure conditions, and may be a RAW image (12-14 bits) with a higher resolution than a PQ image (10 bits). In this case, for example, synthesis processing can be realized by taking multiple RAW images obtained from shooting with different exposure levels as input using a module configuration as shown in Figure 10. Here, the input RAW images are assumed to be of three types: an appropriate RAW image 1002 obtained from shooting with proper exposure, an overexposed RAW image 1001 obtained from shooting with overexposure, and an underexposed RAW image 1003 obtained from shooting with underexposure. Also in Figure 9, the same reference numerals are used for components that are common with the module configuration related to the synthesis processing of Embodiment 1, and detailed explanations below are omitted.

[0044] The synthesis process of this embodiment includes the conversion (development processing) of the three types of input RAW images before development into PQ images by the development processing module 1004. The development processing performed by the development processing module 1004 is subdivided into processing by the functional modules shown in Figure 11. Specifically, the white balance module 1101 performs white balance processing on the input RAW image to make whites white, and applies gains to each of R, G, and B such that the R, G, and B in the areas that should be white have the same signal value. Then, the noise reduction module 1102 performs noise reduction processing on the input image to reduce sensor-induced noise and other noises that do not originate from the subject image. Then, the color interpolation module 1103 applies color interpolation processing to the input color mosaic image to generate a color image in which R, G, and B color information is available in all pixels. The generated color image is then subjected to matrix conversion processing by the matrix conversion module 1104 and gamma conversion processing by the gamma conversion module 1105 to generate a basic color image. Here, the gamma conversion process performed by the gamma conversion module 1105 uses, for example, the OETF corresponding to the shooting mode, which is attached to each RAW image. Subsequently, the color adjustment module 1106 applies image correction processing to the color image to improve its appearance, such as saturation enhancement, hue correction, and edge enhancement, and outputs a PQ image.

[0045] Thus, even when a linear RAW image is input, by first applying OETF to convert it into a PQ image, a similarly suitable composite image can be generated using the functional module related to the synthesis process of Embodiment 1.

[0046] In this embodiment, a method of reusing the functional module related to the synthesis process of Embodiment 1 has been described so that it can be used in combination with the synthesis process of Embodiment 1. However, it will be easily understood that the implementation of the present invention is not limited to this. That is, since the information of the brightness range of each image related to the determination of the synthesis ratio is obtained by adjusting the exposure of the linearized image, the development process by the development processing module 1004 and the linearization process by the PQ de gamma module 404 do not need to be performed.

[0047] [Example 1] In the embodiments described above, a configuration was described in which the OETF used / to be used for development and encoding of the HDR image, which is the input to the synthesis process, can be obtained. On the other hand, including OETF information (or EOTF information corresponding to its inverse characteristics) as metadata in the image file can increase the file size, so such information may not be included. In this case, the inverse function of the OETF used for linearization by the PQ de gamma module 404 may be the reference EOTF defined in ITU-R BT.2100, although the accuracy may be reduced. Also in this case, the inverse function of the reference EOTF may be used for nonlinearization by the PQ gamma module 406.

[0048] [Differentiation 2] Furthermore, in the embodiments and modifications described above, it was explained that the HDR images, which are the input to the synthesis process, are obtained by shooting with different exposure levels, and that the relative magnitudes of the peak brightness values ​​of the dynamic range are distributed in a manner suitable for the synthesis process. That is, it was explained that a weighted addition ratio that allows for suitable synthesis of the input images is derived by scaling the reference Mix table 409 based on the MaxDRL of the exposure-matched HDR images.

[0049] However, if, for example, the input HDR image has had its luminance conversion characteristics altered (MaxDRL changes) due to retouching applied after shooting, the luminance range for combining the two images may be suppressed, potentially preventing a smooth transition between images. In other words, if the luminance range allocated to the transition is narrow, the change in the weighted addition ratio within that luminance range becomes abrupt, potentially resulting in an unnatural appearance (hereinafter referred to as a transition step) in the resulting composite image.

[0050] Furthermore, if, for example, as in Modification Example 1, the OETF cannot be obtained for the input HDR image, or if retouching is applied that alters the luminance conversion characteristics (changing the EOTF required for linearization), linearization and exposure adjustment may be insufficient. In other words, it may not be possible to assign appropriate luminance to subjects near the saturation luminance of each HDR image, potentially resulting in color distortion or other issues in the composite image.

[0051] In this modified example, in order to prevent the occurrence of such transition steps and color distortions in the composite image, a method for generating a modified Mix table related to determining the weighting ratio of each HDR image will be explained with reference to the figure. Specifically, in this modified example, an embodiment will be described that allows the weighting ratio of each image to be suitably set not only according to the MaxDRL of the input HDR images, but also according to their combination.

[0052] In this embodiment, unlike the embodiment described above, a reference Mix table as shown in Figure 12 is provided for Max(R,G,B), which is the maximum value of any of the R, G, or B color signals of the reference pixel of the input image. The reference Mix table in Figure 12, like the reference Mix table 409 shown in Figure 7(a), shows the weighting ratio for over-HDR images with a dashed-dot line 1201, for appropriate HDR images with a solid line 1202, and for under-HDR images with a dashed-dot line 1203. The difference from the reference Mix table 409 shown in Figure 7(a) is that the reference Mix table in Figure 12 specifies the maximum signal value (Max(R,G,B)) among R, G, and B, rather than the luminance value. In the following, for the sake of simplicity, the signal threshold 1206 at which the weighting ratio for over-HDR images begins to decrease from 100% and the weighting ratio for appropriate HDR images begins to increase from 0%, or the corresponding threshold, is referred to as Over 1End It is sometimes referred to as such. Also, the signal threshold 1204 at which the weighting ratio of the over-HDR image reaches 0% and the weighting ratio of the appropriate HDR image reaches 100%, or the corresponding threshold, is called Suit 1StartIt is sometimes referred to as such. Also, the signal threshold 1207 at which the weighting ratio of suitable HDR images begins to decrease from 100% and the weighting ratio of under-HDR images begins to increase from 0%, or the corresponding threshold, is called Suit 1End It is sometimes referred to as the signal threshold 1205, or the corresponding threshold, at which the weighted addition ratio of the appropriate HDR image reaches 0% and the weighted addition ratio of the under-HDR image reaches 100%. 1Start This is sometimes referred to as such. The ratio change module 408 in this modified example configures a modified Mix table for synthesis processing in the synthesis module 410 by adjusting these specified signal thresholds 1204 to 1207.

[0053] 《Overview of the Modified Mix Table Configuration》 Here, we will briefly explain the configuration of the modified Mix table that determines the weighting ratio of each HDR image using the standard Mix table in this modified example.

[0054] The color distortion described above occurs mainly due to nonlinear transformations (changes in the degree of flatness of the output curve) near the saturation luminance value, as shown in Figure 6. Therefore, if the transformation characteristics applied during development or transformation characteristics that take retouching into account cannot be obtained, the luminance values ​​of the input HDR image cannot be accurately linearized, and consistency between images cannot be guaranteed in the synthesis of the relevant value range. On the other hand, if the area near the saturation luminance value is excluded, there is a high possibility that a linear transformation has been performed. For this reason, the ratio change module 408 in this modified example defines a number of steps where nonlinear transformation may occur (synthesis avoidance step) as a constant, and derives the maximum display luminance value (converted MaxDRL) by taking this synthesis avoidance step into account for the over-HDR image and the appropriate HDR image. More specifically, the ratio change module 408 derives the converted MaxDRL by applying, for example, the reference EOTF to the MaxDRL of the input image, linearizing the value, and then decreasing it by the value of the synthesis avoidance step, and then modifies the reference Mix table based on this.

[0055] Furthermore, the transition step described above can occur because the signal range defined by signal thresholds 1206 to 1204 and signal thresholds 1207 to 1205 are suppressed. In particular, when changing the reference Mix table using a converted MaxDRL set to exclude the luminance range near the saturation luminance, as in this modified example, suppression of the signal range occurs. Since there is sufficient headroom for the under-HDR image up to the saturation signal value, in this modified example, the ratio change module 408 assigns the weighted addition ratios for the over-HDR image and the appropriate HDR image based on the signal value corresponding to the converted MaxDRL of the appropriate HDR image. Therefore, the range of values ​​from signal value 0 to the signal value corresponding to the converted MaxDRL of the appropriate HDR image includes the following two types of ranges. One is the range of values ​​where the weighted addition ratio of the over-HDR image is 100% and the range of values ​​from 0% to 100% (hereinafter referred to as the over-usage range). The other side is the range of values ​​where the weighted addition ratio of the appropriate HDR image is 100% and the range of values ​​from 0% to 100% (hereinafter referred to as the appropriate usage range). In this modified example, a threshold W is set for the width of the signal range that should be secured in order to perform smooth synthesis for the overuse range and the appropriate usage range, and the modified mix table is constructed by adjusting each signal threshold so that the appropriate usage range has a range width of at least threshold W.

[0056] The processing performed by the ratio change module 408 first involves deriving the converted MaxDRL for both the over-HDR image and the appropriate HDR image using the following formula. Specifically, the converted MaxDRL for each image is derived by linearizing the MaxDRL of each image by applying, for example, a reference EOTF, then applying a common OETF to the linearized value, reducing the gain according to the difference from the appropriate exposure and the number of steps to avoid composite exposure, and making it nonlinear. In other words, in order to avoid the mismatch that occurs due to the difference in characteristics between the inverse characteristics of the OETF used during the development of the input image and the EOTF applied to the linearization, the ratio change module 408 obtains a converted MaxDRL that excludes the area near the saturation brightness by reducing the gain by the number of steps to avoid composite exposure.

[0057] Then, the ratio change module 408 scales each signal threshold of the reference Mix table based on the converted MaxDRL obtained for each of the over-HDR image and the appropriate-HDR image, and derives each signal threshold for the changed Mix table. More specifically, the ratio change module 408 normalizes the value range of the signal values 0 to the signal threshold 1204 of the reference Mix table, and multiplies this by the signal value corresponding to the converted MaxDRL of the over-HDR image. As a result, for the over-HDR image, Over 1End (corresponding to signal threshold 1206) and Suit 1Start (corresponding to signal threshold 1204) are obtained. Also, the ratio change module 408 normalizes the value range of the signal values 0 to the signal threshold 1205 of the reference Mix table, and multiplies this by the signal value corresponding to the converted MaxDRL of the appropriate-HDR image. As a result, for the appropriate-HDR image, Suit 1End (corresponding to signal threshold 1207) and Under 1Start (corresponding to signal threshold 1205) are obtained.

[0058] Thus, if the over-use area and the appropriate-use area of the four types of signal thresholds (Over 1End , Suit 1Start , Suit 1End and Under 1Start ) obtained based on the converted MaxDRL of each image have a value range width of threshold W, there is no need to adjust. That is, even if a changed Mix table is configured based on the four types of signal thresholds, a composite image that avoids the occurrence of a switching step can be obtained. On the other hand, if the condition of the value range width is not satisfied for each use area, a switching step may occur. Therefore, the ratio change module 408 uses the four types of signal thresholds to determine and perform adjustments, for example, as follows, whether adjustment is necessary.

[0059] As an extreme example, Suit 1Start and Under 1StartLet's consider the case where and are the same. In this case, if the ratio change module 408 simply constructs a change mix table, it will look like Figure 13(a). According to this change mix table, in particular the value range for combining the appropriate HDR image and the under-HDR image (Suit 1End ~Under 1Start ) may not be sufficiently secured, and a step may occur during switching. In order to avoid such a situation, the ratio changing module 408 in this modified example is configured such that the appropriate usage range has a value range width equal to the threshold W, as shown in Figure 13(b). 1Start Adjust it. That is, Suit 1Start Convert the appropriate HDR image to a signal value corresponding to MaxDRL (Under 1Start By changing the value to the value obtained by subtracting the threshold W from ), the appropriate usage range is ensured. As a result, a value range for combining the appropriate HDR image and the under-HDR image is ensured, and the occurrence of a transition step is avoided.

[0060] On the other hand, prioritizing the securing of the appropriate usage range may suppress the over-usage range. In the embodiment shown in Figure 13(a) Under 1Start If the value exceeds twice the threshold W, the threshold W range can be maintained for both the appropriate usage range and the over-usage range even with the adjustment shown in Figure 13(b). However, as shown in Figure 14(a), Under 1Start If the value is less than or equal to twice the threshold W, the overuse area can be suppressed by ensuring that the threshold W range is maintained for the appropriate usage area (Figure 14(b)). For more details, see Under 1Start If it is less than or equal to twice the threshold W, and Suit 1Start Under 1Start If it is more than 1 / 2 of the value, the overuse area is suppressed by adjusting the appropriate usage area to have a threshold W width. As a result, a value range for combining the over-HDR image and the appropriate HDR image is not secured, and a transition step may occur. Therefore, in such cases, the ratio change module 408, in order to avoid the occurrence of a transition step in the overuse area and the appropriate usage area as much as possible, adjusts the signal value to, for example, 0 to Under, as shown in Figure 14(c). 1StartAdjust to divide the range of values ​​evenly. Specifically, the ratio change module 408 is used. 1Start Under 1Start Adjust it to half the value.

[0061] In other words, the ratio change module 408 of this modified example derives four types of signal thresholds based on the converted MaxDRL of the over-HDR image and the appropriate HDR image, and then constructs the modified Mix table as follows. (1) Under 1Start >Threshold W × 2, and Under 1Start -Suit 1Start <If the threshold is W, then the appropriate usage range should be set to threshold W. 1Start Adjust (Figure 13(b)) to configure the modified Mix table. (2) Under 1Start ≤ threshold W × 2, and Suit 1Start ≥Under 1Start If the multiplier is ×1 / 2, adjust each signal threshold so that the over-usage range and the appropriate usage range are of equal width (Figure 14(c)) to construct the modified Mix table. (3) Otherwise, configure the modified Mix table without adjusting each signal range. Furthermore, when configuring the modified Mix table, the ratio modification module 408 ensures that the relative relationships of each signal threshold in the reference Mix table are maintained. 1End and Suit 1End The following adjustments shall be made. By doing so, a modified Mix table can be constructed that ensures a threshold W width or an equal width for each signal range where the same HDR image combination is synthesized.

[0062] By using the weighted addition ratios related to the modified Mix table configured in this way, the synthesis module 410 can generate a composite image with reduced transition steps and color distortion. In this modified example, a threshold W is set for each usage range to adjust the signal threshold and configure the modified Mix table, but the implementation of the present invention is not limited to this. For example, if the change in the weighted addition ratio is linearly determined for the signal range in which HDR images are synthesized, a threshold may be set for the slope of the linear function (rate of change of the weighted addition ratio), and the adjustment and configuration of the modified Mix table may be performed.

[0063] As described in the embodiments and modifications above, the image processing apparatus according to the present invention can determine a suitable synthesis ratio for generating a composite image with an expanded dynamic range based on an HDR image.

[0064] In this specification, the input for the synthesis process is described as three types of HDR images taken with different exposure levels. However, the implementation of the present invention is not limited to this, and it goes without saying that it can be applied to the synthesis process of any multiple HDR images. Furthermore, in the generation of the synthesized image, the maximum number of HDR images synthesized for a single signal range is not limited to two types; there may be two or more, as long as the sum of the weighted addition ratios falls within 100%.

[0065] Furthermore, although this specification describes an embodiment in which the weighted addition ratio for each HDR image can be obtained by modifying the reference Mix table to construct a modified Mix table, the embodiment of the present invention is not limited to this. That is, when implementing the present invention, the weighted addition ratio for each HDR image does not need to be defined in the form of a table, but may be defined by a function or the like that derives the weighted addition ratio by performing calculations each time.

[0066] Furthermore, in this specification, in order to synthesize PQ-type HDR images to generate a composite image, the PQ gamma module 406 is described as applying a common gamma related to the same method to create a nonlinear image. However, it should be understood that this nonlinearization is not essential when determining the weighting ratio. That is, the weighting ratio of each HDR image can be determined according to the format of the HDR images synthesized in the synthesis process, or the format of the composite image to be generated.

[0067] Furthermore, the file format of the composite image output after the synthesis process only needs to be able to store at least the upper limit of the dynamic range related to the composite image. For example, the MP4 format shown in Figure 8(b) may be used. In this case, the upper limit should be stored in the meta data 822 in the mdat box 821.

[0068] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0069] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0070] 100: Image processing unit, 101: CPU, 102: ROM, 103: RAM, 104: Recording device, 404: PQ de-gamma module, 405: Exposure adjustment module, 406: PQ gamma module, 408: Ratio change module, 410: Composition module

Claims

1. An image processing device that combines multiple HDR (High Dynamic Range) images taken with different exposure levels, wherein the scene brightness is expressed using an absolute brightness method, For each of the aforementioned multiple HDR images, a means for identifying a signal value that indicates the upper limit of the output dynamic range, A determination means for determining the composite ratio of the plurality of HDR images based on the signal value identified by the identification means, Acquisition means for acquiring a reference signal threshold for switching the trend of the composite ratio of each of the plurality of HDR images, It has, The identification means identifies the signal value from metadata corresponding to the upper limit of the output dynamic range attached to each of the plurality of HDR images, The image processing apparatus is characterized in that the determination means modifies the signal threshold acquired by the acquisition means based on the signal value indicating the upper limit of the output dynamic range of each of the plurality of HDR images, and determines the composite ratio based on the modified signal threshold.

2. The image processing apparatus according to claim 1, wherein the determination means further determines the composite ratio of the plurality of HDR images based on the exposure amounts of the plurality of HDR images.

3. The image processing apparatus according to claim 2, characterized in that the identifying means identifies the signal value indicating the upper limit of the output dynamic range by adjusting the exposure of the plurality of HDR images.

4. The image processing apparatus according to claim 3, wherein the determination means corrects the signal value indicating the upper limit of the output dynamic range of each of the plurality of HDR images based on the exposure amount of the image, and determines the composite ratio of the plurality of HDR images based on the signal value indicating the corrected upper limit of the output dynamic range of the plurality of HDR images.

5. The image processing apparatus according to claim 4, characterized in that the correction is a process of adjusting the gain to match the exposure amount of one of the plurality of HDR images to the exposure amount of another HDR image.

6. The image processing apparatus according to claim 4 or 5, characterized in that the determination means determines the synthesis ratio so as not to synthesize images in signal ranges exceeding the upper limit of the corrected output dynamic range of each of the plurality of HDR images.

7. The system further includes means for obtaining a reference Mix table that defines a reference signal threshold for switching the trend of the blending ratio of each of the multiple HDR images in the signal range for blending the multiple HDR images, The determination means modifies the signal threshold of the reference Mix table based on a signal value indicating the upper limit of the corrected output dynamic range of the plurality of HDR images to construct a modified Mix table, and determines the composite ratio based on the modified Mix table. The image processing apparatus according to any one of claims 4 to 6.

8. Each of the aforementioned multiple HDR images is an image in which gamma is applied and the signal values ​​are nonlinearized. The determination means performs the correction after linearizing the signal value, which indicates the upper limit of the output dynamic range of each of the plurality of HDR images, by applying a degamma. The image processing apparatus according to any one of claims 4 to 7.

9. The image processing apparatus according to any one of claims 1 to 8, characterized in that the identifying means identifies the signal value indicating the upper limit of the output dynamic range of each of the plurality of HDR images, according to the imaging mode set at the time of image acquisition.

10. The image processing apparatus according to any one of claims 1 to 9, characterized in that the signal value indicating the upper limit of the output dynamic range, which the identifying means identifies for each of the plurality of HDR images, is lower than the maximum signal value at the bit depth of the image.

11. The image processing apparatus according to any one of claims 1 to 10, characterized in that the signal range in which the composite ratio is determined by the determination means is the luminance range.

12. The image processing apparatus according to any one of claims 1 to 11, characterized in that the signal range in which the composite ratio is determined by the determination means is the signal range of any of the color signals.

13. A synthesis means for generating a composite image by combining the plurality of HDR images based on the synthesis ratio determined by the determination means, Output means for outputting the composite image generated by the synthesis means as an image file, It further possesses, The output means outputs the image file containing signal value information indicating the upper limit of the output dynamic range of the composite image. The image processing apparatus according to any one of claims 1 to 12.

14. The identification means identifies a signal value indicating the upper limit of the output dynamic range of the HDR image captured at the lowest exposure among the plurality of HDR images. The image processing apparatus according to claim 13, characterized in that the upper limit of the output dynamic range of the composite image is determined based on a signal value indicating the upper limit of the output dynamic range of the HDR image captured at the lowest exposure among the plurality of HDR images and the exposure amount of the said image.

15. The image processing apparatus according to any one of claims 1 to 14, characterized in that the amount of exposure is determined based on the aperture, shutter speed in seconds, and ISO sensitivity at the time of image capture.

16. The image processing apparatus according to any one of claims 1 to 15, characterized in that each of the plurality of HDR images is an image represented using the PQ (Perceptual Quantization) method.

17. An image processing apparatus according to any one of claims 1 to 16, An imaging means for capturing the multiple HDR images with different exposure levels, An imaging device characterized by having the following features.

18. A control method for an image processing device that synthesizes multiple HDR (High Dynamic Range) images taken with different exposure levels, wherein the scene brightness is expressed using an absolute brightness method, A selection step for each of the aforementioned multiple HDR images, which involves identifying a signal value that indicates the upper limit of the output dynamic range, A determination step in which the synthesis ratio of the plurality of HDR images is determined based on the signal value identified in the specified step, An acquisition step to acquire a reference signal threshold for switching the trend of the composite ratio of each of the plurality of HDR images, It has, In the aforementioned specific step, the signal value is identified from metadata corresponding to the upper limit of the output dynamic range attached to each of the plurality of HDR images. A control method characterized in that, in the determination step, the signal threshold acquired in the acquisition step is changed based on the signal value indicating the upper limit of the output dynamic range of each of the plurality of HDR images, and the composite ratio is determined based on the changed signal threshold.

19. A program that causes a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 16.

Images