Image processing device, imaging device, control method, and program

Abstract

To determine a composition ratio suitable for generation of a composite image having an extended dynamic range.SOLUTION: An image processing device includes: acquisition means that, for each of a plurality of HDR (High Dynamic Range) images photographed with different amounts of exposure, acquires exposure information of photographing and an upper limit value of a dynamic range expressed in the image; determination means that, for a composition process for composing the plurality of HDR images to generate a composite image in which the dynamic range is further extended, determines a composition ratio of each HDR image for each signal region of the composite image, based on the exposure information and the upper limit value of the dynamic range acquired by the acquisition means.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an imaging device, a control method, and a program, and more particularly to a technique for synthesizing high dynamic range image signals. [Background technology]

[0002] There is a technology that generates a high dynamic range (HDR) composite image with an expanded dynamic range by combining multiple standard dynamic range (SDR) images captured at different exposures. Patent Document 1 describes a technology that generates a high dynamic range (HDR) composite image by combining three types of SDR images—an appropriate image captured at a correct exposure, an underexposed image captured at an underexposed exposure, and an overexposed image captured at an overexposed exposure—according to a predetermined combination ratio. More specifically, brightness thresholds Y1, Y2, Y3, and Y4 (see FIG. 10 of Patent Document 1) are set as combination standards, and combination is controlled so that an overexposed image is used in a brightness range darker than Y1, an appropriate image is used in a brightness range between Y2 and Y3, and an underexposed image is used in a brightness range brighter than Y4. Furthermore, for the intermediate ranges between Y1 and Y2 and between Y3 and Y4, combination is controlled so that the combination ratios (weighted addition coefficients) of the overexposed image and the appropriate image and the appropriate image and the underexposed image are gradually changed. By using this type of synthesis control, an HDR composite image with an appropriately expanded dynamic range can be obtained from SDR images under three different exposure conditions.

[0003] Furthermore, in recent years, the performance of light-emitting elements such as LEDs has improved, leading to the emergence of display devices known as HDR displays, which have a wider dynamic range of display brightness than conventional displays. Such display devices are capable of more faithfully displaying images with high-brightness color and detail (HDR images). The signal characteristics that represent the relationship between the video signal level and display brightness in HDR images are specified by the EOTF (Electro-Optical Transfer Function), and the following two methods are used. One is the HLG (Hybrid Log Gamma) method, standardized in ARIB STD-B67, which converts the video signal level into a relative value of the display brightness, resulting in a display brightness that corresponds to the maximum brightness that the display device can output. The other is the PQ (Perceptual Quantization) method, standardized in SMPTE ST 2084 or ITU-R BT.2100, which limits the video signal level to a maximum of 10,000 nits (or cd / m 2 ) to an absolute value of display luminance. Therefore, when displaying an HDR image obtained by capturing a scene, the former method converts the scene luminance to a display luminance that corresponds to the maximum luminance that the display device can output, while the latter method converts the scene luminance to a display luminance that is absolutely determined regardless of the display device. Therefore, when assuming display on a display device that employs the PQ method, for example, during encoding in the imaging device, it is necessary to convert the image signal of the scene luminance to indicate an absolute luminance value and generate an HDR image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-240031 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, by acquiring and combining such HDR images with different exposure amounts for a scene, it is possible to generate a composite image with an even wider dynamic range. That is, as in Patent Document 1, by combining three types of HDR images, namely, an appropriate HDR image obtained by shooting with a proper exposure, an under-HDR image obtained by underexposure, and an over-HDR image obtained by overexposure, it is possible to generate a composite image that expresses more detail.

[0006] On the other hand, with PQ encoding, which expresses scene luminance as absolute, the peak luminance (maximum display luminance, maximum output dynamic range) contained in an HDR image can vary depending on the exposure level, even when capturing the same scene. This is because the scene luminance at which sensor output saturates varies depending on the exposure level, and different gamma curves are used for conversion to assign absolute display luminance to the same scene luminance. For example, as shown in Figure 1, the input-output characteristics (relationship between input step number and output luminance) for two shooting modes with different exposure levels have different peak luminances (maximum output luminance). Here, the input-output characteristics 11 for the high-exposure shooting mode are represented by a solid line, and the input-output characteristics 12 for the low-exposure shooting mode are represented by a dashed-dotted line. As shown in the figure, the two shooting modes share common input-output characteristics except for the high-luminance range, resulting in the same display luminance regardless of the exposure level. However, in the high-luminance range, the peak luminance differs between values 13 and 14 due to differences in saturated luminance. The value 15 indicates the maximum value (1023) in 10 bits, and corresponds to the maximum display brightness of 10,000 nits in the PQ system.

[0007] Therefore, simply applying a compositing method such as that described in Patent Document 1 may not result in a desirable composite image. More specifically, the compositing method described in Patent Document 1 is based on 8-bit SDR images in which a maximum pixel value of 255 is assigned for each exposure condition, and therefore it is possible to always refer to the pixel values of both images in the luminance range in which two types of images are combined. However, when combining HDR images with different exposures, in the luminance range in which the two types of images are combined, subjects exceeding the peak luminance may not be expressed in the high-exposure HDR image, and a desirable composite result may not be obtained.

[0008] For example, consider a case where the blending ratio of Patent Document 1 is applied to an appropriate HDR image having a luminance distribution as shown in histogram 24 in FIG. 2 (where the peak luminance of the appropriate HDR image is 25). In the figure, dashed-two-dot line 21, solid line 22, and dashed-dotted line 23 indicate the weighted addition coefficients of the over-HDR image, the appropriate HDR image, and the under-HDR image, respectively, corresponding to the blending ratios of Patent Document 1, and the sum of the coefficients is 1 across the entire luminance range. In this case, as shown in the figure, if the peak luminance of 25 of the appropriate HDR image is included in the luminance range (Y3 to Y4) where the appropriate HDR image and the under-HDR image are blended, the appropriate HDR image is not blended in the luminance range from the peak luminance to Y4, resulting in an unsuitable blending result. In other words, because the appropriate HDR image does not include pixels with a peak luminance of 25 or higher due to saturation of the sensor output, there is a possibility that unnatural appearance may occur in the area of the blended image where subjects near the peak luminance are distributed.

[0009] As in the above-described example, when a plurality of images with different peak luminances are combined, an unnatural image may result.

[0010] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an image processing device, an imaging device, a control method, and a program that determine an appropriate combination ratio for generating a composite image with an expanded dynamic range. [Means for solving the problem]

[0011] In order to achieve the above-mentioned object, an image processing device of the present invention is an image processing device that synthesizes a plurality of HDR (High Dynamic Range) images that are taken with different exposure amounts and in which scene luminance is expressed in an absolute luminance format, and includes: a specifying means that specifies a signal value that indicates an upper limit value of an output dynamic range for each of the plurality of HDR images; a determining means that determines a synthesis ratio of the plurality of HDR images based on the signal value specified by the specifying means; and an obtaining means that obtains a reference signal threshold value that switches the tendency of the synthesis ratio of each of the plurality of HDR images, The specifying means specifies a signal value indicating an upper limit of the output dynamic range from information based on the setting of the shooting mode when each of the plurality of HDR images was captured; The determination means is characterized in that it changes the signal threshold value acquired by the acquisition means based on a signal value indicating the upper limit value of the output dynamic range of each of the multiple HDR images, and determines the synthesis ratio based on the changed signal threshold value. [Effects of the Invention]

[0012] With this configuration, the present invention makes it possible to determine an appropriate blending ratio for generating a blended image with an expanded dynamic range. [Brief explanation of the drawings]

[0013] [Figure 1] A diagram to explain the input / output characteristics for two shooting modes with different exposure amounts. [Figure 2] FIG. 10 is a diagram showing a manner in which a weighted addition ratio based on an SDR image is applied to synthesis processing of an HDR image. [Figure 3] FIG. 1 is a block diagram illustrating a hardware configuration of an image processing device 100 according to an embodiment and a modification of the present invention. [Figure 4] FIG. 1 is a block diagram illustrating a module configuration of a synthesis process according to a first embodiment of the present invention. [Figure 5] 1 is a flowchart illustrating a synthesis process according to a first embodiment of the present invention. [Figure 6] FIG. 1 is a diagram illustrating the difference in characteristics between an OETF applied according to the exposure amount of a photograph and an OETF applied in relation to the generation of a composite image. [Figure 7] FIG. 1 is a diagram illustrating a reference mix table and a modified mix table according to an embodiment of the present invention. [Figure 8] FIG. 10 is a diagram illustrating an example of an output file format of a composite image according to an embodiment and a modification of the present invention. [Figure 9] A diagram illustrating MaxDRL determined according to the shooting mode [Figure 10] FIG. 10 is a block diagram illustrating a module configuration of a synthesis process according to a second embodiment of the present invention. [Figure 11] FIG. 10 is a block diagram illustrating a module configuration of a development process according to a second embodiment of the present invention. [Figure 12] FIG. 10 is a diagram illustrating a reference mix table according to a second modification of the present invention. [Figure 13] FIG. 10 is a diagram illustrating a modified Mix table according to a second modification of the present invention. [Figure 14] FIG. 10 is another diagram illustrating a modified Mix table according to the second modification of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] [Embodiment 1] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0015] In the embodiment described below, an example of an image processing device in which the present invention is applied to an image processing device capable of generating a composite image with an expanded dynamic range by combining HDR images encoded in the PQ format is described. However, the present invention can be applied to any device capable of deriving the combination ratio of each HDR image involved in generating the composite image.

[0016] In addition, in this specification, unless otherwise specified, an "HDR image" will be described as an image (PQ signal) to which PQ encoding has been applied and configured so that absolute display luminance can be obtained by decoding. Furthermore, an image with an expanded dynamic range obtained by combining HDR images will be described simply as a "composite image" to distinguish it from an HDR composite image obtained by combining conventional SDR images and the input (source) HDR image. In the following description, the composite image will be a PQ signal, just like the HDR image.

[0017] Here, a minimum of 10-bit depth is required to represent HDR characteristics with a PQ signal, and HDR images and composite images are not files that store 8-bit images, such as JPEG format. For example, HDR images and composite images may be HEIF files stored using a container of the High Efficiency Image File Format (hereinafter referred to as HEIF), an image file format developed by the Moving Picture Experts Group (MPEG) and defined in MPEG-H Part 12 (ISO / IEC 23008-12). HEIF can store not only the original image but also thumbnails, multiple temporally related images, and metadata such as EXIF and XMP in a single file. Furthermore, HEIF can also store 10-bit image sequences encoded with HEVC, making it suitable for storing HDR images and composite images in the present invention. However, the implementation of the present invention is not limited to this. HDR images and composite images may be in any format that can store image data of 10 bits or more, such as RAW data or TIFF data after development processing.

[0018] <<Hardware configuration of image processing device>> 3 is a block diagram showing the hardware configuration of an image processing device 100 according to this embodiment. As shown in the figure, the image processing device 100 according to this embodiment includes a CPU 101, a ROM 102, a RAM 103, a recording device 104, an operation I / F 105, a display device 106, and a system bus 107.

[0019] The CPU 101 performs overall control of the image processing device 100. The ROM 102 is a storage device that stores control programs such as BIOS required to start up the image processing device 100, programs that do not require modification, parameters, and data. The RAM 103 is a storage device that has a work area for the CPU 101, a primary storage area for temporarily storing various data, a load area for various programs, etc.

[0020] The recording device 104 is a recording device that stores various data such as an OS program, various control programs, various software programs executable on the OS program, HDR images used in the compositing process described below, and composite images obtained as a result of the compositing process. The recording device 104 includes, for example, a hard disk or flash memory built into or detachably connected to the image processing device 100, a detachably connected flexible disk, optical disk, magneto-optical disk, IC card, memory card, etc. Therefore, the CPU 101 can control the image processing device 100 by loading various programs stored in the ROM 102 or the recording device 104 into the RAM 103 and executing them.

[0021] The operation I / F 105 is a user interface provided in the image processing device 100, such as a keyboard, mouse, or touch panel. When the operation I / F 105 detects that a user has input an operation, it sends a control signal related to the operation to the CPU 101. The display device 106 is, for example, a liquid crystal display, and displays various information, including a graphical user interface related to the OS and software running on the image processing device 100. The system bus 107 connects the various blocks constituting the image processing device 100 so that they can communicate with each other.

[0022] The image processing device 100 may be a dedicated device for developing and editing captured RAW images, in which case a compositing processing program can be stored in the ROM 102. A general-purpose personal computer (PC) can function as the image processing device 100 by having the CPU 101 load the compositing processing program stored in the recording device 104 into the RAM 103. Furthermore, an imaging device capable of acquiring RAW images by capturing images can also function as the image processing device 100 by having its control unit execute the compositing processing program or by having an image processing unit that performs the compositing processing perform a specified operation.

[0023] <<Composition Processing>> Next, a compositing process executed in the image processing device 100 according to this embodiment for generating a composite image by compositing multiple HDR images captured with different exposure amounts will be described in detail with reference to the drawings. This compositing process is implemented by the CPU 101 reading out a compositing process program from the ROM 102 or the recording device 104, and expanding and executing the program in the RAM 103. Fig. 4 illustrates an example of a module configuration implemented during execution of the compositing process, and Fig. 5 is a diagram illustrating an example of a flowchart for explaining the flow of the compositing process.

[0024] In the following description, it is assumed that the multiple HDR images captured with different exposure amounts and used as input for the compositing process are all HDR PQ images (hereinafter referred to as PQ images) encoded using the PQ method. Furthermore, in the compositing process of this embodiment, the input PQ images are assumed to be three types: an appropriate PQ image 402 obtained by capturing images with a proper exposure, an over-PQ image 401 obtained by capturing images with an overexposure, and an under-PQ image 403 obtained by capturing images with an underexposure. The over-PQ image 401, the appropriate PQ image 402, and the under-PQ image 403 may be images obtained by capturing images of a common scene sequentially in shooting modes with different exposure settings for the purpose of compositing a composite image. In this embodiment, for convenience, shooting images with different exposure settings will be described as shooting images with different shooting modes, but it will be understood that the implementation of the present invention is not limited to this.

[0025] In this embodiment, each input PQ image is assumed to be accompanied by metadata containing exposure information indicating the exposure setting of the imaging device when the PQ image was captured, and information on the OETF (Opto-Electronic Transfer Function) used to encode the PQ image. The exposure information may be, for example, information on the aperture, shutter speed, and ISO sensitivity set at the time of capture. The OETF is the product of an OOTF (Opto-Optical Transfer Function) based on the production intent of the output image, which is preset for each exposure setting or set by the user, and the inverse function of the reference EOTF defined in ITU-R BT.2100. Furthermore, each PQ image is assumed to be accompanied by metadata containing MaxDRL (Maximum Dynamic Range Level) as described in JP 2020-039118 A. MaxDRL is the maximum PQ signal output value after applying the corresponding OETF during development and encoding of the captured image signal, and is the peak luminance value (upper limit) of the dynamic range that can be represented by each PQ image. In this embodiment, MaxDRL is described as a peak luminance value, but it may also be a value indicating the nit value corresponding to the peak luminance value. MaxDRL is used to determine the dynamic range of an input image when tone mapping an HDR image to an HDR or SDR signal with a narrower dynamic range. In this specification, the three types of input PQ images may be input in different HEIF files, or may be stored in a single HEIF file.

[0026] In S501, the over PQ image 401, the appropriate PQ image 402, and the under PQ image 403 are read from the recording device 104 and developed in the RAM 103. By the processing of this step, the PQ images to be synthesized into the synthesized image are input.

[0027] In S502, the exposure of the three types of PQ images is adjusted. Here, the accuracy of exposure adjustment may decrease if the PQ images are left as nonlinear signals, so the PQ images are linearized first before exposure adjustment. More specifically, the exposure adjustment of the three types of PQ images is achieved by linearization using the PQ degamma module 404, exposure adjustment using the exposure adjustment module 405, and nonlinearization using the PQ gamma module 406.

[0028] The linearization by the PQ degamma module 404 may be performed by referencing the information of the OETF used for encoding, which is attached to each PQ image, and applying the inverse function (degamma) of the OETF.

[0029] The exposure adjustment by the exposure adjustment module 405 is performed by applying a gain based on the exposure step to each linearized image. In this embodiment, exposure adjustment is performed based on the appropriate exposure (the exposure used to acquire the appropriate PQ image 402), and gain is applied to the linearized overexposed and underexposed images, correcting each to the same exposure as the appropriate exposure. For example, if the appropriate PQ image 402 was captured with an aperture of F5.6, a shutter speed of 1 / 60, and an ISO sensitivity of 200, and the under-PQ image 403 was captured with an aperture of F5.6, a shutter speed of 1 / 500, and an ISO sensitivity of 200, the exposure step is three steps. Therefore, the exposure adjustment module 405 corrects the exposure to the same as the appropriate PQ image 402 by increasing the gain of the linearized image of the under-PQ image 403 by three steps.

[0030] The PQ gamma module 406 then applies a common OETF to each exposure-corrected image, thereby nonlinearizing the images. As shown in FIG. 6 , the OETF applied by the PQ gamma module 406 is an OETF (dashed-dotted line 603) that corresponds to a wider dynamic range than the OETF (solid line 601) applied to the input PQ image. As shown in the figure, the OETF applied by the PQ gamma module 406 may correspond to a higher luminance 604 so as to obtain a composite image with an expanded dynamic range in the subsequent composition process. Here, the OETF applied by the PQ gamma module 406 may be a fixed function that is set in advance, such as an inverse function of the reference EOTF defined in ITU-R BT.2100. Alternatively, the OETF may be adaptively changed so that the maximum luminance (luminance 604) is set according to the under-PQ image 403′.

[0031] By performing the process of S502, an over-exposed PQ image, an appropriate PQ image, and an under-exposed PQ image are obtained. Hereinafter, for convenience, the PQ images after the process of S502 will be referred to as the over-exposed PQ image 401', the appropriate PQ image 402', and the under-exposed PQ image 403'. Here, the over-exposed PQ image 401' has less noise in dark areas than the appropriate PQ image 402', and the under-exposed PQ image 403' has less blown-out highlights in bright areas than the appropriate PQ image 402', resulting in an image with gradation.

[0032] In S503, the misalignment correction module 407 detects misalignment between the over-PQ image 401′ and the appropriate PQ image 402′, and between the under-PQ image 403′ and the appropriate PQ image 402′, and performs a process to correct the misalignment if any is detected. Any known method may be used to detect and correct misalignment. For example, the image may be divided into blocks, edges may be extracted, and a motion vector that minimizes the sum of absolute differences (SAD) between pixel values for each block may be derived. Then, affine coefficients obtained based on the motion vectors may be used to affine-transform the image to be corrected, thereby correcting the misalignment. When misalignment correction is performed based on the appropriate PQ image 402′, the affine transformation may be performed on the other images, i.e., the over-PQ image 401′ and the under-PQ image 403′. In this embodiment, since detection accuracy is improved by reducing the difference in brightness between images, the description will be given assuming that misalignment detection and correction is performed after the processing of S502, but it will be understood that there is no dependency between these, and therefore the order of processing may be reversed.

[0033] In S504, the ratio change module 408 determines the blending ratio (weighted addition ratio, weighted addition coefficient) for each image. In this embodiment, the weighted addition ratio for each image is determined by modifying the reference mix table 409 as shown in FIG. 7(a) to create a modified mix table and referencing the modified mix table. As shown in the figure, the reference mix table 409, similar to Patent Document 1, distributes the weighted addition ratios of an over-PQ image 401′, an appropriate PQ image 402′, and an under-PQ image 403′ so that they differ depending on the luminance value. The reference mix table 409 includes an over-PQ image ratio 701 (two-dot chain line), an appropriate image ratio 702 (solid line), and an under-PQ image ratio 703 (one-dot chain line), which indicate the weighted addition ratio for each image within the PQ display luminance range (corresponding to an upper limit of 10,000 nit). Here, the weighted addition ratios for each image in the reference mix table 409 are configured so that the sum total for each luminance value is 1 (100%). Therefore, each luminance range is divided into a range in which only the over-PQ image 401' is used, a range in which the over-PQ image 401' and the appropriate PQ image 402' are combined, a range in which only the appropriate PQ image 402' is used, a range in which the appropriate PQ image 402' and the under-PQ image 403' are combined, and a range in which only the under-PQ image 403' is used. These range divisions are defined based on a reference point (luminance threshold) set for each image that switches the tendency of combining.

[0034] The ratio change module 408 reads the reference mix table 409 from, for example, the ROM 102 or the recording device 104, and then acquires the exposure information and MaxDRL assigned to each of the over PQ image 401, the appropriate PQ image 402, and the under PQ image 403. The ratio change module 408 then changes the predetermined brightness thresholds provided in the reference mix table 409 based on the acquired exposure information and MaxDRL, thereby constructing a table indicating weighted addition ratios that optimize the composition results of the subsequent composition module 410. In this embodiment, the predetermined brightness thresholds provided are a brightness threshold 704 at which the weighted addition ratio for the over PQ image 401' is 0% and a brightness threshold 705 at which the weighted addition ratio for the appropriate PQ image 402' is 0%, and these are subject to change.

[0035] As described above, the luminance ranges for combining the over-PQ image 401′ and the appropriate PQ image 402′ and the luminance ranges for combining the appropriate PQ image 402′ and the under-PQ image 403′ may not appear as appropriate luminance value signals in images captured with a high exposure. In other words, applying a weighted addition ratio exceeding 0% to an image that does not represent a corresponding luminance range, or applying a weighted addition ratio less than 100% to the other image, may result in undesirable compositing results. Therefore, the ratio change module 408 of this embodiment changes the luminance thresholds 704 and 705 to values that do not exceed the maximum display luminance that can be represented by the corresponding images.

[0036] In this embodiment, the exposure of the over PQ image 401 and the under PQ image 403 is adjusted based on the appropriate PQ image 402, so the maximum display luminance of the appropriate PQ image 402' is the MaxDRL of the appropriate PQ image 402. On the other hand, the maximum display luminance of the over PQ image 401' is different from the MaxDRL of the over PQ image 401 due to the exposure adjustment in S502, so the ratio change module 408 derives it using the following formula. JPEG0007720751000001.jpg13148 That is, the ratio change module 408 applies the EOTF (OETF) to the MaxDRL of the over PQ image 401 in the same way as in S502. -1 ) to linearize the over-PQ image 401 and the appropriate PQ image 402, and then apply a gain related to the exposure difference between the over-PQ image 401 and the appropriate PQ image 402. The ratio change module 408 then applies the OETF to the obtained value to nonlinearize it, and derives the maximum display luminance for the over-PQ image 401′.

[0037] Therefore, the ratio change module 408 changes the brightness threshold 704 and brightness threshold 705 to the maximum display brightness of the over-PQ image 401' and the appropriate PQ image 402' obtained in this manner, respectively, to obtain a new changed Mix table as shown in FIG. 7(b). More specifically, the ratio change module 408 linearly scales the over-PQ image ratio 701 so that the brightness threshold 704 becomes the brightness threshold 714 (the maximum display brightness of the over-PQ image 401'), thereby obtaining a changed over-PQ image ratio 711. The ratio change module 408 linearly scales the appropriate image ratio 702 so that the brightness threshold 705 becomes the brightness threshold 715 (the maximum display brightness of the appropriate PQ image 402'), thereby obtaining a changed appropriate image ratio 712. The ratio change module 408 also linearly scales the under-PQ image ratio 703 at the same ratio as the appropriate image ratio 712, thereby obtaining a changed under-PQ image ratio 713.

[0038] Note that, because the scaling ratios applied to the overexposed image ratio 711 and the appropriate image ratio 712 may differ, the luminance range for combining the overexposed PQ image 401' and the appropriate PQ image 402' may be determined based on the overexposed image ratio 711. That is, the luminance range for the appropriate image ratio 712 may be changed so that the weighted addition ratio of the appropriate PQ image 402' becomes 100% at the luminance threshold 714. Furthermore, the luminance range may be changed so that the weighted addition ratio of the appropriate PQ image 402' begins to increase from 0% at the luminance value 716 where the weighted addition ratio for the overexposed image ratio 711 begins to decrease. That is, the modified Mix table may be configured by adjusting the weighted addition ratios, particularly for the high-exposure image, so that the total weighted addition ratio of the images to be combined becomes 100% at any luminance value.

[0039] In S505, the synthesis module 410 performs a luminance-specific synthesis process using the aligned over PQ image 401', the appropriate PQ image 402', and the under PQ image 403' based on the modified Mix table constructed in S504 to generate a synthesized image. That is, the synthesis module 410 stores pixel values at the same positions in the aligned over PQ image 401' for dark areas (luminance ranges where the over image ratio 711 is 100%). The synthesis module 410 also stores pixel values at the same positions in the appropriate PQ image 402' for midtone areas (luminance ranges where the appropriate image ratio 712 is 100%). The synthesis module 410 also stores pixel values at the same positions in the under PQ image 403' for bright areas (luminance ranges where the under image ratio 713 is 100%). Furthermore, for regions from dark to midtones, the synthesis module 410 derives and stores pixel values by multiplying the pixel values at the same positions of the appropriate PQ image 402' and the over-PQ image 401' after alignment by their respective weighted addition coefficients and adding the resulting values. For regions from midtones to bright areas, the synthesis module 410 derives and stores pixel values by multiplying the pixel values at the same positions of the appropriate PQ image 402' and the under-PQ image 403' after alignment by their respective weighted addition coefficients and adding the resulting values. This makes it possible to obtain a synthesized image in which the luminance range where images switch is smoothed and the scene is expressed in an appropriate manner from dark to bright areas.

[0040] The generated composite image may be stored in the recording device 104 as, for example, a new PQ image in an HEIF format image file. An HEIF format image file has a file structure as shown in FIG. 8(a). That is, a HEIF format file is composed of an ftyp box 801 indicating the file format, a meta box 802 storing metadata including a thumbnail image 811, and an mdat box 803 storing encoded data (main image 812). Therefore, the composition module 410 stores data obtained by encoding the generated composite image in the PQ format in the mdat box 803, and stores a thumbnail image of the composite image and the MaxDRL related to the composite image in the meta box 802, thereby generating a file.

[0041] Here, the MaxDRL of the composite image may be, for example, the maximum display luminance of the under-PQ image 403'. That is, since the upper limit of the dynamic range of the composite image is determined mainly based on the under-PQ image 403 that is synthesized by gain-up, the synthesis module 410 may derive information on the maximum display luminance using the following formula and use it as MaxDRL: JPEG0007720751000002.jpg14135 In other words, the MaxDRL of the under-PQ image 403 is converted into a linearized value by applying the inverse characteristics of the OETF used for encoding, and this value is then gain-up and made nonlinear by applying the common OETF, and the resulting value is stored as the MaxDRL of the composite image.

[0042] Alternatively, MaxDRL does not need to be determined based on the MaxDRL of the image. For example, MaxDRL information may be stored in advance for the shooting mode used to acquire the input PQ image, as shown in Fig. 9, and the corresponding value may be used. For example, if three PQ images captured in a normal shooting mode with ±3 stops are input, the composition module 410 can determine the MaxDRL of the composite image to be 888 (2906 nits) based on the information (maximum signal PQ code value) stored in the recording device 104. This allows the output composite image to be saved as a file in an easily usable format.

[0043] [Embodiment 2] In the above-described embodiment, the input HDR image to be subjected to the compositing process is a PQ image. However, the present invention is not limited to this. That is, the input image may be any HDR image in which scene luminance is expressed using a method different from the method of assigning a fixed value range regardless of exposure conditions, as in SDR images. For example, the input image may be a RAW image (12-14 bits) with a higher resolution than a PQ image (10 bits). In this case, for example, a module configuration such as that shown in FIG. 10 may be used to implement the compositing process by inputting multiple RAW images obtained by shooting with different exposure amounts. Here, the input RAW images are assumed to be three types: an appropriate RAW image 1002 obtained by shooting with a proper exposure, an overexposed RAW image 1001 obtained by shooting with an overexposed image, and an underexposed RAW image 1003 obtained by shooting with an underexposed image. In FIG. 9, components common to the module configuration related to the compositing process of the first embodiment are designated by the same reference numerals, and detailed description thereof will be omitted below.

[0044] The compositing process of this embodiment involves conversion (development process) of three types of input undeveloped RAW images into PQ images by the development process module 1004. The development process executed by the development process module 1004 is subdivided into processes by the functional modules shown in FIG. 11. Specifically, a white balance module 1101 performs white balance processing on the input RAW images to make white white, and applies gains to R, G, and B so that R, G, and B in areas that should be white have the same signal value. A noise reduction module 1102 then performs noise reduction processing on the input images to reduce sensor-induced noise that is not derived from the subject image. A color interpolation module 1103 then applies color interpolation processing to the input color mosaic image to generate a color image in which R, G, and B color information is consistent for all pixels. The generated color image then undergoes matrix conversion processing by a matrix conversion module 1104 and gamma conversion processing by a gamma conversion module 1105 to generate a basic color image. Here, for example, an OETF corresponding to the shooting mode and assigned to each RAW image is used for the gamma conversion process by the gamma conversion module 1105. After that, the color adjustment module 1106 applies image correction processes to the color image to improve the appearance of the image, such as saturation emphasis, hue correction, and edge emphasis, and outputs a PQ image.

[0045] In this way, even when a linear RAW image is input, by first applying the OETF to convert it into a PQ image, a similarly suitable composite image can be generated using the functional module related to the synthesis processing of embodiment 1.

[0046] In this embodiment, the functional modules related to the compositing process of Embodiment 1 are used so that the present embodiment can be used in combination with the compositing process of Embodiment 1, but it will be readily understood that the present invention is not limited to this. That is, the information on the luminance range of each image used to determine the compositing ratio is obtained by adjusting the exposure of linearized images, so the development process by the development process module 1004 and the linearization process by the PQ degamma module 404 do not have to be performed.

[0047] [Variation 1] In the above-described embodiment, an aspect has been described in which the OETF used / to be used for development and encoding of an HDR image that is input to the compositing process can be obtained. However, since including OETF information (or information on the EOTF corresponding to its inverse characteristic) as metadata in an image file can increase the file size, such information may not be included. In this case, the inverse function of the OETF used for linearization by the PQ degamma module 404 may be the reference EOTF defined in ITU-R BT.2100, although this may result in reduced accuracy. In this case, the inverse function of the reference EOTF may be used for nonlinearization by the PQ gamma module 406.

[0048] [Variation 2] In the above-described embodiment and modified examples, it has been explained that the HDR images that are input to the compositing process are obtained by photographing with different exposure amounts, and that the magnitude relationship of the peak luminance values in the dynamic range is distributed in a manner suitable for the compositing process. In other words, it has been explained that the weighted addition ratio that allows the input images to be suitably composited is derived by scaling the reference Mix table 409 based on the MaxDRL of the HDR image with adjusted exposure.

[0049] However, for example, if the brightness conversion characteristics of the input HDR image have been changed by retouching applied after shooting (MaxDRL changes), the brightness range in which the two images are combined may be suppressed, and the transition between the images may not be expressed smoothly. In other words, if the brightness range allocated to the transition is narrow, the change in the weighted addition ratio in that brightness range may be abrupt, which may result in an unnatural appearance in the generated composite image (hereinafter referred to as a transition step).

[0050] Furthermore, for example, when the OETF cannot be acquired for the input HDR image as in Modification Example 1, or when retouching that changes the luminance conversion characteristics (changes the EOTF required for linearization) is applied, linearization and exposure adjustment may be insufficient. In other words, since it is not possible to assign appropriate luminance to subjects near the saturation luminance of each HDR image, color distortion and the like may occur in the composite image.

[0051] In this modification, a method for generating a modified Mix table related to determining the weighted addition rate of each HDR image in order to prevent such switching steps and color distortion from occurring in the composite image will be described with reference to the drawings. That is, in this modification, an aspect will be described in which the weighted addition rate of each image can be suitably set not only according to the MaxDRL of the input HDR image, but also according to the combination thereof.

[0052] In this embodiment, unlike the above-described embodiments, a reference mix table as shown in FIG. 12 is provided for Max(R,G,B), which is the maximum value of any one of the R, G, and B color signals of a reference pixel of an input image. Similar to the reference mix table 409 shown in FIG. 7(a), the reference mix table of FIG. 12 indicates weighted addition rates for an over HDR image with a two-dot chain line 1201, an appropriate HDR image with a solid line 1202, and an under HDR image with a dashed dot line 1203. The difference from the reference mix table 409 shown in FIG. 7(a) is that the reference mix table of FIG. 12 specifies the maximum signal value (Max(R,G,B)) of R, G, and B instead of a luminance value. For ease of explanation, the signal threshold 1206 at which the weighted addition rate of an over HDR image starts to decrease from 100% and the weighted addition rate of an appropriate HDR image starts to increase from 0%, or a threshold corresponding thereto, will be referred to as Over 1End The signal threshold 1204 at which the weighted addition rate of the over-HDR image reaches 0% and the weighted addition rate of the appropriate HDR image reaches 100%, or a threshold corresponding thereto, is referred to as Suit 1StartThe signal threshold 1207 at which the weighted addition rate of the suitable HDR image starts to decrease from 100% and the weighted addition rate of the under-HDR image starts to increase from 0%, or a threshold corresponding thereto, is referred to as Suit 1End The signal threshold 1205 at which the weighted addition rate of the appropriate HDR image reaches 0% and the weighted addition rate of the under HDR image reaches 100%, or a threshold corresponding thereto, is referred to as the Under HDR. 1Start The ratio change module 408 of this modified example configures a changed Mix table for synthesis processing in the synthesis module 410 by adjusting these specified signal thresholds 1204 to 1207.

[0053] <<Overview of the Changed Mix Table Configuration>> Here, an overview will be given of the configuration of the modified Mix table that determines the weighted addition ratio of each HDR image using the reference Mix table in this modified example.

[0054] As shown in FIG. 6, the aforementioned color distortion occurs primarily due to nonlinear conversion (changes in the slope of the output curve) near saturation luminance values. Therefore, if conversion characteristics applied during development or conversion characteristics taking retouching into account cannot be acquired, accurate linearization of the luminance values of the input HDR image cannot be performed, and consistency between images cannot be ensured when compositing the corresponding value range. On the other hand, excluding the saturation luminance values, linear conversion is likely to have occurred. For this reason, the ratio change module 408 of this modified example determines the number of steps (compositing avoidance steps) at which nonlinear conversion may occur as a constant, and derives the maximum display luminance value (converted MaxDRL) for the over-HDR image and the appropriate HDR image by taking into account this number of compositing avoidance steps. More specifically, the ratio change module 408 derives the converted MaxDRL by reducing the value linearized by applying, for example, a reference EOTF to the MaxDRL of the input image by the value of the number of compositing avoidance steps, and then modifies the reference Mix table based on this.

[0055] The above-described switching step may occur due to suppression of the signal range defined by signal thresholds 1206 to 1204 and the signal range defined by signal thresholds 1207 to 1205. In particular, when the reference Mix table is changed using a converted MaxDRL set to exclude a luminance range near saturation luminance, as in this modified example, suppression of the signal range occurs. Because there is a margin before the signal value saturates for an under-HDR image, in this modified example, the ratio change module 408 assigns weighted addition ratios for the over-HDR image and the appropriate HDR image based on the signal value corresponding to the converted MaxDRL of the appropriate HDR image. Therefore, the range from the signal value 0 to the signal value corresponding to the converted MaxDRL of the appropriate HDR image includes the following two types of ranges. One range is a range in which the weighted addition ratio for the over-HDR image is 100% and a range of 0% to 100% (hereinafter referred to as the over-use range). The other is a range where the weighted addition ratio of the appropriate HDR image is 100% and a range of 0% to 100% (hereinafter referred to as appropriate use range). In this modified example, for the over use range and appropriate use range, a threshold W is set for the width of the signal range that should be secured to perform smooth synthesis, and each signal threshold is adjusted so that the appropriate use range has a range width of at least the threshold W, thereby configuring the modified Mix table.

[0056] In the process performed by the ratio change module 408, first, a converted MaxDRL is derived for the over HDR image and the appropriate HDR image using the following formula: JPEG0007720751000003.jpg14131Specifically, the converted MaxDRL for each image is derived by linearizing the MaxDRL for each image by applying, for example, a reference EOTF, and then applying a common OETF to the linearized value, reducing the gain according to the difference from the appropriate exposure and the number of steps to avoid blending, and then nonlinearizing the value.That is, to avoid inconsistencies that arise due to differences between the inverse characteristics of the OETF used when developing the input image and the characteristics of the EOTF applied for linearization, the ratio change module 408 reduces the gain by the number of steps to avoid blending, thereby obtaining a converted MaxDRL that excludes the vicinity of saturation luminance.

[0057] The proportion change module 408 then scales each signal threshold in the reference Mix table based on the converted MaxDRL obtained for each of the over HDR image and the appropriate HDR image, and derives each signal threshold for the modified Mix table. More specifically, the proportion change module 408 normalizes the range of signal value 0 to signal threshold 1204 in the reference Mix table, and multiplies this by the signal value corresponding to the converted MaxDRL of the over HDR image. As a result, for the over HDR image, the Over 1End (corresponding to signal threshold 1206) and Suit 1Start (corresponding to the signal threshold value 1204) is obtained. In addition, the ratio change module 408 normalizes the range of the signal value 0 to the signal threshold value 1205 of the reference Mix table, and multiplies this by the signal value corresponding to the converted MaxDRL of the suitable HDR image. As a result, the Suit 1End (corresponding to signal threshold 1207) and Under 1Start (corresponding to signal threshold 1205) is obtained.

[0058] In this way, four types of signal thresholds (Over 1End , Suit 1Start , Suit 1End and Under 1Start ) does not need to be adjusted if the over-use range and the appropriate use range have a range width equal to the threshold W. In other words, even if a modified Mix table is constructed based on the four signal thresholds, a composite image can be obtained in which the occurrence of switching gaps is avoided. On the other hand, if the range width conditions are not met for each use range, switching gaps may occur, so the ratio change module 408 uses the four signal thresholds to determine whether or not adjustment is necessary and makes the adjustment, for example, as follows:

[0059] As an extreme example, Suit 1Start and Under 1StartIn this case, if the ratio change module 408 simply creates a modified Mix table, it will look like FIG. 13(a). According to the modified Mix table, the value range (Suit 1End ~Under 1Start ) is not sufficiently secured, and a step may occur when switching. In order to avoid such a situation, the ratio change module 408 of this modified example changes Suit so that the appropriate use range has a range width of the threshold W, as shown in FIG. 1Start Adjust the Suit 1Start The signal value corresponding to the converted MaxDRL of the appropriate HDR image (Under 1Start ) minus the threshold W, the appropriate range is secured. As a result, the range of values for combining the appropriate HDR image and the under-HDR image is secured, and the occurrence of a step at the switching point is avoided.

[0060] On the other hand, by prioritizing the appropriate use range, it is possible to suppress the over-use range. 1Start If the threshold value W is more than twice the threshold value W, the range of the threshold value W can be secured for the appropriate use range and the over-use range even if the adjustment as shown in FIG. 13(b) is performed. However, as shown in FIG. 1Start When the threshold value W is less than twice the threshold value W, the over-usage range can be suppressed by ensuring the range of the threshold value W for the appropriate use range (FIG. 14(b)). 1Start is less than twice the threshold W, and Suit 1Start Under 1Start , the over-use range is suppressed by adjusting the appropriate use range so that it has the width of the threshold W. As a result, a value range for combining the over-HDR image and the appropriate HDR image is not secured, and a switching step may occur. Therefore, in such a case, the ratio change module 408 adjusts the signal value from 0 to Under as shown in FIG. 14(c), for example, to avoid the occurrence of a switching step in the over-use range and the appropriate use range as much as possible. 1StartSpecifically, the ratio change module 408 adjusts the range of Suit 1Start Under 1Start Adjust to half the value of .

[0061] In other words, the proportion change module 408 of this modified example derives four types of signal thresholds based on the converted MaxDRL of the over HDR image and the appropriate HDR image, and then configures a modified Mix table as follows. (1) Under 1Start > Threshold W × 2 and Under 1Start -Suit 1Start <If the threshold is W, then Suit is set so that the appropriate use range is W. 1Start (Fig. 13(b)) to create a modified Mix table. (2) Under 1Start ≦Threshold W×2 and Suit 1Start ≧Under 1Start If it is ×1 / 2, adjust each signal threshold so that the over-usage range and the appropriate use range are equal in width (Fig. 14(c)) and create a modified Mix table. (3) Otherwise, construct a modified Mix table without adjusting each signal range. When constructing the modified Mix table, the ratio change module 408 adjusts the ratio of Over to Over so that the relative relationship between the signal thresholds in the reference Mix table is maintained. 1End and Suit 1End In this way, it is possible to configure a modified Mix table that ensures the width of the threshold W or an equal width for each signal range in which the same combination of HDR images is synthesized.

[0062] By using the weighted addition ratios associated with the modified Mix table configured in this manner, the combining module 410 can generate a combined image in which the occurrence of switching steps and color distortion is reduced. Note that, in this modified example, a threshold value W is set for each use range to adjust the signal threshold and configure the modified Mix table, but the implementation of the present invention is not limited to this. For example, if the change in weighted addition ratio for the signal range in which the HDR image is combined is determined linearly, a threshold value may be set for the slope of the linear function (the rate of change of the weighted addition ratio), and the adjustment and configuration of the modified Mix table may be performed.

[0063] As described in the above embodiment and modified examples, the image processing device according to the present invention can determine an appropriate combination ratio for generating a composite image with an expanded dynamic range based on an HDR image.

[0064] Although the present specification has described the inputs for the compositing process as three types of HDR images captured with different exposure amounts, the present invention is not limited to this example and can be applied to the compositing process of any number of HDR images. Furthermore, in generating a composite image, the maximum number of HDR images to be composited for one signal range is not limited to two, and may be two or more as long as the total weighted addition rate is within 100%.

[0065] Furthermore, although the present specification has described an aspect in which the weighted addition ratio for each HDR image can be obtained by configuring a modified Mix table by modifying the reference Mix table, the implementation of the present invention is not limited to this. That is, when implementing the present invention, the weighted addition ratio for each HDR image does not need to be defined in the form of a table, but may be defined by a function or the like that derives the weighted addition ratio by performing a calculation each time.

[0066] Furthermore, in this specification, in order to generate a composite image by combining HDR images of the PQ method, the PQ gamma module 406 has been described as applying a common gamma related to the same method to perform nonlinearization. However, it should be understood that such nonlinearization is not essential when determining the weighted addition ratio. In other words, the weighted addition ratio of each HDR image may be determined according to the format of the HDR image to be combined in the combination process or the format of the composite image to be generated.

[0067] The file format of the composite image output after the compositing process may be any format that can store at least the upper limit value of the dynamic range of the composite image, such as the MP4 format shown in Fig. 8(b). In this case, the upper limit value may be stored in the meta data 822 in the mdat box 821.

[0068] [Other embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0069] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0070] 100: Image processing device, 101: CPU, 102: ROM, 103: RAM, 104: Recording device, 404: PQ degamma module, 405: Exposure adjustment module, 406: PQ gamma module, 408: Ratio change module, 410: Composition module

Claims

1. 1. An image processing device that synthesizes a plurality of HDR (High Dynamic Range) images that are taken with different exposure amounts, and in which scene luminance is expressed in an absolute luminance format, an identification means for identifying a signal value indicating an upper limit of an output dynamic range for each of the plurality of HDR images; a determining means for determining a combining ratio of the plurality of HDR images based on the signal value identified by the identifying means; an acquisition means for acquiring a reference signal threshold value for switching the tendency of the synthesis ratio for each of the plurality of HDR images; and the specifying means specifies the signal value indicating the upper limit of the output dynamic range from information based on a shooting mode setting when each of the plurality of HDR images was captured; the determining means changes the signal threshold acquired by the acquiring means based on the signal value indicating an upper limit value of the output dynamic range of each of the plurality of HDR images, and determines the combining ratio based on the changed signal threshold.

2. The image processing device according to claim 1 , wherein the determining unit determines a blending ratio of the plurality of HDR images based on the exposure amounts of the plurality of HDR images.

3. The image processing device according to claim 2 , wherein the specifying unit specifies the signal value indicating the upper limit of the output dynamic range by adjusting the exposure of the plurality of HDR images.

4. 4. The image processing device according to claim 3, wherein the determining means corrects the signal value indicating the upper limit of the output dynamic range of each of the plurality of HDR images based on an exposure amount of the image, and determines a combining ratio of the plurality of HDR images based on the signal value indicating the upper limit of the output dynamic range of the plurality of HDR images after the correction.

5. The image processing device according to claim 4 , wherein the correction is a process of adjusting a gain to match the exposure amount of one of the plurality of HDR images with the exposure amount of the other HDR images.

6. 6. The image processing device according to claim 4, wherein the determining means determines the combining ratio so as not to combine images in a signal range exceeding an upper limit value of the corrected output dynamic range of each of the plurality of HDR images.

7. The method further includes means for acquiring a reference mix table that defines a reference signal threshold value for switching a tendency of a blending ratio of each of the plurality of HDR images for a signal range in which the plurality of HDR images are blended, The determining means changes the signal threshold value of the reference mix table based on a signal value indicating an upper limit value of an output dynamic range of the plurality of HDR images after the correction to form a modified mix table, and determines the blending ratio based on the modified mix table.

7. The image processing device according to claim 4, wherein the image processing device is a computer.

8. Each of the plurality of HDR images is an image in which a signal value is nonlinearized by applying gamma; The determining means performs the correction after linearizing the signal value indicating the upper limit value of the output dynamic range of each of the plurality of HDR images by applying de-gamma.

8. The image processing device according to claim 4, wherein the image processing device is a computer.

9. 9. The image processing device according to claim 1, wherein the specifying unit specifies the signal value indicating the upper limit of the output dynamic range of each of the plurality of HDR images according to an imaging mode that was set when the image was captured.

10. 10. The image processing device according to claim 1, wherein the signal value indicating the upper limit of the output dynamic range identified by the identification means for each of the plurality of HDR images is lower than the maximum signal value at the bit depth of the image.

11. 11. The image processing apparatus according to claim 1, wherein the signal range for which the combining ratio is determined by the determining means is a luminance range.

12. 12. The image processing apparatus according to claim 1, wherein the signal range for which the combining ratio is determined by the determining means is the signal range of any one of the color signals.

13. a synthesis means for synthesizing the plurality of HDR images based on the synthesis ratio determined by the determination means to generate a synthesized image; an output means for outputting the composite image generated by the composition means as an image file; and The output means outputs the image file including information on a signal value indicating an upper limit value of an output dynamic range of the composite image.

13. The image processing device according to claim 1, wherein the image processing device is a computer.

14. the specifying means specifies a signal value indicating an upper limit of the output dynamic range of an HDR image captured with the lowest exposure among the plurality of HDR images; 14. The image processing device according to claim 13, wherein an upper limit value of the output dynamic range of the composite image is determined based on a signal value indicating an upper limit value of the output dynamic range of an HDR image captured at the lowest exposure among the plurality of HDR images and an exposure amount of the image.

15. 15. The image processing device according to claim 1, wherein the exposure amount is determined based on an aperture, a shutter speed, and an ISO sensitivity when the image is captured.

16. 16. The image processing device according to claim 1, wherein each of the plurality of HDR images is an image expressed using a PQ (Perceptual Quantization) method.

17. An image processing device according to any one of claims 1 to 16; an imaging means for capturing the plurality of HDR images with different exposure amounts; An imaging device comprising:

18. 1. A control method for an image processing device that synthesizes a plurality of HDR (High Dynamic Range) images that are taken with different exposure amounts, and in which scene luminance is expressed in an absolute luminance format, comprising: a step of identifying a signal value indicating an upper limit of an output dynamic range for each of the plurality of HDR images; a determination step of determining a synthesis ratio of the plurality of HDR images based on the signal value identified in the identification step; an acquisition step of acquiring a reference signal threshold value for switching the tendency of the synthesis ratio for each of the plurality of HDR images; and In the specifying step, the signal value indicating an upper limit value of the output dynamic range is specified from information based on a shooting mode setting when each of the plurality of HDR images was captured, a control method characterized in that, in the determination step, the signal threshold acquired in the acquisition step is changed based on the signal value indicating an upper limit value of the output dynamic range of each of the plurality of HDR images, and the combining ratio is determined based on the changed signal threshold.

19. A program that causes a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 16.

Images