Image processing device and method, electronic device, program, and storage medium

Abstract

To suppress luminance differences in images when converting a plurality of images with gain maps into images having different dynamic ranges.SOLUTION: An image processing device has: conversion means for converting an input image into a first image having a first dynamic range and a second image having a second dynamic range; correction means for correcting a plurality of the first images, which have a predetermined relationship with each other, to align their brightness; and generation means for generating a gain map for converting the second image into an image having the first dynamic range, using the second image corresponding to the first image for each of the plurality of corrected first images.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device and method, an electronic device, a program, and a storage medium, and more particularly to a technique for image conversion using a gain map. [Background technology]

[0002] A conventional image conversion technique is a gain map that stores a gain for each pixel used in image conversion. Because the gain map allows each pixel to have an arbitrary different gain, it is often used in area processing where the difference in correction amount between adjacent pixels is large.

[0003] Furthermore, when applying a gain map to multiple images that are closely related in the time direction, such as continuous still images or consecutive frames of a video, or images that are closely related in the position direction, such as stereo images, where the same subject is captured, the multiple images are often checked as a set. Therefore, it is often desirable that the amount of gain map correction between images be the same.

[0004] With the aim of matching the amount of correction between images that have such a strong correlation, Patent Document 1 discloses the following technology: First, multiple images obtained by photographing the same subject but with different imaging fields are acquired, and a common area of ​​a predetermined size is set at a common position in each image. Then, a shading correction gain is calculated based on the image pair in which the subject images overlap in the set common area.

[0005] Furthermore, Patent Document 2 discloses the following technology. First, two images having a common area captured from different directions are acquired, and a flatness indicating the gradient of the shading component is calculated based on the brightness ratio of the common area. Then, a correction gain is calculated based on the area where the gradient detected based on each flatness is the smallest.

[0006] Considering the recent trend toward HDR (High Dynamic Range) content, let's consider the case of converting SDR (Standard Dynamic Range) content to HDR content. For example, in the luminance range that can be expressed in SDR, the amount of correction required for conversion to HDR is small, while in areas where the luminance difference between HDR and SDR is large and the HDR effect is pronounced, the amount of correction is large. Furthermore, when converting in the opposite direction, from HDR content to SDR content, local contrast processing and other effects can be considered to be expressed as gain, so conversion between HDR and SDR content works well with gain maps. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-026966 [Patent Document 2] International Publication No. 2016 / 194234 Summary of the Invention [Problem to be solved by the invention]

[0008] When performing corrections that result in a large difference in luminance before and after conversion, such as between SDR and HDR, there is a concern that even a slight difference in luminance of the base image may become significant in the corrected image. However, the conventional technologies disclosed in Patent Documents 1 and 2 mainly refer to shading correction, and do not mention correction methods for converting images using different gamma characteristics, such as between SDR and HDR.

[0009] For example, when processing multiple images, if subjects with large differences in brightness are arranged irregularly within the images and the brightness between the images also differs, the magnitude of the correction amount between SDR and HDR will follow the difference in gamma shape, so the difference in correction amount will differ for each subject and the correction amount for the same subject will also differ between images. In such cases, it is difficult to align the correction amounts between images without considering the gamma characteristics.

[0010] The present invention has been made in consideration of the above problems, and aims to suppress brightness differences in the converted images when multiple images are converted into images with different dynamic ranges using a gain map. [Means for solving the problem]

[0011] In order to achieve the above object, the image processing device of the present invention has a conversion means for converting an input image into a first image having a first dynamic range and a second image having a second dynamic range, a correction means for correcting a plurality of the first images having a predetermined relationship with each other so that the brightness is uniform, and a generation means for generating, for each of the corrected plurality of first images, a gain map for converting the second image into an image having the first dynamic range using the second image corresponding to the first image. [Effects of the Invention]

[0012] According to the present invention, when a plurality of images are converted into images having different dynamic ranges using a gain map, it is possible to suppress the luminance difference between the converted images. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a schematic configuration of an imaging apparatus according to an embodiment of the present invention. [Figure 2] 5A and 5B are conceptual diagrams showing an image and a gain map when the gain map is generated in the first embodiment. [Figure 3] 5 is a flowchart of a gain map generation process according to the first embodiment. [Figure 4] FIG. 4 is an explanatory diagram of how to match the brightness of a subject between images in the first embodiment. [Figure 5] FIG. 10 is a diagram showing a sequence during continuous shooting in the second embodiment. [Figure 6]10 is a flowchart of a gain map generation process during continuous shooting in the second embodiment. [Figure 7] FIG. 10 is a conceptual diagram of a stereo image according to a modified example. [Figure 8] FIG. 11 is a conceptual diagram illustrating a process of matching brightness between generated gain maps according to the third embodiment. [Figure 9] FIG. 13 is a conceptual diagram illustrating a process of matching brightness between generated gain maps according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0015] FIG. 1 is a block diagram showing an example of the basic functional configuration of a digital camera 100 (hereinafter referred to as "camera 100") as an example of an imaging device equipped with an image processing device according to an embodiment of the present invention. Note that imaging devices to which the present invention can be applied may be any electronic device equipped with a camera function. Examples of electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, drive recorders, etc. These are merely examples, and the present invention can also be applied to other electronic devices.

[0016] In the camera 100, the optical system 101 is an imaging optical system that includes a lens group, a shutter, an aperture, etc., and forms an optical image of a subject on the imaging plane of the image sensor 102. The lens group includes fixed lenses and movable lenses, and the movable lenses include a lens for image stabilization, a focus lens, a variable magnification lens, etc. The aperture may also function as a mechanical shutter. The operations of the movable lenses, aperture, and shutter are controlled by a CPU 103, which is the main control unit of the camera 100. The optical system 101 may be configured as an integral part of the camera 100, or may be configured to be replaceable.

[0017] The image sensor 102 is, for example, a CMOS image sensor, and has a plurality of pixels, each having a photoelectric conversion region, arranged two-dimensionally. The image sensor 102 also has a color filter having a specific color pattern, with each pixel provided with a filter of one color corresponding to the color pattern. While the present invention does not depend on the color pattern of the color filter, it is assumed here that each pixel is provided with a color filter of one of R (red), G (green), and B (blue) in a primary color Bayer array. The image sensor 102 photoelectrically converts the optical image formed on the light receiving surface by the optical system 101 at each pixel, converting it into an analog image signal indicating luminance information for each pixel.

[0018] An analog image signal generated by the image sensor 102 is converted into a digital image signal by an A / D converter (not shown). The A / D converter may be included in the image sensor 102, or the CPU 103 may perform the A / D conversion. The pixel signals constituting the digital image signal obtained by the A / D conversion are RAW data containing only the luminance components of the colors of the color filters provided in the pixels that generated the signals. The CPU 103 stores this RAW data in the primary storage device 104.

[0019] CPU 103 controls each unit of camera 100 and realizes various functions of camera 100 by transferring a program stored in secondary storage device 107 to primary storage device 104 and executing the program. Note that in the following description, at least some of the functions realized by CPU 103 executing a program may be realized by dedicated hardware such as an ASIC.

[0020] The primary storage device 104 is a volatile storage device such as a RAM, etc. The primary storage device 104 is used by the CPU 103 to execute programs, and is also used as a buffer memory for image data, a work area for image processing, a video memory for display, etc.

[0021] The secondary storage device 107 is a rewritable nonvolatile storage device such as an EEPROM, etc. The secondary storage device 107 stores programs (instructions) that can be executed by the CPU 103, settings for the camera 100, GUI data, etc.

[0022] Recording medium 106 is a rewritable nonvolatile storage device such as a semiconductor memory card, and may be built into camera 100 or configured to be detachable from camera 100. Data (still image data, video data, audio data, etc.) generated by camera 100 can be recorded on recording medium 106. In other words, camera 100 has a read / write function for recording medium 106 and, if recording medium 106 is detachable, an attachment / detachment mechanism. Note that the recording destination of data generated by camera 100 is not limited to recording medium 106; for example, the data may be transmitted to an external device via a communication interface provided by camera 100 and recorded on a recording device accessible by the external device.

[0023] The display unit 108 is configured by, for example, a liquid crystal display. The CPU 103 functions as a display control device for the display unit 108. In a shooting standby state or while a moving image is being recorded, the captured moving image is displayed in real time on the display unit 108, and the display unit 108 functions as an electronic viewfinder. The display unit 108 also displays image data recorded on the recording medium 106 and GUI images such as menu screens.

[0024] The operation unit 109 is a general term for a group of input devices that accept user operations, and may include, for example, buttons, levers, and touch panels. The operation unit 109 may also include input devices that do not require physical operation, such as voice or line of sight. The input devices included in the operation unit 109 are given names according to the functions assigned to them, and examples include a shutter button, menu button, directional keys, a decision (set) button, and a mode switching dial. Different functions may be selectively assigned to one input device.

[0025] The image processing unit 105 applies predetermined image processing to image data (which may be RAW data or image data after development processing) to generate image data in different formats, and acquire and / or generate various types of information. The image processing unit 105 may be, for example, a dedicated hardware circuit such as an ASIC designed to achieve a specific function, or may be configured to achieve a specific function by a programmable processor such as a DSP executing software.

[0026] The image processing applied by the image processing unit 105 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when a color filter is provided on the imaging element, and is a process of interpolating the values ​​of color components that are not included in the individual pixel data that make up the image data.It is also called demosaic processing. The correction processing may include white balance adjustment, tone correction, correction of image degradation caused by optical aberration of the optical system 101 (image restoration), correction of the effects of vignetting of the optical system 101, color correction, and the like.

[0027] The detection process can include detection of characteristic regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. Data processing may include superimposition, gain map generation, area extraction (trimming), synthesis, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or image data for recording. The evaluation value calculation process can include processes such as generating signals and evaluation values ​​used in autofocus (AF) detection, and generating evaluation values ​​used in automatic exposure control (AE). The special effect processing may include adding a blur effect, changing color tones, relighting, and the like. It should be noted that these are examples of processes that the image processing unit 105 can apply, and do not limit the processes that the image processing unit 105 can apply.

[0028] Here, HDR images, SDR images, and gain maps will be described in detail. In this embodiment, an HDR image is an image with a wider dynamic range than an SDR image. For example, SMPTE ST 2084, an HDR standard used in HDR display devices, describes Electronic-Opto Transfer Function (EOTF) characteristics. The HDR image in this embodiment is a YUV image to which Opto-Electronic Transfer Function (OETF) characteristics, which are the inverse characteristics of the EOTF characteristics, are applied. An SDR image is a YUV image with a narrower dynamic range than an HDR image, for example, a YUV image to which sRGB gamma characteristics are applied. In the following description, the gamma of an HDR image is referred to as HDR gamma, and the gamma of an SDR image is referred to as SDR gamma, and the description will be given assuming that the color gamuts are the same. Furthermore, development using HDR gamma is referred to as HDR development, and development using SDR gamma is referred to as SDR development.

[0029] In this embodiment, the signal characteristics that represent the relationship between the video signal level and display brightness in HDR refer to two: PQ (Perceptual Quantization), whose EOTF (Electro-Optical Transfer Function) is standardized in SMPTE ST 2084, and HLG (Hybrid Log Gamma), whose OETF is standardized in ARIB STD-B67.

[0030] First Embodiment The gain map generation process according to the first embodiment, which is performed by the camera 100 having the above configuration, will be described below.

[0031] FIG. 3 is a flowchart of the gain map generation process. First, in S301, the CPU 103 controls the optical system 101 and the image sensor 102 to capture an image and acquire a RAW image. Next, in S303, the image processing unit 105 develops the image acquired in S301 using a gamma for HDR (first gamma).

[0032] Next, in S304, the CPU 103 determines whether there are any previously captured images, and if so, whether the previously captured images have a strong relationship with the current image. Note that the strength of the relationship here refers to whether the images contain similar subjects. Therefore, methods for determining the relationship may include, for example, subject recognition using the images developed in S303 or calculating the positional difference between the images. Note that subject recognition methods may include object recognition using deep learning or other common methods such as analyzing color components or frequency components. Other information may also be used, such as the state of the shutter button, the time of capture, the exposure during capture, and information regarding changes in equipment such as lenses.

[0033] If it is determined in S304 that there is a strong relationship between the previously captured image and the current image, the process proceeds to S305; if there are no previously captured images or if it is determined that there is a weak relationship between the previously captured image and the current image, the process proceeds to S306.

[0034] First, a case where it is determined that there are no images captured in the past or that the relationship between the past images and the current image is weak will be described.

[0035] In S306, the image processing unit 105 develops the RAW image acquired in S301 using an SDR gamma (second gamma). Then, in S307, the image processing unit 105 generates a gain map for converting the SDR image into an HDR image, using the HDR image developed in S303 and the SDR image developed in S306.

[0036] The gain map may use the ratio of YUV or RGB values ​​between the HDR image and the SDR image as is, or it may convert both images into a common gamma color space and use the correction amount in that common gamma color space. For example, the correction amount may be calculated by converting both images into a linear space, or by converting the SDR image into an HDR (PQ or HLG) space. Furthermore, if you want to correct the color components of images in different color spaces, you can convert them into a common color space using, for example, ITU-R Recommendation BT.2087, and then calculate the correction amount.

[0037] In S308, the CPU 103 adds the gain map generated in S307 to the SDR image developed in S306 and saves the result as a file. By adding the gain map to the file in this way, the gain map can be applied to the SDR image when the file is played back, and the SDR image can be converted into an HDR image for display.

[0038] In S309, the CPU 103 determines whether the next shooting instruction has been given using information from the operation unit 109, etc., and if the next shooting is not to be performed, ends the processing, and if the next shooting is to be performed, returns to S301 and repeats the above-mentioned processing.

[0039] Next, a case where it is determined in S304 that there is a strong relationship between the image captured in the past and the current image will be described.

[0040] If it is determined in S304 that there is a strong relationship between the image captured in the past (reference image) and the current image, in S305 the image processing unit 105 uses the information used in the determination in S304 to adjust the brightness of the RAW image captured in S301 and the HDR image developed in S303 to the brightness of the RAW image and HDR image captured in the past. Note that, depending on how the SDR image is prepared in S306 (described later), control may be performed to adjust the brightness of only one of the RAW image and the HDR image.

[0041] In S306, the image processing unit 105 performs SDR development on the image whose brightness has been adjusted to that of the previously captured image in S305, generating an SDR image. Note that the image whose brightness has been adjusted to that of the previously captured image may be either a RAW image or an HDR image, and therefore when creating an SDR image from an HDR image, the SDR image may be generated using any conversion such as local contrast processing or gamut mapping. Also, when generating an SDR image from a RAW image, the image may be SDR developed in the same way as an HDR image, and then the brightness may be adjusted to that of the previously captured SDR image.

[0042] Furthermore, by performing the subject recognition used in the determination in S304 on an image other than the image developed in S303, for example, an image that has been linearly converted, the processing of S303 can be performed after S305. In addition, in both S303 and S306, the RAW image may be developed in S305 after adjusting the brightness with that of the previous image.

[0043] In S307, the image processing unit 105 generates a gain map for converting the SDR image into an HDR image, using the HDR image and the SDR image whose brightness has been adjusted to that of the image captured in the past by the processes of S303 to S306. The processing from S308 onwards is the same as the processing described above, and therefore the description will be omitted.

[0044] Note that, although the above-described S309 is described on the assumption that an SDR image is converted into an HDR image, the present invention is not limited to this. An alternative configuration is to generate a gain map for converting an HDR image into an SDR image in S307, and add the gain map to the HDR image in S308 and save it as a file.

[0045] Next, a specific example of the processing performed in S305 and S307 when it is determined in S304 that there is a strong relationship between the image captured in the past and the current image will be described with reference to FIG. Figure 2 is a conceptual diagram showing the image at the time of gain map generation and the luminance components of the gain map, with high brightness representing the high brightness side and the side with a large amount of gain map correction, and low brightness representing the low brightness side and the side with a small amount of gain map correction. Also, the images captured consecutively are lined up horizontally from left to right.

[0046] 2(a) shows RAW images 201 to 203 that were captured continuously. It is assumed that there are a mixture of images in which a high-brightness subject 200 appears and images in which it does not appear, or that there are slight differences in brightness even for the same subject, as in RAW images 201 and 202.

[0047] 2(b) shows an HDR image 211 obtained by developing the RAW image 201 in FIG. 2(a), an HDR image 212 obtained by developing the RAW image 202, and an HDR image 213 obtained by developing the RAW image 203. Therefore, if the RAW image 202 does not contain a high-brightness subject 200, the maximum brightness within the image will be different between the HDR image 212 and the HDR images 211 and 213. Furthermore, the difference in brightness between the RAW image 202 and the RAW images 201 and 203 also appears between the HDR image 212 and the HDR images 211 and 213. Furthermore, when creating an image that emphasizes brightness, the difference between the HDR image 212 and the HDR images 211 and 213 may be greater than the difference between the RAW image 202 and the RAW images 201 and 203.

[0048] Figure 2(c) shows an SDR image 221 obtained by developing the RAW image 201 in Figure 2(a), an SDR image 222 obtained by developing the RAW image 202, and an SDR image 223 obtained by developing the RAW image 203. Basically, they are the same as the HDR image in Figure 2(b), but because the image is created by squeezing high-brightness areas into the narrow range of SDR, brightness differences are often less likely to occur than in HDR images.

[0049] 2(d) shows a gain map 231 generated using an HDR image 211 and an SDR image 221, a gain map 232 generated using an HDR image 212 and an SDR image 222, and a gain map 233 generated using an HDR image 213 and an SDR image 223. Note that, as an example, these gain maps are generated assuming conversion from an SDR image to an HDR image.

[0050] In this way, if gain maps are generated as they are without adjusting the brightness of the HDR image and the SDR image, even if the same subject is photographed, there is a possibility that differences in the amount of correction will occur, as between gain map 232 and gain maps 231 and 233. Then, if gain maps 231 to 233 are used to convert SDR images 221 to 223 into HDR images, differences in brightness that were not noticeable in SDR images 221 to 223 may become noticeable, as between HDR image 212 and HDR images 211 and 213.

[0051] 3, in this embodiment, the brightness of the HDR image 212 is corrected to be like that of the image 214, thereby matching the brightness of the same subject as that of the HDR image 211. Furthermore, the brightness of the SDR image 202 is corrected to be like that of the image 204, thereby matching the brightness of the same subject as that of the RAW image 201. Then, by SDR developing the image 204, the brightness of the SDR image 222 and the brightness of the SDR images 221 and 223 can be matched. Then, by generating a gain map 234 using the SDR image 222 and HDR image 214 whose brightnesses have been matched, the gains can be matched.

[0052] Note that, since the purpose of this embodiment is to suppress brightness differences in an image after conversion using a gain map, the SDR image to which the gain map is added may be either an image before or after brightness adjustment. When the gain map is added to an SDR image after brightness adjustment, the brightness after application of the gain map is also uniform, and when the gain map is added to an SDR image before brightness adjustment, although brightness differences before application of the gain map remain, it is possible to suppress the enhancement of brightness differences due to application of the gain map.

[0053] Although the luminance component has been described here for ease of understanding, color components may also be corrected by applying similar processing to each RGB channel. Furthermore, regardless of whether color components are corrected, the generated gain map may be either one-channel or multi-channel.

[0054] Next, an example of how to match the brightness of the subject between images will be described with reference to FIG. FIG. 4 is a diagram showing the correspondence between the HDR image of FIG. 2(b) and the gamma characteristics used during development, and gamma characteristics 401 represent the characteristics of HDR gamma used during HDR development.

[0055] FIG. 4(a) shows, by points 412 to 415, which input / output characteristics of the gamma characteristic 401 each of the objects 402 to 405 in the HDR image 211 corresponds to. Similarly, Figure 4(b) shows which input / output characteristic of the gamma characteristic 401 the object 420 in the HDR image 212 corresponds to by point 430, and which input / output characteristic of the gamma characteristic the object 421 in the HDR image 214 after brightness correction corresponds to by point 431. Note that the objects 403, 420, and 421 refer to the same object. In other words, by correcting point 430 so that it becomes point 431, it is possible to make it have the same brightness as point 413.

[0056] Although the gain can be calculated as is, it is possible to calculate the correction amount for both the HDR image and the RAW image by, for example, converting points 413 and 430 back into linear signals based on the gamma characteristics and calculating the luminance ratio of the same subject in linear space. By using this method, even if subject 200 in image 201 in Figure 2 appears in HDR image 212, the correction amount can be calculated based on the luminance ratio of other subjects.

[0057] As described above, according to the first embodiment, by matching the brightness of the images and then generating a gain map, it is possible to match the correction amount of the gain map between images and suppress the brightness difference of the converted images.

[0058] <Second embodiment> Next, a second embodiment of the present invention will be described, in which omission of processing during continuous shooting will be explained. Figure 5 shows the sequence for continuous shooting, with the horizontal axis representing time. Here, we assume that there is only one development circuit, and that HDR development and SDR development cannot be performed simultaneously.

[0059] 5A shows an example of a case where development processes are performed in order without any blockages. A series of processes including RAW shooting 502, HDR development 503, SDR development 504, and gain map generation 505, and a series of processes including RAW shooting 506, HDR development 507, SDR development 508, and gain map generation 509 are performed in order without any blockages.

[0060] 5B shows an example of a case where the speed of continuous shooting increases and development processing cannot keep up. If the interval between RAW shooting 502 and 506 is shortened to period 510 to bring the series of processes forward, SDR development 504 of the first image and HDR development 507 of the second image will overlap during period 511, making this impossible to achieve.

[0061] Figure 5(c) is an example of a case where some of the processes in Figure 5(b) are delayed. To achieve development processing, it is necessary to delay processing from HDR development 507 for the second image onwards, so the timing of gain map generation 509 remains the same as in Figure 5(a). In other words, the speed of continuous shooting can be increased, but the timing at which development processing ends remains the same, so unprocessed images gradually accumulate, and if continuous shooting continues for a long time, the buffer memory may become full and it may become impossible to shoot.

[0062] A method for solving this problem during continuous shooting will be described with reference to Fig. 6. Fig. 6 is a flowchart for the case where the development process for continuous shooting is omitted. Note that S301 to S304 and S306 are the same processes as those described in Fig. 3, and therefore descriptions thereof will be omitted where appropriate.

[0063] In S304, if it is determined that there is a strong relationship between the previously captured image and the current image, the process proceeds to S603; if there are no previously captured images or if it is determined that there is a weak relationship between the previously captured image and the current image, the process proceeds to S306.

[0064] First, a case where it is determined that there are no images captured in the past or that the relationship between the images captured in the past and the current image is weak will be described.

[0065] In S306, the image processing unit 105 develops the RAW image acquired in S301 using an SDR gamma (second gamma). Next, in S602, the image processing unit 105 creates a conversion table between HDR signals and SDR signals using the HDR image developed in S303 and the SDR image developed in S306, and stores the table in the primary storage device 104 so that it can be used in subsequent image capture. Here, the conversion table is, for example, a one-dimensional lookup table that expresses conversion characteristics in which an HDR signal is input and an SDR signal is output. Note that if the characteristics of the gamma curve used during development are known, the conversion table may be created from the characteristics of the gamma curve.

[0066] In S604, the image processing unit 105 generates a gain map for converting the HDR image into an SDR image using the HDR image developed in S303 and the SDR image developed in S306, and then the process proceeds to S308.

[0067] Next, a case where it is determined in S304 that there is a strong relationship between the image captured in the past and the current image will be described.

[0068] If it is determined in S304 that there is a strong relationship between the previously captured image and the current image, then in S603 the image processing unit 105 converts the HDR image developed in S303 into a signal equivalent to an SDR image based on the conversion table created in S602. At this time, by absorbing the difference in brightness between the previous and current HDR images based on the subject recognition results and image difference information between the images when the relationship was determined in S304, and then referring to the conversion table, it is possible to make the amount of gain map correction the same.

[0069] Note that a configuration may be adopted in which an image captured in the past that was SDR-developed in S602 is saved, and a conversion table between HDR signals and SDR signals is created at the timing of S603. Alternatively, a configuration may be adopted in which a conversion table is not created in S602 for a previous capture, and reduced RAW images are SDR-developed, thereby reducing the processing load in S306.

[0070] In S604, the image processing unit 105 generates a gain map using the HDR image developed in S303 and a signal equivalent to the SDR image converted in S603. When generating a gain map for converting an HDR image into an SDR image, the gain map may be generated directly based on the conversion table characteristics, rather than creating a signal equivalent to the SDR image in S603.

[0071] FIG. 5(d) shows an example in which the processing shown in FIG. 6 is carried out. The processes from 501 to 507 are the same as in Fig. 5(c), but the conversion table can be used to omit or simplify the SDR development 508 process, which makes it possible to bring forward the gain map generation 509. Furthermore, if the subsequent development processes and gain map generation processes can be arranged without overlap, such as RAW shooting 512, HDR development 513, and gain map generation 514, the speed of continuous shooting can be increased without unprocessed image data accumulating in the buffer memory.

[0072] As described above, according to the second embodiment, the image development processing can be partially omitted or simplified, thereby making it possible to more quickly align the correction amounts of the gain maps between images.

[0073] 2(d) has been described as being aimed at generating the gain map 234, but it is also possible to generate a gain map 232 that does not take into account the relationship between images and add both gain maps to the images. This allows for different uses depending on whether emphasis is placed on luminance continuity or on the optimal luminance for each image, for example, by using gain map 234 when used in a slide show or frame-by-frame playback of images, and gain map 232 when used with just one image.

[0074] Furthermore, by adjusting the gain map correction amount of the current image to that of an image captured in the past, if the image signal value after conversion by applying gain map 234 reaches its maximum value, the gain map 232 can be used as is to prevent image gradation from being lost.

[0075] Furthermore, if continuously captured images are regarded as consecutive frames of a video, the above-described processing can also be applied to the video. Since a video has a large number of images and is generally played back to view consecutive images, it is possible to add different gain maps for still images and video, for example, by adding only gain map 234, in order to reduce the file size.

[0076] Furthermore, a gain map for converting an HDR image to an SDR image can also be generated in the same way. When converting an HDR image to an SDR image, the gain map 234 can be added to the HDR image and the correction amount can be inverted when applied, or an inverted correction amount can be added.

[0077] <Modification> By replacing the continuously captured images with two adjacent images, the processing described in the first embodiment can be applied to processing a stereo image consisting of a left eye image and a right eye image, as shown in Figure 7.

[0078] 3, since there are always images with a strong correlation between the left eye image and the right eye image, the determination in S304 is YES, and processing to make the brightness of the left eye image and the right eye image uniform is performed in S305. Also, the continuous shooting determination in S309 may be omitted.

[0079] However, since parallax occurs between the images in stereo images, more accurate processing can be achieved by absorbing the parallax by aligning the images during object recognition in S304 and image correction in S305. Regarding the presence or absence of an object between images due to parallax, it is sufficient to apply the processing for an object that is only present in one of the images, as explained in Figures 2 and 4.

[0080] 7(a) shows the concept of an imaging device 701 having two imaging units, each including a lens 711 and an image sensor 712. When a scene 700 is captured by the imaging device 701, a left-eye image 721 and a right-eye image 722 having parallax are obtained. In this case, processes that can be performed simultaneously for both eyes, such as image capture in S301, development in S303 and S306, and gain map generation in S307, may be performed simultaneously.

[0081] 7(b) shows the concept of an image capturing device 702 having two lenses 711 and one image sensor 732. When a scene 700 is captured by the image capturing device 702, a single image 740 contains a left-eye image 741 and a right-eye image 742. Therefore, for the image correction in S305, the left-eye image 741 and the right-eye image 742 must be separated and then processed separately.

[0082] The stereo image shown in FIG. 7 can also be processed in the same manner as described with reference to FIG. 6, but the description thereof will be omitted here.

[0083] <Third embodiment> Next, a third embodiment of the present invention will be described. In the third embodiment, a process of generating gain maps and then matching brightness between the gain maps will be described.

[0084] FIG. 8 is a conceptual diagram of the process of matching brightness between gain maps, showing an example of adjusting the gain map by applying a gain.

[0085] In S801, the image processing unit 105 prepares images for generating a gain map. The images can be prepared by utilizing the processes of S301 to S303 and S306 in FIG. 3. Note that in the following processes, the explanation will be continued assuming that multiple image sets of HDR images and SDR images have been acquired.

[0086] In S802, the image processing unit 105 generates a gain map using the image set. The method for generating the gain map may be the method described with reference to FIG. 2. Note that while FIG. 8 shows gain maps for multiple image sets being generated simultaneously, a separate gain map may be generated for each image captured during continuous shooting or video capture. In the case of stereo images, the area with the highest brightness among multiple images may be detected, and a gain map between the left-eye image and the right-eye image may be generated based on this maximum brightness. By doing this, the difference in the amount of correction of the gain maps between images can be reduced to some extent at this stage.

[0087] In S803, the image processing unit 105 classifies each area between the images into a common area and a difference area by performing object recognition. At this time, in the case of stereo images, the images may be aligned in consideration of parallax before detection.

[0088] In S804, the image processing unit 105 calculates a correction amount for uniforming the brightness of the gain map calculated in S802, using information about the common areas classified in S803. The correction amount may be calculated from any one common area as shown in the figure, or may be calculated from multiple common areas as shown in 806. Furthermore, a single correction amount may be obtained by applying a weighted average process or the like to the correction amounts calculated from multiple common areas. Also, while FIG. 8 illustrates a process in which a gain map is input and the correction amount is calculated from the gain map, it is also possible to input an HDR image and calculate the correction amount from the HDR image.

[0089] In S805, the correction amount calculated in S804 is applied to the gain map generated in S802, thereby matching the brightness of the gain maps between the images.

[0090] As described above, according to the third embodiment, by correcting the gain map by multiplying it with a gain, it is possible to make the correction amount of the gain map between images uniform and suppress the brightness difference of the converted image.

[0091] <Fourth embodiment> Next, a fourth embodiment of the present invention will be described. In the fourth embodiment, a process of generating a gain map and then replacing and adjusting the correction amount of the gain map will be described.

[0092] Fig. 9 is a conceptual diagram of the process of matching brightness between gain maps, showing an example of adjusting by replacing the correction amount of the gain map. Note that S801 to S804 are the same as in Fig. 8, so the explanation will be omitted. Note that the same idea as S806 in Fig. 8 can also be adopted in Fig. 9.

[0093] In S901, the image processing unit 105 classifies the gain map generated in S802 into a common region gain map and a differential region gain map using the information on the common region and differential region classified in S803.

[0094] In S902, the image processing unit 105 standardizes the common area gain maps separated in S901 by replacing the correction amount of one gain map with the correction amount of the other gain map. Furthermore, the correction amount calculated in S804 is applied to the differential area gain map separated in S901, thereby performing correction while maintaining the relative brightness relationship with the common area. During continuous shooting or video recording, the correction amount of the current image is replaced with the correction amount of the previous image, allowing images to be processed in order starting from the previous image.

[0095] In S903, the image processing unit 105 combines the common region gain map corrected in S902 and the differential region gain map to generate one gain map.

[0096] As described above, according to the fourth embodiment, by replacing the correction amounts of the gain maps, it is possible to make the correction amounts of the gain maps between images uniform, and also to suppress the luminance difference between the converted images.

[0097] Up to this point, the explanation has been given on the assumption that the gain map is a linear signal, but it may also be subjected to gamma or the like. When the gain map is subjected to gamma, the correction amount may be calculated as is, as with a linear signal, or the correction amount may be calculated after converting to linear space as in the first embodiment. Note that when calculating the correction amount from an SDR image and an HDR image, it can also be calculated in the same manner as in the first embodiment.

[0098] In the third and fourth embodiments, several application examples to continuous shooting, video, and stereo images have been described, but the contents described in the first embodiment can also be applied to the third and fourth embodiments in the same way.

[0099] Furthermore, in the above-described embodiment, the present invention has been described as being implemented in the camera 100, but the present invention is not limited to this, and the image used to generate the gain map may be acquired from outside. In this case, for example, an image processing device realized by an information processing device or the like may have an input unit, and may input and process an image captured by an external imaging device.

[0100] Furthermore, when there are multiple images that are highly related to each other, one of them may be selected as a reference image regardless of the order in which the images were obtained, and processing may be performed according to the brightness of the selected reference image.

[0101] Although several patterns have been described above as preferred embodiments of the present invention, the present invention is not limited to these embodiments and various modifications and changes are possible within the scope of the gist of the present invention. For example, by using both HDR and SDR images as images with detailed image creation, correction can be performed that changes the image creation as well as the gamma conversion before and after correction.

[0102] <Other embodiments> The present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device.

[0103] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0104] <Summary> The disclosure of this embodiment includes the following configuration.

[0105] (Item 1) a conversion means for converting an input image into a first image having a first dynamic range and a second image having a second dynamic range; a correction means for correcting the plurality of first images having a predetermined relationship with each other so that the brightnesses of the first images are uniform; a generating means for generating, for each of the corrected first images, a gain map for converting the second image corresponding to the first image into an image having the first dynamic range, using the second image corresponding to the first image; 1. An image processing device comprising: (Item 2) the correction means further corrects brightness of the second images corresponding to the first images having the predetermined relationship with each other so as to be uniform; The generating means generates the gain map using the corrected second images. 2. The image processing device according to item 1, (Item 3) 3. The image processing device according to item 1 or 2, wherein the conversion means converts the input image into the second image. (Item 4) 3. The image processing device according to item 1 or 2, wherein the conversion means converts the first image into the second image. (Item 5) further comprising a creating means for creating a conversion table for converting the first dynamic range into the second dynamic range; 3. The image processing device according to item 1 or 2, wherein the conversion means converts each of the corrected first images into the second image using the conversion table. (Item 6) 6. The image processing device according to item 5, wherein the creation means creates the conversion table using the first image converted by the conversion means by developing an input image using a first gamma and the second image converted by the conversion means by developing using a second gamma. (Item 7) the conversion means converts an input image into the first image by developing it using a first gamma, and converts it into the second image by developing it using a second gamma; 6. The image processing device according to item 5, wherein the creating means creates the conversion table based on the conversion characteristics of the first gamma and the second gamma. (Item 8) 8. The image processing device according to any one of items 1 to 7, further comprising a recording unit that adds the gain map generated by the generating unit to the corresponding second image and records the result. (Item 9) the generating means generates, for each of the plurality of first images before the correction, a second gain map for converting the second image into an image having the first dynamic range, using the second image corresponding to the first image; 9. The image processing device according to item 8, wherein the recording means further adds the second gain map to the corresponding second image and records the second gain map. (Item 10) 10. The image processing device according to any one of items 1 to 9, further comprising a determination means for determining the predetermined relationship using at least one of subject recognition for the first image, the positional relationship of the input images, operation and exposure when the input images were taken, and information on a device that took the input images. (Item 11) a conversion means for converting an input image into a first image having a first dynamic range and a second image having a second dynamic range; a generating means for generating, for each input image, a gain map using the first image and the second image to convert the second image into an image having the first dynamic range; a detection means for detecting a common area in the plurality of first images having a predetermined relationship with each other; a calculation means for calculating a ratio of brightness of the common region of one of the plurality of first images to one first image used as a reference, or a ratio of gain of a region corresponding to the common region of a second gain map corresponding to the other first image to a first gain map corresponding to the reference first image; a correction means for correcting the second gain map using the ratio; 1. An image processing device comprising: (Item 12) the generating means further detects a difference region that differs between the reference first image and the other first images among the plurality of first images; The correction means replaces the gain of a region of the second gain map corresponding to the common region with the gain of the first gain map, and corrects the gain of a region corresponding to the difference region using the ratio. Item 12. The image processing device according to item 11. (Item 13) 13. The image processing device according to item 11 or 12, further comprising a recording unit that adds the first gain map or the second gain map corrected by the correction unit to the corresponding second image and records the result. (Item 14) Item 14. The image processing device according to item 13, wherein the recording means further adds the gain map generated by the generating means to the corresponding second image and records the image. (Item 15) 15. The image processing device according to any one of items 11 to 14, further comprising a determination means for determining the predetermined relationship using at least one of subject recognition for the first image, the positional relationship of the input images, operation and exposure when the input images were taken, and information on a device that took the input images. (Item 16) an imaging means for capturing an image; An image processing device according to any one of items 1 to 15; An electronic device comprising: (Item 17) a first conversion step of converting an input image into a first image having a first dynamic range; a correcting step of correcting brightness of the plurality of first images having a predetermined relationship with each other so as to be uniform; a second conversion step of converting the input image or the corrected first image into a second image having a second dynamic range; generating a gain map for each of the corrected first images using the second image corresponding to the first image to convert the second image into an image having the first dynamic range; An image processing method comprising: (Item 18) a conversion step of converting an input image into a first image having a first dynamic range and a second image having a second dynamic range; a generating step of generating, for each input image, a gain map for converting the second image into an image having the first dynamic range, using the first image and the second image; a detection step of detecting a common area in the plurality of first images having a predetermined relationship with each other; a calculation step of calculating a ratio of brightness of the common region of one first image serving as a reference among the plurality of first images to the other first images, or a ratio of gain of a region corresponding to the common region in a second gain map corresponding to the other first image to a first gain map corresponding to the reference first image; a correction step of correcting the second gain map using the ratio; An image processing method comprising: (Item 19) A program for causing a computer to function as each means of the image processing device according to any one of items 1 to 15. (Item 20) Item 19. A computer-readable storage medium storing the program described in item 19.

[0106] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0107] 100...imaging device, 101...optical system, 102...imaging element, 103...CPU, 104...primary storage device, 105...image processing unit, 106...recording medium, 107...secondary storage device, 108...display unit, 109...operation unit

Claims

1. a conversion means for converting an input image into a first image having a first dynamic range and a second image having a second dynamic range; a correction unit that corrects the plurality of first images having a predetermined relationship with each other so that the brightnesses of the first images are uniform; a generating means for generating, for each of the corrected first images, a gain map for converting the second image into an image having the first dynamic range by using the second image corresponding to the first image; 1. An image processing device comprising:

2. The correction means further corrects brightness of the plurality of second images corresponding to the plurality of first images having the predetermined relationship with each other so as to be uniform; The generating means generates the gain map using the corrected second images.

2. The image processing device according to claim 1, wherein:

3. 2. The image processing apparatus according to claim 1, wherein said conversion means converts said input image into said second image.

4. 2. The image processing apparatus according to claim 1, wherein said conversion means converts said first image into said second image.

5. further comprising a creating means for creating a conversion table for converting the first dynamic range into the second dynamic range; The conversion means converts each of the corrected first images into the second image using the conversion table.

2. The image processing device according to claim 1, wherein:

6. 6. The image processing device according to claim 5, wherein the creation means creates the conversion table using the first image converted by the conversion means by developing an input image using a first gamma and the second image converted by the conversion means by developing it using a second gamma.

7. the conversion means converts an input image into the first image by developing it using a first gamma, and converts it into the second image by developing it using a second gamma; The creating means creates the conversion table based on the conversion characteristics of the first gamma and the second gamma.

6. The image processing device according to claim 5,

8. 2. The image processing apparatus according to claim 1, further comprising a recording unit that adds the gain map generated by the generating unit to the corresponding second image and records the result.

9. the generating means generates, for each of the plurality of first images before the correction, a second gain map for converting the second image into an image having the first dynamic range, using the second image corresponding to the first image; The recording means further adds the second gain map to the corresponding second image and records the second image.

9. The image processing device according to claim 8,

10. 2. The image processing device according to claim 1, further comprising a determination means for determining the predetermined relationship using at least one of subject recognition for the first image, the positional relationship of the input images, operation and exposure when the input images were taken, and information on a device that took the input images.

11. a conversion means for converting an input image into a first image having a first dynamic range and a second image having a second dynamic range; a generating means for generating, for each input image, a gain map for converting the second image into an image having the first dynamic range, using the first image and the second image; a detection means for detecting a common area in the plurality of first images having a predetermined relationship with each other; a calculation means for calculating a ratio of brightness of the common region of one of the plurality of first images to one first image used as a reference, or a ratio of gain of a region corresponding to the common region in a second gain map corresponding to the other first image to a first gain map corresponding to the reference first image; a correction means for correcting the second gain map using the ratio; 1. An image processing device comprising:

12. the generating means further detects a difference region that differs between the reference first image and the other first images among the plurality of first images; The correction means replaces the gain of a region of the second gain map corresponding to the common region with the gain of the first gain map, and corrects the gain of a region corresponding to the difference region using the ratio.

12. The image processing device according to claim 11.

13. 12. The image processing apparatus according to claim 11, further comprising a recording unit that adds the first gain map or the second gain map corrected by the correction unit to the corresponding second image and records the map.

14. 14. The image processing apparatus according to claim 13, wherein the recording means further adds the gain map generated by the generating means to the corresponding second image and records the image.

15. 12. The image processing device according to claim 11, further comprising a determination means for determining the predetermined relationship using at least one of subject recognition for the first image, the positional relationship of the input images, operation and exposure when the input images were taken, and information on a device that took the input images.

16. An imaging means for capturing an image; The image processing device according to any one of claims 1 to 15, An electronic device comprising:

17. a first conversion step of converting an input image into a first image having a first dynamic range; a correcting step of correcting brightness of the plurality of first images having a predetermined relationship with each other so as to be uniform; a second conversion step of converting the input image or the corrected first image into a second image having a second dynamic range; generating a gain map for each of the corrected first images using the second image corresponding to the first image to convert the second image into an image having the first dynamic range; An image processing method comprising:

18. a conversion step of converting an input image into a first image having a first dynamic range and a second image having a second dynamic range; a generating step of generating, for each input image, a gain map for converting the second image into an image having the first dynamic range, using the first image and the second image; a detection step of detecting a common area in the plurality of first images having a predetermined relationship with each other; a calculation step of calculating a ratio of brightness of the common region of one of the plurality of first images to one first image used as a reference, or a ratio of gain of a region corresponding to the common region in a second gain map corresponding to the other first image to a first gain map corresponding to the reference first image; a correction step of correcting the second gain map using the ratio; An image processing method comprising:

19. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 15.

20. A computer-readable storage medium storing the program according to claim 19.

Images