Image processing system and image processing method

Abstract

To provide an image processing system and an image processing method capable of generating a gain map in consideration of a photographic scene.SOLUTION: An image processing system generates a first gain map from SDR image data and HDR image data from the same image. The first gain map is data containing a plurality of two-dimensionally arranged gain values in which application of one of SDR image data and HDR image data enables the generation of the other. According to a photographic scene determined from the SDR image data or the HDR image data, the image processing system generates a second gain map to be used in combination with the first gain map.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method, and more particularly to a dynamic range conversion technique for an image. [Background technology]

[0002] For example, the ITU-R BT.2100 standard is known, which relates to images with a wider luminance dynamic range (HDR) than the luminance dynamic range (SDR) conforming to ITU-R BT.709. Also, a gain map has been proposed for dynamically generating an image with an appropriate dynamic range from an SDR image or an HDR image according to the capabilities of a display device, etc. (Non-Patent Document 1). The gain map is a map of coefficients (gains) to be applied to each pixel that constitutes an image. The resolution of the gain map may be equal to or less than the resolution of the image to which the gain map is applied. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Eric Chan, “Gain Maps” Version 1.0 draft 15”, [online], February 28, 2024, [Searched on May 15, 2020], Internet <URL:https: / / helpx.adobe.com / content / dam / help / en / camera-raw / using / gain-map / jcr_content / root / content / flex / items / position / position-par / table / row-3u03dx0-column-4a63daf / download_section / download-1 / Gain_Map_1_0d15.pdf> Summary of the Invention [Problem to be solved by the invention]

[0004] In order to display images with the intended contrast on various display devices, it is considered desirable to prepare a gain map that takes into account not only the characteristics of the display device but also the shooting scene. However, Non-Patent Document 1 does not mention a gain map that takes the shooting scene into consideration.

[0005] In consideration of the above problems with the conventional techniques, the present invention provides, in one embodiment, an image processing device and an image processing method capable of generating a gain map that takes into account the photographed scene. [Means for solving the problem]

[0006] In one aspect, the present invention provides an image processing device comprising: an acquisition means for acquiring SDR image data and HDR image data based on the same image; a first generation means for generating a first gain map in which a plurality of gain values ​​are arranged two-dimensionally, the first gain map being capable of generating the other of SDR image data and HDR image data by applying the first gain map to base image data, which is one of the SDR image data and the HDR image data; a determination means for determining a photographed scene based on the SDR image data or the HDR image data; and a second generation means for generating a second gain map to be used in combination with the first gain map in accordance with the photographed scene determined by the determination means. [Effects of the Invention]

[0007] According to the present invention, an image processing device and an image processing method are provided that are capable of generating a gain map that takes into account a photographed scene. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an imaging apparatus as an image processing apparatus according to an embodiment of the present invention. [Figure 2] 1 is a flowchart illustrating the operation of an image processing unit in the first embodiment. [Figure 3] 1 is a flowchart illustrating the operation of an image processing unit in the first embodiment. [Figure 4]Diagram of gain map generation for the device [Figure 5] Diagram of gain map generation for a captured scene [Figure 6] FIG. 10 is a diagram illustrating the operation of the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] In the following, the present invention will be described as being embodied in an imaging device (digital camera) as an example of an image processing device. However, imaging functionality is not essential to the present invention, and the present invention can be implemented in any electronic device having one or more arithmetic circuits or processors. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, smart watches, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0011] ●(First embodiment) 1 is a block diagram showing an example of the functional configuration of an imaging device as an image processing device according to a first embodiment of the present invention. The imaging device 100 includes an optical system 101, an image sensor 102, a central processing unit (CPU) 103, a primary storage device 104, a photometric sensor 105, an external communication unit 106, an image processing unit 107, a recording medium 108, a secondary storage device 109, a display unit 110, and an operation unit 111.

[0012] FIG. 1 is a block diagram showing an example of the functional configuration of an imaging device 100 capable of implementing the present invention. Each functional block of the imaging device 100 can be implemented by software or a combination of software and hardware, except for parts that can clearly be realized only by hardware (e.g., lenses included in the optical system 101, pixels of the image sensor 102, etc.). For example, a functional block may be realized by dedicated hardware such as an ASIC. Alternatively, a functional block may be realized by a processor such as a CPU executing a program stored in memory. Note that multiple functional blocks may be realized by a common configuration (e.g., a single ASIC). Furthermore, hardware that realizes part of the functions of one functional block may be included in hardware that realizes another functional block.

[0013] In this embodiment, the image processing unit 107 generates an SDR image, an HDR image, and a gain map based on a RAW image obtained by capturing an image using the image sensor 102. The gain map is data that can be applied to a base rendition (either an HDR image or an SDR image) to generate the other rendition. Furthermore, by applying a weight between 0 and 1 to the gain map during display, an adaptive HDR rendition can be realized.

[0014] For example, an image data file is generated with a gain map attached to an SDR image as a base representation. When displaying, the image to be displayed can be generated by applying appropriate weights and gain maps to the SDR image according to the capabilities of the display device. If the display device does not support HDR, the gain map is ignored (the weights are set to 0) and the SDR image is displayed.

[0015] In this embodiment, two signal characteristics are assumed to represent the relationship between the video signal level and display brightness in an HDR image: PQ (Perceptual Quantization) and HLG (Hybrid Log Gamma). PQ is specified as an EOTF (Electro-Optical Transfer Function) in SMPTE ST 2084, and HLG is specified as an OETF (Optical to Electrical Transfer Function) in ARIB STD-B67.

[0016] An SDR image is an image with a narrower output range (dynamic range) than an HDR image. In the following explanation, the gamma of an HDR image is assumed to be the OETF characteristic (PQ Inverse EOTF) based on SMPTE ST 2084, and the color gamut is assumed to conform to the ITU-R BT. 2020 standard. The gamma of an SDR image is assumed to be sRGB gamma, and the color gamut is assumed to be sRGB.

[0017] The optical system 101 includes a lens, a shutter, an aperture, and a mechanism (such as a motor) for driving these. The optical system 101 may be integrated with the imaging device 100, or may have the form of an interchangeable lens. The optical system 101 forms an optical image of a subject on the imaging surface of the imaging element 102.

[0018] The image sensor 102 may be, for example, a known CCD or CMOS color image sensor with a primary-color Bayer array color filter. The image sensor 102 has a pixel array in which multiple pixels are arranged two-dimensionally, and peripheral circuits for reading out signals from each pixel. Each pixel accumulates charge according to the amount of incident light through photoelectric conversion. By reading out signals from each pixel having voltages according to the amount of charge accumulated during the exposure period, a group of pixel signals (analog image signals) representing the optical image formed on the imaging surface is obtained.

[0019] The analog image signal is A / D converted to a digital image signal (image data) and then stored in the primary storage device 104. The A / D conversion may be performed by the image sensor 102, or may be performed by another component (for example, an A / D converter not shown) outside the image sensor 102. Image data stored in the primary storage device 104 at this stage, which is composed of pixel data having one color component of R, G, or B, is called RAW image data.

[0020] The CPU 103 determines the imaging conditions (shutter speed or exposure time, aperture value, and imaging sensitivity).

[0021] An optical image formed by the optical system 101 is projected onto the photometric sensor 105. The photometric sensor 105 measures the luminance information of the optical image for each of a plurality of photometric regions (for example, 12 horizontally and 8 vertically, for a total of 96 regions). The photometric sensor 105 outputs the measurement results to the CPU 103. Note that the number, positions, and sizes of the photometric regions may change depending on the settings of the imaging device 100, etc.

[0022] The CPU 103 loads a program stored in the secondary storage device 109 into the primary storage device 104 and executes it, thereby controlling the operation of each unit of the imaging device 100 and realizing various functions of the imaging device 100. Note that at least some of the functions described as being realized by the CPU 103 executing a program may be executed by hardware such as an ASIC or FPGA configured to execute the function.

[0023] The primary storage device 104 is a volatile storage device such as a RAM, and is used to temporarily store various data such as programs executed by the CPU 103, data required for executing the programs, data before, during, and after processing by the image processing unit 107, image data for display, and image data for recording.

[0024] The secondary storage device 109 is a non-volatile storage device such as an EEPROM, and stores programs executed by the CPU 103 to control the image capture device 100, various settings, information about the image capture device 100, GUI data, and the like.

[0025] The recording medium 108 is used as a recording destination for the image data for recording stored in the primary storage device 104. The recording medium 108 is removable from the imaging device 100 and may be, for example, a semiconductor memory card. The image data recorded on the recording medium 108 can be read by other devices such as a personal computer. The imaging device 100 has a mechanism for attaching and detaching the recording medium 108 and a read / write function.

[0026] The image processing unit 107 applies predetermined image processing to image data stored in the primary storage device 104, such as after being read from the image sensor 102 or the recording medium 108, to acquire and / or generate signals, image data, and information according to the intended use. The image processing unit 107 may be a dedicated hardware circuit, such as an ASIC (Application Specific Integrated Circuit) designed to achieve a specific function. Alternatively, the image processing unit 107 may be configured such that a processor, such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit), executes software to achieve a specific function. The image processing unit 107 outputs the acquired or generated signals, information, and data to the CPU 103 or stores them in the primary storage device 104 according to the intended use.

[0027] The image processing applied by the image processing unit 107 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when a color filter is provided on the image sensor 102, and is a process of interpolating the values ​​of color components that are not included in the individual pixel data that make up the image data. Color interpolation processing is also called demosaic processing. The correction processing may include white balance adjustment, tone correction (gamma), correction of image degradation caused by optical aberrations of the optical system 101 (image restoration), correction of the effects of peripheral light falloff of the optical system 101, color correction, and the like. The detection process may include detection of characteristic regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. Data processing can include processes such as area extraction (trimming), compositing, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or recording, the generation of gain maps (described later), and conversion of color gamuts and dynamic ranges. The evaluation value calculation process can include processes such as generating signals and evaluation values ​​used in autofocus (AF) detection, and generating evaluation values ​​used in automatic exposure control (AE). Special effect processing can include adding a blur effect, changing color tones, relighting, and the like. It should be noted that these are examples of processes that the image processing unit 107 can apply, and do not limit the processes that the image processing unit 107 can apply.

[0028] The display unit 110 is, for example, a liquid crystal display provided on the surface of the housing of the imaging device 100. The display unit 110 displays information about the imaging device 100, a menu screen, images captured by the image sensor 102, images read from the recording medium 108, and the like. Note that the imaging device 100 can cause the display unit 110 to function as an electronic viewfinder (EVF) by continuously capturing video and displaying the video on the display unit 110 while in a shooting standby state. The operation of causing the display unit 110 to function as an EVF is called live view display, and the image used for the live view display is called a live view image.

[0029] The operation unit 111 is a collective term for input devices (such as buttons, switches, and dials) provided for the user to input various instructions to the imaging device 100. The input devices constituting the operation unit 111 are named according to the assigned functions. For example, the operation unit 111 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, and a confirmation key. The release switch is a switch for recording still images, and the CPU 103 recognizes a half-pressed state of the release switch as an instruction to prepare for shooting and a full-pressed state as an instruction to start shooting. The CPU 103 also recognizes a press of the video recording switch in a shooting standby state as an instruction to start video recording, and a press of the video recording switch during video recording as an instruction to stop recording. Note that the functions assigned to the same input device may be variable. The input device may also be software buttons or keys using a touch display. The operation unit 111 may also include an input device compatible with non-contact input methods such as voice input or eye-gaze input.

[0030] In this embodiment, an operation mode for capturing a specific scene can be set in the imaging device 100. Typical examples of the specific scene include, but are not limited to, a portrait, a sunset, a night view, fireworks, a starry sky, and cooking.

[0031] The external communication unit 106 is a communication interface between an external device and the imaging device 100. The external communication unit 106 supports one or more wired and / or wireless communication standards. The external communication unit 106 has a transmission / reception circuit according to the supported standard, and an associated connector or antenna. Typical examples of standards supported by the external communication unit 106 include, but are not limited to, USB, wireless LAN, Bluetooth (registered trademark), and HDMI (registered trademark).

[0032] The CPU 103 transmits image data to an external device via the external communication unit 106, acquires various information from the external device, controls the operation of the external device, and receives commands from the external device that control the operation of the imaging device 100.

[0033] Next, the gain map generation operation of the image processing unit 107 will be described using the flowchart shown in Fig. 2. Here, it is assumed that a gain map is generated for still image RAW image data captured by the image sensor 102 and stored in the primary storage device 104. However, the same can be applied to the case where a gain map is generated for still image RAW image data read from the recording medium 108 and stored in the primary storage device 104. Furthermore, a gain map can also be generated for RAW video data in the same way as for still image RAW image data by treating each frame as RAW image data.

[0034] The process shown in FIG. 2 can be executed, for example, when a still image is captured or when a user instructs generation of a gain map for specific RAW image data.

[0035] The gain map is recorded in association with the data of the image (hereinafter referred to as the base image) used as the base representation when the gain map was generated. The base image may be either an SDR image or an HDR image, but here it is assumed to be an SDR image. This is to enable the display of an SDR image even if the device displaying the recorded image does not support HDR images and / or gain maps.

[0036] Here, we will handle RAW image data of an image captured by an image sensor having a primary color Bayer array color filter. Therefore, the RAW image data processed by the image processing unit 107 has information on one color component of R (red), G (green), or B (blue) for each pixel. Note that the gain map generation process does not depend on the type of color filter.

[0037] In S201, the image processing unit 107 generates SDR image data from the RAW image data stored in the primary storage device 104 and stores the SDR image data in the primary storage device 104.

[0038] In S202, the image processing unit 107 generates HDR image data from the RAW image data stored in the primary storage device 104 and stores the HDR image data in the primary storage device 104.

[0039] The image processing applied to RAW image data by the image processing unit 107 in S201 and S202 is a process for generating image data in which each pixel contains three primary color components (RGB, YUV, etc.), and is also called development processing. This generally includes gamma processing, white balance adjustment processing, and color interpolation processing. The gamma used in gamma processing differs between generating SDR images and generating HDR images.

[0040] Note that, although SDR image data and HDR image data are generated in S201 and S202 here, pre-recorded SDR image data and HDR image data based on the same image may be acquired from the recording medium 108, for example.

[0041] In S203, the image processing unit 107 matches the color space and gamma (EOTF or OETF) of the SDR image data and the HDR image data. The image processing unit 107 can match the color space and gamma of both sets of image data by converting the color space and gamma of one or both sets of image data using any known method.

[0042] For example, when matching an SDR image with an HDR image, the color gamut conversion method from BT 709 to BT 2020 described in ITU-R BT 2087 can be used. Note that sRGB shares a color gamut with BT 709. The conversion method used in S203 can be changed as appropriate depending on the color gamut and gamma of the SDR image and HDR image. The gamma of the SDR image and HDR image may also be converted to linear (γ=1.0).

[0043] In S204, the image processing unit 107 generates a gain map in which a plurality of gains are arranged two-dimensionally from the SDR image and HDR image in which the color space and gain are combined. The gain map G is defined by the following equation: G=log2((HDR image + khdr ) / (SDR image + k sdr )) k hdr and k sdr is an offset coefficient that is determined so that the value in the parentheses for which the logarithm is calculated is positive. A gain map is generated by calculating the gain for each pixel in the HDR image and SDR image that corresponds to the gain calculated using the above formula.

[0044] The gain map may be generated for each component (channel) such as RGB or YUV, or may be generated for one component such as the luminance component (grayscale). The luminance component may be, for example, the Y component of YUV or the G component of RGB. The image processing unit 107 converts the image data from RGB format to YUV format as necessary.

[0045] The gain that makes up the gain map has a minimum value of 0 and a maximum value of 2. N The gain ranges from -1 to 1, where N is the number of encoding bits. Therefore, when encoding with 8 bits, the maximum gain value is 255. A gain of 0 has no effect on brightness. A negative gain means darkening, and a positive gain means brightening.

[0046] The image processing unit 107 can generate a gain map using, for example, the method described in Non-Patent Document 1. The image processing unit 107 also generates metadata for the gain map. The metadata may be, but is not limited to, the number of gain maps (number of channels), the type of base image, the minimum and maximum gain values, an offset coefficient, the size of the gain map, and information at the time of image capture (such as the capture mode and scene information).

[0047] The gain map generation process in S204 will be described in detail with reference to the flowchart shown in FIG.

[0048] In S301, the image processing unit 107 (first generation means) generates a gain map (gain map for the device) for each display device capability. This is to allow HDR images to be displayed with appropriate contrast on display devices with different capabilities. Here, as an example of different capabilities depending on the display device, multiple gain maps for the device corresponding to different display peak luminance are generated. For example, in S202, the image processing unit 107 generates multiple HDR images with different peak luminances, and generates a gain map for the device using the SDR image and each HDR image.

[0049] Here, an example will be described in which gain maps are generated assuming display devices with display peak luminances of 400 nit and 600 nit. Note that gain maps may be generated for three or more display peak luminances, for example, adding 1000 nit. Also, multiple gain maps may be generated for each model of display device, rather than for each capability value.

[0050] Figure 4 shows examples of gain curves for HDR images generated by S202 with peak luminances of 400 nit and 600 nit, corresponding to an SDR image with a peak luminance of 100 nit. Here, the two gain curves have the same characteristics as the SDR image gain curve up to an input luminance of 18 nit. However, if brightness contrast occurs in an image to which a gain map based on these gain maps has been applied, the gain curve may be adjusted to suppress the brightness contrast. Brightness contrast is a phenomenon in which a color appears darker or brighter than it actually is depending on the brightness of the surrounding colors.

[0051] The image processing unit 107 stores the generated gain maps for the multiple devices in the primary storage device 104 together with the respective metadata.

[0052] In S302, the image processing unit 107 (determination means) determines the photographic scene of the image. The image processing unit 107 can determine the photographic scene by analyzing the SDR image generated in S201 or the HDR image generated in S202 using a known method. Furthermore, the image processing unit 107 may consider an image captured in a shooting mode for capturing a specific scene at the time of shooting as an image capturing the specific scene assumed by the shooting mode. Note that, as will be described in the second embodiment, one frame of an image may correspond to multiple photographic scenes. For example, an image of a person captured against a night scene background corresponds to both a night scene scene and a portrait scene. However, in the first embodiment, it is assumed that only one type of photographic scene is determined for one frame of an image.

[0053] In S303, the image processing unit 107 (second generating means) generates a gain map for the photographic scene determined in S302. The gain map for the photographic scene is a gain map to be used in combination with the gain map for the device generated in S301. Specifically, the gain map for the photographic scene is used to correct the gain map for the device in accordance with the photographic scene. Unlike the gain map for the device, the gain map for the photographic scene is not generated based on the SDR image and the HDR image.

[0054] Specifically, information on the area where the gain should be corrected and information on the amount of correction are associated with each photographic scene and are stored in advance in the secondary storage device 109. Then, the image processing unit 107 reads out from the secondary storage device 109 the information on the area where the gain should be corrected and information on the amount of correction that corresponds to the type of photographic scene determined in S302.

[0055] The image processing unit 107 identifies an area in the SDR image or HDR image where the gain should be corrected based on information about the area where the gain should be corrected. Then, based on information about the correction amount, the image processing unit 107 generates a gain map for correcting the gain in the area where the gain should be corrected from the gain map for the device generated in S301.

[0056] The gain map for the shooting scene is a gain map for the device in which the gain in the area where the gain should be corrected has a significant value (other than 1) according to the correction amount, and is 1 in other areas. The gain map for the shooting scene is intended to be multiplied by the gain map for the device to correct the gain map for the device. Therefore, in areas where the gain value is not corrected, the gain value of the gain map for the scene is set to 1 so that the gain value in the gain map for the device is maintained.

[0057] The region for which the gain should be corrected may be the entire image or a specific region depending on the shooting scene. In the case of a specific region, it can be set from the types of regions that the image processing unit 107 can detect from the image. For example, if the image processing unit 107 can detect faces (people, dogs, cats, etc.), birds, human bodies, and vehicles as characteristic regions, it can set one or more of these types of regions as the region for which the gain should be corrected.

[0058] The correction amount may be a specific target brightness value (absolute value), or may be information indicating a target state, such as "appropriate exposure," "+1 stop," or "increase / decrease in color temperature." When a specific brightness value is set as the correction amount, the image processing unit 107 determines a correction amount that will set the (average) display brightness of the region in the base image where the gain is to be corrected to the specific brightness value. When "appropriate exposure" is set as the correction amount, the image processing unit 107 determines a correction amount that will make the (average) display brightness of the region in the base image where the gain is to be corrected appropriate brightness for the peak brightness of the image. The appropriate brightness for the peak brightness can be determined in advance as a percentage of the peak brightness. When "+1 stop" is set as the correction amount, the image processing unit 107 determines a correction amount that will make the (average) display brightness of the region in the base image where the gain is to be corrected appropriate exposure +1 stop. For color temperature, the image processing unit 107 determines a correction amount that will increase / decrease one or more color components by a predetermined amount.

[0059] In FIG. 5(a), 501 denotes a base image determined to be a night scene in S302. Here, it is assumed that night scene settings are configured to provide appropriate exposure for a person's face region or human body region. FIG. 5(b) is a diagram illustrating the concept of the correction amount. 503 denotes the peak luminance of the base image 501. The peak luminance of a specific image can vary within a range equal to or less than the peak luminance that the base image can achieve. Also, 504 denotes a representative luminance value of a person's face region 502 in the base image 501. 505 denotes a set target luminance value. In this case, the image processing unit 107 generates a scene gain map for correcting the gain of the region corresponding to the face region 502 in the device gain map so that the face region 502 is displayed at the target luminance value 505. When capturing a night scene, an exposure amount lower than the typical appropriate exposure is selected, resulting in a dark face region. Therefore, a scene gain map for correcting the device gain map so that the face region is appropriately exposed is generated.

[0060] The image processing unit 107 generates a gain map for a photographic scene for a channel for which a gain map for a device has been generated. The image processing unit 107 also generates metadata including identification information for the photographic scene and identification information for the corresponding gain map for the device. The image processing unit 107 stores the generated gain map for the photographic scene and metadata in the primary storage device 104.

[0061] Then, the image processing unit 107 generates an image data file that stores the data of the base image, the gain map and metadata for the device, and the gain map and metadata for the shooting scene, and records the file on the recording medium 108.

[0062] Consider a case where the image data file recorded in this way is displayed on an electronic device that has a display device or is connected to a display device. If the electronic device or display device does not support HDR images or gain maps, and the base image is an SDR image, the electronic device will display the SDR image on the display device. This operation is the same as conventional.

[0063] On the other hand, if the display device supports HDR and the electronic device supports gain maps, the electronic device corrects a device gain map according to the capabilities of the display device with a scene gain map, and then applies the corrected gain map to the base image data to generate HDR image data for display. The operation of applying a gain map to base image data to generate HDR image data is described in Non-Patent Document 1, and therefore a detailed description thereof will be omitted.

[0064] As described above, according to this embodiment, in addition to the device gain map generated based on the SDR image and HDR image based on the same RAW data, a shooting scene gain map is generated to modify the device gain map depending on the shooting scene, which enables more detailed display control in line with the photographer's intentions depending on the shooting scene.

[0065] In this embodiment, the gain map for the device and the gain map for the photographic scene are recorded separately in the image data file. However, when one type of photographic scene is determined for one frame of image as in this embodiment, only the gain map for the device corrected with the gain map for the photographic scene may be recorded in the image data file. This makes it possible to prevent an increase in the size of the image data file.

[0066] Furthermore, even when a gain map for a captured scene is separately recorded, only the portion having significant values ​​may be generated and recorded. In this case, for example, the entire gain map may be divided into multiple blocks in the horizontal and vertical directions, and the blocks containing significant values ​​and information indicating the positions of the blocks may be recorded. This can prevent an increase in the image data file size.

[0067] ●(Second embodiment) Next, a second embodiment of the present invention will be described. In this embodiment, a gain map generation operation for a photographic scene in S303 will be described when multiple types of photographic scenes can be determined for one frame of an image in the scene determination process in S302 of the first embodiment.

[0068] Fig. 6(a) is the base image 501 shown in Fig. 5(a). In this embodiment, it is assumed that both a night scene and a portrait scene are determined for the base image 501 in the scene determination process in S302.

[0069] In this case, the image processing unit 107 generates a gain map for a night scene and a gain map for a portrait scene in S303. Here, it is assumed that for a night scene, the color temperature of the entire image is set to be increased (to be cooler), and for a portrait scene, the color temperature is set to be decreased (to be warmer) while providing appropriate exposure to the skin color areas of a person (face, limbs, etc.).

[0070] Fig. 6(b) schematically shows a gain map 601 for a portrait scene, and Fig. 6(d) shows a gain map 602 for a night scene. The gain map 601 for a portrait scene does not have significant values ​​for the background region except for the face region 502 in Fig. 5(a).

[0071] FIG. 6( c ) schematically shows an image 603 obtained by applying a gain map 601 for a portrait scene to the base image 501 . FIG. 6( e ) schematically shows an image 604 obtained by applying a gain map 602 for a night scene to the base image 501 . FIG. 6(f) schematically shows an image 605 obtained by applying both a gain map 601 for a portrait scene and a gain map 602 for a night scene to the base image 501. When multiple gain maps for a scene are applied in combination, there may be areas that are multiplied by multiple gain values. In this case, if any of the gain values ​​is large, it becomes difficult to obtain the effect of the remaining gain values. Therefore, the gain values ​​in areas that are multiplied by multiple gain values ​​may be corrected so that they do not exceed an upper limit value. The upper limit value of the gain may be set, for example, within a range of values ​​that does not cause color unevenness when applied alone.

[0072] When multiple types of photographic scenes can be determined for one frame of image, if a gain map is generated for each combination of device (capability) type n (≧2) and photographic scene type m (≧2), the total number of gain maps will be n × m. However, by separately generating gain maps for photographic scenes, the total number of gain maps will be n + m. Therefore, if at least one of n and m is 3 or greater, the total number of gain maps can be reduced. This makes it possible to suppress an increase in the size of individual image data files due to the inclusion of gain maps. Note that, in this embodiment as well, as described in the first embodiment, only portions of gain maps for photographic scenes that have significant values ​​may be recorded.

[0073] (Other embodiments) In the above-described embodiment, when a shooting mode for capturing a specific scene is set, if the imaging device 100 performs correction similar to the gain map for the scene in the process of generating an SDR image and an HDR image, the generation of the gain map for the scene may be omitted.

[0074] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0075] The disclosure of the present embodiment includes the following image processing device, imaging device, image processing method, and program. (Item 1) an acquisition means for acquiring SDR image data and HDR image data based on the same image; a first generation means for generating a first gain map in which a plurality of gain values ​​are two-dimensionally arranged, the first gain map being applied to base image data that is one of the SDR image data and the HDR image data, thereby being capable of generating the other of the SDR image data and the HDR image data; A determination means for determining a photographed scene based on the SDR image data or the HDR image data; a second generating means for generating a second gain map to be used in combination with the first gain map in accordance with the photographic scene determined by the determining means; 1. An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the second gain map is for correcting at least one of brightness and color. (Item 3) 3. The image processing device according to item 1 or 2, further comprising a file generation means for generating an image data file that stores the base image data, the first gain map and the second gain map, or the first gain map corrected by the second gain map. (Item 4) Item 4. The image processing device according to item 3, characterized in that when the determination means determines one photographic scene for the same image data, the file generation means stores the first gain map corrected by the second gain map in the image data file. (Item 5) Item 4. The image processing device according to item 3, characterized in that, when the determination means is capable of determining a plurality of photographic scenes for the same image data, the file generation means stores the first gain map and the second gain map in the image data file. (Item 6) When the determination means determines a plurality of photographic scenes for the same image data, the second generating means generates the second gain map for each of the plurality of photographed scenes. 6. The image processing device according to any one of items 1 to 3 and 5, (Item 7) 7. The image processing device according to item 6, wherein when generating a plurality of the second gain maps, the second generation means generates each of the second gain maps so that the gain value of an area overlapping with another second gain map does not exceed an upper limit value. (Item 8) 8. The image processing device according to any one of items 1 to 7, wherein the second generating means generates the second gain map based on a type of area and a correction amount according to a photographed scene. (Item 9) 9. The image processing device according to item 8, wherein the second generating means generates the second gain map only for an area of ​​a type corresponding to the photographed scene. (Item 10) 10. The image processing device according to any one of items 1 to 9, wherein the first generating means generates a plurality of the first gain maps according to the capabilities of the display device. (Item 11) 11. The image processing device according to item 10, wherein the capability of the display device is the display peak luminance of the display device. (Item 12) An imaging element; The image processing device according to any one of items 1 to 11, The imaging device, characterized in that the image processing device uses the SDR image data and the HDR image data generated based on an image captured using the imaging element. (Item 13) An image processing method executed by an image processing device, Obtaining SDR image data and HDR image data based on the same image; generating a first gain map in which a plurality of gain values ​​are two-dimensionally arranged, wherein the first gain map is data that can generate the other of the SDR image data and the HDR image data by applying the first gain map to base image data that is one of the SDR image data and the HDR image data; determining a photographed scene based on the SDR image data or the HDR image data; generating a second gain map for use in combination with the first gain map in accordance with the determined photographic scene; An image processing method comprising: (Item 14) 12. A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 1 to 11.

[0076] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0077] 100...imaging device, 101...optical system, 102...imaging element, 103...CPU, 104...primary storage device, 107...image processing device, 108...recording medium, 109...secondary storage device, 110...display unit, 111...operation unit

Claims

1. an acquisition means for acquiring SDR image data and HDR image data based on the same image; a first generating means for generating a first gain map in which a plurality of gain values ​​are two-dimensionally arranged, the first gain map being applied to base image data that is one of the SDR image data and the HDR image data, thereby being able to generate the other of the SDR image data and the HDR image data; a determination means for determining a photographed scene based on the SDR image data or the HDR image data; a second generating means for generating a second gain map to be used in combination with the first gain map in accordance with the photographic scene determined by the determining means; 1. An image processing device comprising:

2. 2. The image processing apparatus of claim 1, wherein the second gain map is for modifying at least one of luminance and color.

3. 2. The image processing device according to claim 1, further comprising a file generating means for generating an image data file that stores the base image data, the first gain map and the second gain map, or the first gain map corrected by the second gain map.

4. 4. The image processing device according to claim 3, wherein when the determination means determines one photographic scene for the same image data, the file generation means stores the first gain map corrected by the second gain map in the image data file.

5. 4. The image processing device according to claim 3, wherein when the determination means is capable of determining a plurality of photographic scenes for the same image data, the file generation means stores the first gain map and the second gain map in the image data file.

6. When the determination means determines a plurality of photographic scenes for the same image data, the second generating means generates the second gain map for each of the plurality of photographic scenes.

2. The image processing device according to claim 1, wherein:

7. 7. The image processing device according to claim 6, wherein when generating a plurality of the second gain maps, the second generation means generates each of the second gain maps so that the gain value of an area overlapping with another second gain map does not exceed an upper limit value.

8. 2. The image processing apparatus according to claim 1, wherein the second generating means generates the second gain map based on a type of area and a correction amount according to a photographic scene.

9. 9. The image processing apparatus according to claim 8, wherein the second generating means generates the second gain map only for an area of ​​a type corresponding to the photographed scene.

10. 2. The image processing device according to claim 1, wherein the first generating means generates a plurality of the first gain maps according to the capabilities of the display device.

11. 11. The image processing device according to claim 10, wherein the capability of the display device is a display peak luminance of the display device.

12. An imaging element; The image processing device according to any one of claims 1 to 11, The imaging device, wherein the image processing device uses the SDR image data and the HDR image data generated based on an image captured using the imaging element.

13. An image processing method executed by an image processing device, Obtaining SDR image data and HDR image data based on the same image; generating a first gain map in which a plurality of gain values ​​are two-dimensionally arranged, wherein the first gain map is data that can generate the other of the SDR image data and the HDR image data by applying the first gain map to base image data that is one of the SDR image data and the HDR image data; determining a photographed scene based on the SDR image data or the HDR image data; generating a second gain map for use in combination with the first gain map in accordance with the determined photographic scene; An image processing method comprising:

14. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 11.

Images