Image processing device and image processing method

Abstract

Provided is an image processing device capable of generating a gain map in consideration of a capturing scene. The image processing device generates a first gain map from SDR image data and HDR image data based on the same image. The first gain map is data in which a plurality of gain values are two-dimensionally arranged and which is capable of being applied to one of the SDR image data and the HDR image data to thereby generate the other. The image processing device generates a second gain map for use in combination with the first gain map according to the capturing scene determined on the basis of the SDR image data or the HDR image data.

Description

Image processing device and image processing method

[0001] The present invention relates to an image processing device and an image processing method, and more particularly to a dynamic range conversion technique for an image.

[0002] For example, the ITU-R BT.2100 standard is known, which relates to images with a wider luminance dynamic range (HDR) than the luminance dynamic range (SDR) conforming to ITU-R BT.709. Also, a gain map has been proposed for dynamically generating an image with an appropriate dynamic range from an SDR image or an HDR image according to the capabilities of a display device, etc. (Non-Patent Document 1). The gain map is a map of coefficients (gains) to be applied to each pixel constituting an image. The resolution of the gain map may be equal to or less than the resolution of the image to which the gain map is applied.

[0003] Eric Chan, "Gain Maps" Version 1.0 draft 15, [online], February 28, 2024, [Retrieved May 15, 2024], Internet <URL: https: / / helpx.adobe.com / content / dam / help / en / camera-raw / using / gain-map / jcr_content / root / content / flex / items / position / position-par / table / row-3u03dx0-column-4a63daf / download_section / download-1 / Gain_Map_1_0d15.pdf>

[0004] In order to display images with the intended contrast on various display devices, it is considered desirable to prepare a gain map that takes into account not only the characteristics of the display device but also the shooting scene. However, Non-Patent Document 1 does not mention a gain map that takes into account the shooting scene.

[0005] In one embodiment, the present invention provides an image processing device and an image processing method capable of generating a gain map that takes into account a photographed scene.

[0006] In one aspect, the present invention provides an image processing device comprising: an acquisition means for acquiring SDR image data and HDR image data based on the same image; a first generation means for generating a first gain map in which a plurality of gain values ​​are arranged two-dimensionally, the first gain map being capable of generating the other of the SDR image data and the HDR image data by applying the first gain map to base image data, which is one of the SDR image data and the HDR image data; a determination means for determining a photographic scene based on the SDR image data or the HDR image data; and a second generation means for generating a second gain map to be used in combination with the first gain map in accordance with the photographic scene determined by the determination means.

[0007] According to the present invention, it is possible to provide an image processing device and an image processing method that are capable of generating a gain map that takes into account a photographed scene.

[0008] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings, in which the same or similar elements are designated by the same reference numerals.

[0009] The accompanying drawings are incorporated in and constitute a part of the specification, illustrate embodiments of the present invention, and together with the description are used to explain the principles of the present invention: a block diagram showing an example of the functional configuration of an imaging device as an image processing device according to an embodiment of the present invention; a flowchart relating to the operation of an image processing unit in a first embodiment; a flowchart relating to the operation of an image processing unit in a first embodiment; a diagram relating to the generation of a gain map for the device; a diagram relating to the generation of a gain map for a photographed scene; a diagram schematically showing the operation of a second embodiment;

[0010] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] In the following, the present invention will be described in terms of an embodiment using an imaging device (digital camera) as an example of an image processing device. However, imaging functionality is not essential to the present invention, and the present invention can be implemented in any electronic device having one or more arithmetic circuits or processors. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, smart watches, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0012] 1 is a block diagram showing an example of the functional configuration of an imaging device as an image processing device according to a first embodiment of the present invention. The imaging device 100 includes an optical system 101, an image sensor 102, a central processing unit (CPU) 103, a primary storage device 104, a photometric sensor 105, an external communication unit 106, an image processing unit 107, a recording medium 108, a secondary storage device 109, a display unit 110, and an operation unit 111.

[0013] FIG. 1 is a block diagram showing an example of the functional configuration of an imaging device 100 capable of implementing the present invention. Each functional block of the imaging device 100 can be implemented by software or a combination of software and hardware, except for parts that can clearly be realized only by hardware (e.g., lenses in the optical system 101, pixels in the image sensor 102, etc.). For example, a functional block may be realized by dedicated hardware such as an ASIC. A functional block may also be realized by a processor such as a CPU executing a program stored in memory. Multiple functional blocks may also be realized by a common configuration (e.g., a single ASIC). Hardware that realizes part of the functions of one functional block may also be included in hardware that realizes another functional block.

[0014] In this embodiment, the image processing unit 107 generates an SDR image, an HDR image, and a gain map based on a RAW image obtained by capturing an image using the image sensor 102. The gain map is data that can be applied to a base rendition (one of an HDR image or an SDR image) to generate a representation of the other image. Furthermore, by applying a weight between 0 and 1 to the gain map during display, an adaptive HDR representation can be realized.

[0015] For example, an image data file is generated in which a gain map is attached to an SDR image as a base representation. At the time of display, an image for display can be generated by applying appropriate weights and gain maps to the SDR image according to the capabilities of the display device. If the display device does not support HDR, the gain map is ignored (the weight is set to 0), and the SDR image is displayed.

[0016] In this embodiment, two signal characteristics are assumed to represent the relationship between the video signal level and display brightness in an HDR image: PQ (Perceptual Quantization) and HLG (Hybrid Log Gamma). PQ is specified as an EOTF (electrical-to-optical transfer function) in SMPTE ST 2084, and HLG is specified as an OETF (optical-to-electrical transfer function) in ARIB STD-B67.

[0017] An SDR image is an image with a narrower output range (dynamic range) than an HDR image. In the following description, the gamma of an HDR image is assumed to be the OETF characteristic (PQ Inverse EOTF) based on SMPTE ST 2084, and the color gamut is assumed to conform to the ITU-R BT.2020 standard. The gamma of an SDR image is assumed to be sRGB gamma, and the color gamut is assumed to be sRGB.

[0018] The optical system 101 includes a lens, a shutter, an aperture, and a mechanism (such as a motor) for driving these. The optical system 101 may be integrated with the imaging device 100 or may have the form of an interchangeable lens. The optical system 101 forms an optical image of a subject on the imaging surface of the imaging element 102.

[0019] The image sensor 102 may be, for example, a known CCD or CMOS color image sensor with a primary-color Bayer array of color filters. The image sensor 102 has a pixel array in which multiple pixels are arranged two-dimensionally, and peripheral circuits for reading out signals from each pixel. Each pixel accumulates charge according to the amount of incident light through photoelectric conversion. By reading out signals from each pixel having voltages according to the amount of charge accumulated during the exposure period, a group of pixel signals (analog image signals) representing the optical image formed on the imaging surface is obtained.

[0020] The analog image signal is A / D converted to a digital image signal (image data) and then stored in the primary storage device 104. The A / D conversion may be performed by the image sensor 102, or may be performed by another component (e.g., an A / D converter not shown) outside the image sensor 102. Image data stored in the primary storage device 104 at this stage, which is composed of pixel data having one color component of R, G, or B, is called RAW image data.

[0021] The imaging conditions (shutter speed or exposure time, aperture value, and imaging sensitivity) are determined by the CPU 103.

[0022] An optical image formed by the optical system 101 is projected onto the photometric sensor 105. The photometric sensor 105 measures luminance information of the optical image for each of a plurality of photometric regions (for example, 12 horizontally and 8 vertically, for a total of 96 regions). The photometric sensor 105 outputs the measurement results to the CPU 103. Note that the number, positions, and sizes of the photometric regions may change depending on the settings of the imaging device 100, etc.

[0023] The CPU 103 loads a program stored in the secondary storage device 109 into the primary storage device 104 and executes it to control the operation of each unit of the image capture device 100 and realize various functions of the image capture device 100. Note that at least some of the functions described as being realized by the CPU 103 executing a program may be executed by hardware such as an ASIC or FPGA configured to execute the function.

[0024] The primary storage device 104 is a volatile storage device such as a RAM, and is used to temporarily store various data, such as programs executed by the CPU 103, data required for executing the programs, data before, during, and after processing by the image processing unit 107, image data for display, and image data for recording.

[0025] The secondary storage device 109 is a non-volatile storage device such as an EEPROM, and stores programs executed by the CPU 103 to control the image capture device 100, various settings, information about the image capture device 100, GUI data, and the like.

[0026] The recording medium 108 is used as a recording destination for the image data for recording stored in the primary storage device 104. The recording medium 108 is removable from the imaging device 100 and may be, for example, a semiconductor memory card. The image data recorded on the recording medium 108 can be read by other devices such as a personal computer. The imaging device 100 has a mechanism for attaching and detaching the recording medium 108 and a read / write function.

[0027] The image processing unit 107 applies predetermined image processing to image data stored in the primary storage device 104, such as after being read from the image sensor 102 or the recording medium 108, to acquire and / or generate signals, image data, and information appropriate for the intended use. The image processing unit 107 may be a dedicated hardware circuit, such as an ASIC (Application Specific Integrated Circuit) designed to implement a specific function. Alternatively, the image processing unit 107 may be configured such that a processor, such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit), executes software to implement a specific function. The image processing unit 107 outputs the acquired or generated signals, information, and data to the CPU 103 or stores them in the primary storage device 104, depending on the intended use.

[0028] Image processing applied by the image processing unit 107 may include, for example, preprocessing, color interpolation, correction, detection, data processing, evaluation value calculation, and special effect processing. Preprocessing may include signal amplification, reference level adjustment, and defective pixel correction. Color interpolation, performed when a color filter is provided on the image sensor 102, interpolates the values ​​of color components not included in the individual pixel data constituting the image data. Color interpolation is also called demosaicing. Correction may include white balance adjustment, gradation correction (gamma), correction of image degradation caused by optical aberrations of the optical system 101 (image restoration), correction of the effects of peripheral light falloff of the optical system 101, and color correction. Detection may include detection of feature regions (e.g., face regions or human body regions) and their movement, person recognition, and the like. Data processing may include region extraction (trimming), synthesis, scaling, encoding and decoding, header information generation (data file generation), and the like. Data processing also includes generating image data for display and image data for recording, generating a gain map (described later), and converting the color gamut and dynamic range. Evaluation value calculation processing can include processing such as generating signals and evaluation values ​​used for autofocus detection (AF) and generating evaluation values ​​used for autoexposure control (AE). Special effect processing can include processing such as adding a blur effect, changing color tones, and relighting. Note that these are examples of processing that can be applied by the image processing unit 107, and do not limit the processing that can be applied by the image processing unit 107.

[0029] The display unit 110 is, for example, a liquid crystal display provided on the surface of the housing of the imaging device 100. The display unit 110 displays information about the imaging device 100, a menu screen, images captured by the image sensor 102, images read from the recording medium 108, and the like. Note that the imaging device 100 can cause the display unit 110 to function as an electronic viewfinder (EVF) by continuously capturing video and displaying the video on the display unit 110 while in a shooting standby state. The operation of causing the display unit 110 to function as an EVF is called live view display, and the image used for the live view display is called a live view image.

[0030] The operation unit 111 is a collective term for input devices (such as buttons, switches, and dials) provided for the user to input various instructions to the imaging apparatus 100. The input devices constituting the operation unit 111 are named according to their assigned functions. For example, the operation unit 111 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, and a confirmation key. The release switch is a switch for recording still images, and the CPU 103 recognizes a half-pressed state of the release switch as an instruction to prepare for shooting and a full-pressed state as an instruction to start shooting. Furthermore, the CPU 103 recognizes a press of the video recording switch in a shooting standby state as an instruction to start video recording, and a press of the video recording switch during video recording as an instruction to stop recording. The functions assigned to the same input device may be variable. Furthermore, the input device may be software buttons or keys using a touch display. Furthermore, the operation unit 111 may include an input device compatible with non-contact input methods such as voice input or eye-gaze input.

[0031] In this embodiment, an operation mode for capturing a specific scene can be set in the imaging device 100. Typical examples of the specific scene include, but are not limited to, a portrait, a sunset, a night view, fireworks, a starry sky, and cooking.

[0032] The external communication unit 106 is a communication interface between an external device and the imaging device 100. The external communication unit 106 supports one or more wired and / or wireless communication standards. The external communication unit 106 has a transmission / reception circuit according to the supported standard, and an associated connector or antenna. Typical examples of standards supported by the external communication unit 106 include, but are not limited to, USB, wireless LAN, Bluetooth (registered trademark), and HDMI (registered trademark).

[0033] The CPU 103 transmits image data to an external device via the external communication unit 106, acquires various information from the external device, controls the operation of the external device, and receives commands from the external device to control the operation of the imaging device 100.

[0034] Next, the gain map generation operation of the image processing unit 107 will be described using the flowchart shown in Fig. 2. Here, it is assumed that a gain map is generated for still image RAW image data captured by the image sensor 102 and stored in the primary storage device 104. However, the gain map can also be similarly applied to generating a gain map for still image RAW image data read from the recording medium 108 and stored in the primary storage device 104. Furthermore, a gain map can also be generated for RAW video data in the same way as for still image RAW image data by treating each frame as RAW image data.

[0035] The process shown in FIG. 2 can be executed, for example, when still image shooting is performed, or when a user instructs generation of a gain map for specific RAW image data.

[0036] The gain map is recorded in association with data of the image (hereinafter referred to as the base image) used as the base representation when the gain map was generated. The base image may be either an SDR image or an HDR image, but here it is assumed to be an SDR image. This is to enable the display of an SDR image even if the device displaying the recorded image does not support HDR images and / or gain maps.

[0037] Here, we will assume that the raw image data handled is that of an image captured by an image sensor having a primary color Bayer array color filter. Therefore, the raw image data processed by the image processing unit 107 contains information on one color component, R (red), G (green), or B (blue), for each pixel. Note that the gain map generation process does not depend on the type of color filter.

[0038] In step S201 , the image processing unit 107 generates SDR image data from the RAW image data stored in the primary storage device 104 , and stores the SDR image data in the primary storage device 104 .

[0039] In S202 , the image processing unit 107 generates HDR image data from the RAW image data stored in the primary storage device 104 , and stores the HDR image data in the primary storage device 104 .

[0040] The image processing applied to the RAW image data by the image processing unit 107 in S201 and S202 is processing for generating image data in which each pixel contains three primary color components (RGB, YUV, etc.), and is also called development processing. This generally includes gamma processing, white balance adjustment processing, and color interpolation processing. The gamma used in the gamma processing differs between generating an SDR image and generating an HDR image.

[0041] Although SDR image data and HDR image data are generated in S201 and S202 here, pre-recorded SDR image data and HDR image data based on the same image may be acquired from the recording medium 108, for example.

[0042] In S203, the image processing unit 107 matches the color space and gamma (EOTF or OETF) of the SDR image data and the HDR image data. The image processing unit 107 can match the color space and gamma of both sets of image data by converting the color space and gamma of one or both sets of image data using any known method.

[0043] For example, when matching an SDR image with an HDR image, the color gamut conversion method from BT 709 to BT 2020 described in ITU-R BT 2087 can be used. Note that sRGB shares a color gamut with BT 709. The conversion method used in S203 can be changed as appropriate depending on the color gamut and gamma of the SDR image and HDR image. Furthermore, the gamma of the SDR image and HDR image may be converted to linear (γ=1.0).

[0044] In S204, the image processing unit 107 generates a gain map in which a plurality of gains are arranged two-dimensionally from the SDR image and HDR image whose color space and gain are combined. The gain map G is defined by the following equation: G=log ２ ((HDR image +k ｈｄｒ ) / (SDR image + k ｓｄｒ )) k ｈｄｒ and k ｓｄｒis an offset coefficient that is determined so that the value in the parentheses for calculating the logarithm is positive. A gain map is generated by calculating the gain for each pixel in the HDR image and the SDR image that corresponds in position according to the above formula.

[0045] The gain map may be generated for each component (channel) such as RGB or YUV, or may be generated for one component such as the luminance component (grayscale). The luminance component may be, for example, the Y component of YUV or the G component of RGB. The image processing unit 107 converts the image data from RGB format to YUV format as necessary.

[0046] The gain that makes up the gain map has a minimum value of 0 and a maximum value of 2. N The gain ranges from -1 to 100. N is the number of encoding bits. Therefore, when encoding with 8 bits, the maximum gain value is 255. A gain of 0 does not affect the brightness. A negative gain means to darken the image, and a positive gain means to brighten the image.

[0047] The image processing unit 107 can generate a gain map using, for example, the method described in Non-Patent Document 1. The image processing unit 107 also generates metadata for the gain map. The metadata may be, but is not limited to, the number of gain maps (number of channels), the type of base image, the minimum and maximum gain values, an offset coefficient, the size of the gain map, and information at the time of image capture (such as the capture mode and scene information).

[0048] The gain map generation process in S204 will be described in detail with reference to the flowchart shown in FIG.

[0049] In S301, the image processing unit 107 (first generation means) generates a gain map (device gain map) for each display device capability. This is to allow HDR images to be displayed with appropriate contrast on display devices with different capabilities. Here, as an example of different capabilities depending on the display device, multiple device gain maps corresponding to different display peak luminances are generated. For example, in S202, the image processing unit 107 generates multiple HDR images with different peak luminances and generates device gain maps using the SDR image and each HDR image.

[0050] Here, an example will be described in which gain maps are generated assuming display devices with display peak luminances of 400 nits and 600 nits. Note that gain maps may be generated for three or more display peak luminances, for example, by adding 1000 nits. Furthermore, multiple gain maps may be generated for each model of display device, rather than for each capability value.

[0051] 4 shows examples of gain curves for HDR images with peak luminances of 400 nits and 600 nits generated in S202 corresponding to an SDR image with a peak luminance of 100 nits. Here, the two gain curves have the same characteristics as the gain curve for the SDR image up to an input luminance of 18 nits. However, if brightness contrast occurs in an image to which a gain map based on these gain maps has been applied, the gain curve may be adjusted to suppress the brightness contrast. Brightness contrast is a phenomenon in which a color appears darker or brighter than its actual color depending on the brightness of the surrounding colors.

[0052] The image processing unit 107 stores the generated gain maps for the multiple devices in the primary storage device 104 together with the respective metadata.

[0053] In S302, the image processing unit 107 (determination means) determines the shooting scene of the image. The image processing unit 107 can determine the shooting scene by analyzing the SDR image generated in S201 or the HDR image generated in S202 using a known method. Furthermore, the image processing unit 107 may consider an image captured in a shooting mode for capturing a specific scene during shooting as an image capturing the specific scene assumed by the shooting mode. Note that, as described in the second embodiment, one frame of an image may correspond to multiple shooting scenes. For example, an image of a person captured against a night scene background corresponds to both a night scene scene and a portrait scene. However, in the first embodiment, only one type of shooting scene is determined for one frame of an image.

[0054] In S303, the image processing unit 107 (second generation means) generates a gain map for the photographic scene determined in S302. The gain map for the photographic scene is a gain map to be used in combination with the gain map for the device generated in S301. Specifically, the gain map for the photographic scene is used to modify the gain map for the device in accordance with the photographic scene. Unlike the gain map for the device, the gain map for the photographic scene is not generated based on the SDR image and the HDR image.

[0055] Specifically, information on the area where the gain should be corrected and information on the amount of correction are associated with each photographic scene and are stored in advance in the secondary storage device 109. Then, the image processing unit 107 reads out from the secondary storage device 109 information on the area where the gain should be corrected and information on the amount of correction that corresponds to the type of photographic scene determined in S302.

[0056] The image processing unit 107 identifies the area in the SDR image or the HDR image where the gain should be corrected based on the information about the area where the gain should be corrected. Then, the image processing unit 107 generates a gain map for correcting the gain in the area where the gain should be corrected from the gain map for the device generated in S301 based on the information about the correction amount.

[0057] The gain map for the photographic scene has a significant value (other than 1) corresponding to the amount of gain correction in the area of ​​the gain map for the device where the gain should be corrected, and is 1 in other areas. The gain map for the photographic scene is intended to be multiplied by the gain map for the device to correct the gain map for the device. Therefore, for areas where the gain value is not corrected, the gain value of the gain map for the scene is set to 1 so that the gain value in the gain map for the device is maintained.

[0058] The region for which the gain should be corrected may be the entire image or a specific region depending on the shooting scene. In the case of a specific region, it can be set from a region type that the image processing unit 107 can detect from the image. For example, if the image processing unit 107 can detect a face (person, dog, cat, etc.), a bird, a human body, or a vehicle as a characteristic region, it can set one or more of these types of regions as the region for which the gain should be corrected.

[0059] The correction amount may be a specific target luminance value (absolute value), or may be information indicating a target state, such as "appropriate exposure," "+1 stop," or "increase / decrease in color temperature." When a specific luminance value is set as the correction amount, the image processing unit 107 determines a correction amount that will set the (average) display luminance of the region in the base image where the gain is to be corrected to the specific luminance value. When "appropriate exposure" is set as the correction amount, the image processing unit 107 determines a correction amount that will set the (average) display luminance of the region in the base image where the gain is to be corrected to an appropriate brightness relative to the peak luminance of the image. The appropriate brightness relative to the peak luminance can be predetermined as a percentage of the peak luminance. When "+1 stop" is set as the correction amount, the image processing unit 107 determines a correction amount that will set the (average) display luminance of the region in the base image where the gain is to be corrected to an exposure that is appropriate exposure +1 stop. For color temperature, the image processing unit 107 determines a correction amount that will increase / decrease one or more color components by a predetermined amount.

[0060] In FIG. 5A , reference numeral 501 denotes a base image determined to be a night scene in S302. Here, it is assumed that the night scene is set to provide appropriate exposure for a person's face region or human body region. FIG. 5B is a diagram illustrating the concept of the correction amount. Reference numeral 503 denotes the peak luminance of the base image 501. The peak luminance of a specific image can vary within a range equal to or less than the peak luminance that the base image can achieve. Reference numeral 504 denotes a representative luminance value of a person's face region 502 in the base image 501. Reference numeral 505 denotes a set target luminance value. In this case, the image processing unit 107 generates a scene gain map for correcting the gain of the region corresponding to the face region 502 in the device gain map so that the face region 502 is displayed at the target luminance value 505. When capturing a night scene, an exposure amount lower than the general appropriate exposure is selected, resulting in a dark face region. Therefore, a capture scene gain map is generated that corrects the device gain map so that the face region is properly exposed.

[0061] The image processing unit 107 generates a gain map for a photographic scene for a channel for which a gain map for a device has been generated. The image processing unit 107 also generates metadata including identification information for the photographic scene and identification information for the corresponding gain map for the device. The image processing unit 107 stores the generated gain map for the photographic scene and metadata in the primary storage device 104.

[0062] The image processing unit 107 then generates an image data file that stores the base image data, the gain map and metadata for the device, and the gain map and metadata for the shooting scene, and records the file on the recording medium 108 .

[0063] Consider a case where the image data file recorded in this manner is displayed on an electronic device that has a display device or is connected to a display device. If the electronic device or display device does not support HDR images or gain maps, and the base image is an SDR image, the electronic device will display the SDR image on the display device. This operation is conventional.

[0064] On the other hand, if the display device is HDR-compatible and the electronic device is gain map-compatible, the electronic device modifies a device gain map according to the capabilities of the display device with a scene gain map, and then applies the modified gain map to the base image data to generate HDR image data for display. The operation of applying a gain map to base image data to generate HDR image data is described in Non-Patent Document 1, and therefore a detailed description thereof will be omitted.

[0065] As described above, according to this embodiment, in addition to the device gain map generated based on the SDR image and the HDR image, which are based on the same RAW data, a shooting scene gain map is generated to modify the device gain map depending on the shooting scene, thereby enabling more detailed display control in line with the photographer's intentions depending on the shooting scene.

[0066] In this embodiment, the gain map for the device and the gain map for the photographic scene are recorded separately in the image data file. However, when one type of photographic scene is determined for one frame of image as in this embodiment, only the gain map for the device corrected with the gain map for the photographic scene may be recorded in the image data file. This makes it possible to prevent an increase in the size of the image data file.

[0067] Furthermore, even when a gain map for a captured scene is separately recorded, only the portion having significant values ​​may be generated and recorded. In this case, for example, the entire gain map may be divided into multiple blocks in the horizontal and vertical directions, and the blocks containing significant values ​​and information indicating the positions of the blocks may be recorded. This can prevent an increase in the image data file size.

[0068] Second Embodiment Next, a second embodiment of the present invention will be described with reference to Fig. 6. This embodiment describes the operation of generating a gain map for a photographic scene in S303 when multiple types of photographic scenes can be determined for one frame of image in the scene determination process in S302 of the first embodiment.

[0069] In this embodiment, it is assumed that in the scene determination process in S302, both a night scene and a portrait scene are determined for the base image 501 shown in FIG. 5A.

[0070] Then, in S303, the image processing unit 107 generates a gain map 602 for a night scene and a gain map 601 for a portrait scene. Here, it is assumed that for a night scene, the color temperature of the entire image is set to be increased (to be cooler), and for a portrait scene, the color temperature is set to be decreased (to be warmer) while providing appropriate exposure to the skin color areas of people (faces and limbs).

[0071] The gain map 601 for the portrait scene has no significant values ​​for the background regions except for the face region 502 in FIG. 5A.

[0072] An image 603 is an image obtained by applying a gain map 601 for a portrait scene to the base image 501. An image 604 is an image obtained by applying a gain map 602 for a night scene to the base image 501. An image 605 is an image obtained by applying both the gain map 601 for a portrait scene and the gain map 602 for a night scene to the base image 501.

[0073] When multiple gain maps for a scene are applied in combination, there may be areas that are multiplied by multiple gain values. In this case, if any of the gain values ​​is large, it becomes difficult to obtain the effect of the remaining gain values. Therefore, the gain values ​​in areas that are multiplied by multiple gain values ​​may be corrected so that they do not exceed an upper limit value. The upper limit value of the gain may be set, for example, within a range of values ​​that does not cause color unevenness when applied alone.

[0074] When multiple types of photographic scenes can be determined for one frame of image, if a gain map is generated for each combination of device (capability) type n (≧2) and photographic scene type m (≧2), the total number of gain maps will be n × m. However, by separately generating gain maps for photographic scenes, the total number of gain maps will be n + m. Therefore, if at least one of n and m is 3 or greater, the total number of gain maps can be reduced. This makes it possible to suppress an increase in the size of individual image data files due to the inclusion of gain maps. Note that, in this embodiment, as described in the first embodiment, only portions of gain maps for photographic scenes that have significant values ​​may be recorded.

[0075] (Other embodiments) In the above-described embodiments, when a shooting mode for capturing a specific scene is set, if the imaging device 100 performs correction similar to the gain map for the scene in the process of generating an SDR image and an HDR image, the generation of the gain map for the scene may be omitted.

[0076] The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

[0077] The disclosure of the present embodiment includes the following image processing device, imaging device, image processing method, and program.

[0078] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention.

[0079] This application claims priority based on Japanese Patent Application No. 2024-87245, filed May 29, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An image processing device comprising: an acquisition means for acquiring SDR image data and HDR image data based on the same image; a first generation means for generating a first gain map in which a plurality of gain values ​​are arranged two-dimensionally, the first gain map being applied to base image data that is one of the SDR image data and the HDR image data to generate the other of the SDR image data and the HDR image data; a determination means for determining a photographic scene based on the SDR image data or the HDR image data; and a second generation means for generating a second gain map to be used in combination with the first gain map in accordance with the photographic scene determined by the determination means.

2. The image processing apparatus according to claim 1, wherein the second gain map is for modifying at least one of luminance and color.

3. An image processing device as described in claim 1 or 2, further comprising a file generation means for generating an image data file that stores the base image data, the first gain map and the second gain map, or the first gain map modified by the second gain map.

4. An image processing device as described in claim 3, characterized in that when the determination means determines one photographic scene for the same image data, the file generation means stores the first gain map corrected by the second gain map in the image data file.

5. An image processing device as described in claim 3, characterized in that when the determination means is capable of determining multiple shooting scenes for the same image data, the file generation means stores the first gain map and the second gain map in the image data file.

6. An image processing device as described in any one of claims 1 to 3 and 5, characterized in that, when the determination means determines multiple photographic scenes for the same image data, the second generation means generates the second gain map for each of the multiple photographic scenes.

7. The image processing device described in claim 6, characterized in that when generating multiple second gain maps, the second generation means generates each of the second gain maps so that the gain value of areas overlapping with other second gain maps does not exceed an upper limit value.

8. An image processing device according to any one of claims 1 to 7, characterized in that the second generating means generates the second gain map based on the type of area and the amount of correction according to the photographed scene.

9. The image processing device according to claim 8, wherein said second generating means generates said second gain map only for an area of ​​a type corresponding to said photographic scene.

10. An image processing device according to any one of claims 1 to 9, wherein the first generating means generates a plurality of the first gain maps according to the capabilities of the display device.

11. An image processing device according to claim 10, wherein the capability of said display device is the display peak luminance of said display device.

12. An imaging device comprising an imaging element and an image processing device according to any one of claims 1 to 11, wherein the image processing device uses the SDR image data and the HDR image data generated based on an image captured using the imaging element.

13. An image processing method executed by an image processing device, comprising: acquiring SDR image data and HDR image data based on the same image; generating a first gain map in which a plurality of gain values ​​are arranged two-dimensionally, wherein the first gain map is data that can generate the other of the SDR image data and the HDR image data by applying it to base image data that is one of the SDR image data and the HDR image data; determining a photographed scene based on the SDR image data or the HDR image data; and generating a second gain map to be used in combination with the first gain map according to the determined photographed scene.

14. A program for causing a computer to function as each of the means possessed by the image processing device according to any one of claims 1 to 11.

Images