Image processing system and image processing method

Abstract

To provide an image processing system and an image processing method capable of improving a representation by applying a reduced gain map.SOLUTION: An image processing system acquires a gain map in which application of one of SDR image data and HDR image data from the same image enables the generation of the other and acquires one of the SDR image data and HDR image data (base image data). The image processing system enlarges the gain map to match the resolution of the base image data and applies it to the base image data. The image processing system applies sharpness processing to an area where a size or value changes by a combination of reduction and enlargement processing, in the enlarged gain map or the image data to which the enlarged gain map has been applied.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device and an image processing method, and more particularly to a dynamic range conversion technique for an image. [Background technology]

[0002] For example, the ITU-R BT.2100 standard is known, which relates to images with a wider luminance dynamic range (HDR) than the luminance dynamic range (SDR) conforming to ITU-R BT.709. Also, a gain map has been proposed for dynamically generating an image with an appropriate dynamic range from an SDR image or an HDR image according to the capabilities of a display device, etc. (Non-Patent Document 1). The gain map is a map of coefficients (gains) applied to each pixel that constitutes an image. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Eric Chan, “Gain Maps” Version 1.0 draft 15”, [online], February 28, 2024, [Searched on May 15, 2020], Internet <URL:https: / / helpx.adobe.com / content / dam / help / en / camera-raw / using / gain-map / jcr_content / root / content / flex / items / position / position-par / table / row-3u03dx0-column-4a63daf / download_section / download-1 / Gain_Map_1_0d15.pdf> Summary of the Invention [Problem to be solved by the invention]

[0004] The gain map is recorded in a data file together with either the SDR or HDR image (base image) used to generate it. Normally, the resolution of the gain map (number of data points) is equal to the resolution (number of pixels) of the base image. However, to reduce the data file size, the resolution of the gain map may be recorded at a lower value than the resolution of the base image.

[0005] Since the gain map has the same data format as an image, its resolution can be reduced in the same way as an image is reduced. When the reduced gain map is applied to the base image, it is enlarged to the original resolution before reduction.

[0006] However, the scaled gain map will not be identical to the original gain map. This is because, if the original gain map is considered as a grayscale image, the scaling operations reduce the contrast of the gain map and cause thin lines and small areas with high brightness to be enlarged from their original size. Therefore, simply scaling the scaled gain map back to the original resolution and applying it to a base image may result in a different appearance than if the unscaled gain map were applied.

[0007] In view of the above problem, one aspect of the present invention provides an image processing device and an image processing method that can improve the representation to which a reduced gain map is applied. [Means for solving the problem]

[0008] In one aspect, the present invention provides an image processing device comprising: acquisition means for acquiring a gain map that can be applied to either SDR image data or HDR image data based on the same image to generate the other, and base image data that is one of the SDR image data and HDR image data; enlargement means for enlarging the gain map so that the resolution of the gain map is equal to the resolution of the image represented by the base image data; application means for applying the enlarged gain map to the base image data and acquiring image data to which the enlarged gain map has been applied; detection means for detecting, from the base image data, the gain map acquired by the acquisition means, or the enlarged gain map, a region whose size or value changes due to a combination of a reduction process and an enlargement process; and correction means for applying sharpness processing to a region of the enlarged gain map or the image data to which the enlarged gain map has been applied, that corresponds to the region detected by the detection means; and when applying sharpness processing to the enlarged gain map, the correction means applies the sharpness processing before the enlarged gain map is applied to the base image data. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide an image processing device and an image processing method that can improve the representation to which a reduced gain map is applied. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an imaging device according to an embodiment; [Figure 2] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing unit in a first embodiment. [Figure 3] 10 is a flowchart showing an example of the operation of the image processing unit in the first embodiment. [Figure 4] FIG. 10 is a diagram showing an example of the relationship between the reduction ratio and the strength of sharpness processing. [Figure 5] FIG. 10 is a block diagram showing an example of the functional configuration of an image processing unit in a second embodiment. [Figure 6]10 is a flowchart showing an example of the operation of an image processing unit in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] The present invention will be described in detail below based on exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Furthermore, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] In the following, the present invention will be described as being embodied in an imaging device (digital camera) as an example of an image processing device. However, imaging functionality is not essential to the present invention, and the present invention can be implemented in any electronic device having one or more arithmetic circuits or processors. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, smart watches, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0013] ●(First embodiment) 1 is a block diagram showing an example of the functional configuration of an imaging device as an image processing device according to a first embodiment of the present invention. The imaging device 100 includes an optical system 101, an image sensor 102, a central processing unit (CPU) 103, a primary storage device 104, a photometric sensor 105, an external communication unit 106, an image processing unit 107, a recording medium 108, a secondary storage device 109, a display unit 110, and an operation unit 111.

[0014] FIG. 1 is a block diagram showing an example of the functional configuration of an imaging device 100 capable of implementing the present invention. Each functional block of the imaging device 100 can be implemented by software or a combination of software and hardware, except for parts that can clearly be realized only by hardware (e.g., lenses included in the optical system 101, pixels of the image sensor 102, etc.). For example, a functional block may be realized by dedicated hardware such as an ASIC. Alternatively, a functional block may be realized by a processor such as a CPU executing a program stored in memory. Note that multiple functional blocks may be realized by a common configuration (e.g., a single ASIC). Furthermore, hardware that realizes part of the functions of one functional block may be included in hardware that realizes another functional block.

[0015] In this embodiment, the image processing unit 107 generates an SDR image, an HDR image, and a gain map based on a RAW image obtained by capturing an image using the image sensor 102. The gain map is data that can be applied to a base rendition (either an HDR image or an SDR image) to generate the other rendition. Furthermore, by applying a weight between 0 and 1 to the gain map during display, an adaptive HDR rendition can be realized.

[0016] For example, an image data file is generated with a gain map attached to an SDR image as a base representation. When displaying, the image to be displayed can be generated by applying appropriate weights and gain maps to the SDR image according to the capabilities of the display device. If the display device does not support HDR, the gain map is ignored (the weights are set to 0) and the SDR image is displayed.

[0017] In this embodiment, two signal characteristics are assumed to represent the relationship between the video signal level and display brightness in an HDR image: PQ (Perceptual Quantization) and HLG (Hybrid Log Gamma). PQ is specified as an EOTF (Electro-Optical Transfer Function) in SMPTE ST 2084, and HLG is specified as an OETF (Optical to Electrical Transfer Function) in ARIB STD-B67.

[0018] An SDR image is an image with a narrower output range (dynamic range) than an HDR image. In the following explanation, the gamma of an HDR image is assumed to be the OETF characteristic (PQ Inverse EOTF) based on SMPTE ST 2084, and the color gamut is assumed to conform to the ITU-R BT. 2020 standard. The gamma of an SDR image is assumed to be sRGB gamma, and the color gamut is assumed to be sRGB.

[0019] The optical system 101 includes a lens, a shutter, an aperture, and a mechanism (such as a motor) for driving these. The optical system 101 may be integrated with the imaging device 100, or may have the form of an interchangeable lens. The optical system 101 forms an optical image of a subject on the imaging surface of the imaging element 102.

[0020] The image sensor 102 may be, for example, a known CCD or CMOS color image sensor with a primary-color Bayer array color filter. The image sensor 102 has a pixel array in which multiple pixels are arranged two-dimensionally, and peripheral circuits for reading out signals from each pixel. Each pixel accumulates charge according to the amount of incident light through photoelectric conversion. By reading out signals from each pixel having voltages according to the amount of charge accumulated during the exposure period, a group of pixel signals (analog image signals) representing the optical image formed on the imaging surface is obtained.

[0021] The analog image signal is A / D converted to a digital image signal (image data) and then stored in the primary storage device 104. The A / D conversion may be performed by the image sensor 102, or may be performed by another component (for example, an A / D converter not shown) outside the image sensor 102. Image data stored in the primary storage device 104 at this stage, which is composed of pixel data having one color component of R, G, or B, is called RAW image data.

[0022] The CPU 103 determines the imaging conditions (shutter speed or exposure time, aperture value, and imaging sensitivity).

[0023] An optical image formed by the optical system 101 is projected onto the photometric sensor 105. The photometric sensor 105 measures the luminance information of the optical image for each of a plurality of photometric regions (for example, 12 horizontally and 8 vertically, for a total of 96 regions). The photometric sensor 105 outputs the measurement results to the CPU 103. Note that the number, positions, and sizes of the photometric regions may change depending on the settings of the imaging device 100, etc.

[0024] The CPU 103 loads a program stored in the secondary storage device 109 into the primary storage device 104 and executes it, thereby controlling the operation of each unit of the imaging device 100 and realizing various functions of the imaging device 100. Note that at least some of the functions described as being realized by the CPU 103 executing a program may be executed by hardware such as an ASIC or FPGA configured to execute the function.

[0025] The primary storage device 104 is a volatile storage device such as a RAM, and is used to temporarily store various data such as programs executed by the CPU 103, data required for executing the programs, data before, during, and after processing by the image processing unit 107, image data for display, and image data for recording.

[0026] The secondary storage device 109 is a non-volatile storage device such as an EEPROM, and stores programs executed by the CPU 103 to control the image capture device 100, various settings, information about the image capture device 100, GUI data, and the like.

[0027] The recording medium 108 is used as a recording destination for the image data for recording stored in the primary storage device 104. The recording medium 108 is removable from the imaging device 100 and may be, for example, a semiconductor memory card. The image data recorded on the recording medium 108 can be read by other devices such as a personal computer. The imaging device 100 has a mechanism for attaching and detaching the recording medium 108 and a read / write function.

[0028] The image processing unit 107 applies predetermined image processing to image data stored in the primary storage device 104, such as after being read from the image sensor 102 or the recording medium 108, to acquire and / or generate signals, image data, and information according to the intended use. The image processing unit 107 may be a dedicated hardware circuit, such as an ASIC (Application Specific Integrated Circuit) designed to achieve a specific function. Alternatively, the image processing unit 107 may be configured such that a processor, such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit), executes software to achieve a specific function. The image processing unit 107 outputs the acquired or generated signals, information, and data to the CPU 103 or stores them in the primary storage device 104 according to the intended use.

[0029] The image processing applied by the image processing unit 107 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when a color filter is provided on the image sensor 102, and is a process of interpolating the values ​​of color components that are not included in the individual pixel data that make up the image data. Color interpolation processing is also called demosaic processing. The correction processing may include white balance adjustment, tone correction (gamma), correction of image degradation caused by optical aberrations of the optical system 101 (image restoration), correction of the effects of peripheral light falloff of the optical system 101, color correction, and the like. The detection process may include detection of characteristic regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. Data processing can include processes such as area extraction (trimming), compositing, scaling (reducing and enlarging), encoding and decoding, and header information generation (data file generation). Data processing also includes the generation of image data for display or recording, and conversion of color gamut or dynamic range. Data processing also includes the generation of gain maps and the application of gain maps. The evaluation value calculation process can include processes such as generating signals and evaluation values ​​used in autofocus (AF) detection, and generating evaluation values ​​used in automatic exposure control (AE). Special effect processing can include adding a blur effect, changing color tones, relighting, and the like. It should be noted that these are examples of processes that the image processing unit 107 can apply, and do not limit the processes that the image processing unit 107 can apply.

[0030] The display unit 110 is, for example, a liquid crystal display provided on the surface of the housing of the imaging device 100. The display unit 110 displays information about the imaging device 100, a menu screen, images captured by the image sensor 102, images read from the recording medium 108, and the like. Note that the imaging device 100 can cause the display unit 110 to function as an electronic viewfinder (EVF) by continuously capturing video and displaying the video on the display unit 110 while in a shooting standby state. The operation of causing the display unit 110 to function as an EVF is called live view display, and the image used for the live view display is called a live view image.

[0031] The operation unit 111 is a collective term for input devices (such as buttons, switches, and dials) provided for the user to input various instructions to the imaging device 100. The input devices constituting the operation unit 111 are named according to the assigned functions. For example, the operation unit 111 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, and a confirmation key. The release switch is a switch for recording still images, and the CPU 103 recognizes a half-pressed state of the release switch as an instruction to prepare for shooting and a full-pressed state as an instruction to start shooting. The CPU 103 also recognizes a press of the video recording switch in a shooting standby state as an instruction to start video recording, and a press of the video recording switch during video recording as an instruction to stop recording. Note that the functions assigned to the same input device may be variable. The input device may also be software buttons or keys using a touch display. The operation unit 111 may also include an input device compatible with non-contact input methods such as voice input or eye-gaze input.

[0032] In this embodiment, an operation mode for capturing a specific scene can be set in the imaging device 100. Typical examples of the specific scene include, but are not limited to, a portrait, a sunset, a night view, fireworks, a starry sky, and cooking.

[0033] The external communication unit 106 is a communication interface between an external device and the imaging device 100. The external communication unit 106 supports one or more wired and / or wireless communication standards. The external communication unit 106 has a transmission / reception circuit according to the supported standard, and an associated connector or antenna. Typical examples of standards supported by the external communication unit 106 include, but are not limited to, USB, wireless LAN, Bluetooth (registered trademark), and HDMI (registered trademark).

[0034] The CPU 103 transmits image data to an external device via the external communication unit 106, acquires various information from the external device, controls the operation of the external device, and receives commands from the external device that control the operation of the imaging device 100.

[0035] (Gain map) Here, the gain map used in this embodiment will be described. The gain map may be generated from an SDR image and an HDR image in which the color space and gain are combined, using, for example, the method described in Non-Patent Document 1.

[0036] The gain map G is defined by the following equation: G=log2((HDR image + k hdr ) / (SDR image + k sdr )) k hdr and k sdr is an offset coefficient that is determined so that the value in the parentheses for which the logarithm is calculated is positive. A gain map is generated by calculating the gain for each pixel in the HDR image and SDR image that corresponds to each other using the above formula.

[0037] The gain map may be generated for each component (channel) of RGB or YUV, or may be generated for one component such as the luminance component (grayscale). The luminance component may be, for example, the Y component of YUV or the G component of RGB.

[0038] The gain that makes up the gain map has a minimum value of 0 and a maximum value of 2. N The gain ranges from -1 to 1, where N is the number of encoding bits. Therefore, when encoding with 8 bits, the maximum gain value is 255. A gain of 0 has no effect on brightness. A negative gain means darkening, and a positive gain means brightening.

[0039] The gain map also has metadata, which may include, but is not limited to, the number of gain maps (number of channels), the type of base image, the minimum and maximum gain values, an offset coefficient, the size of the gain map, information on when the image was captured (such as the capture mode and scene information), and the reduction rate of the gain map.

[0040] In this embodiment, it is assumed that base image data (SDR image data or HDR image data) is recorded together with reduced gain map data and gain map metadata. The reduced gain map has a lower resolution than the base image. In other words, the number of gain data included in the gain map is smaller than the number of pixels in the base image. The gain map may or may not be generated by the image processing unit 107. It is also assumed that the gain map metadata includes the reduction rate of the gain map. Here, the reduction rate (%) indicates the proportion of the reduced size when the size before reduction is 100%. Therefore, a reduction rate of 90% means that the resolution is reduced to 90% when the resolution before reduction is 100% (reducing the resolution by 10%).

[0041] 2 is a diagram showing, as functional blocks, the process steps for applying a reduced gain map to a base image, which are executed by the image processing unit 107. Therefore, the functional blocks 201 to 205 shown in FIG. 2 are actually realized by the image processing unit 107 using its own hardware and software. Note that some of the processes may be executed by a component other than the image processing unit 107 (for example, the CPU 103).

[0042] The operation of the image processing unit 107 will be described with reference to Fig. 2 and the flowchart shown in Fig. 3. Here, it is assumed that the data of the base image to be processed (SDR image data or HDR image data), the data of the reduced gain map, and the metadata of the gain map are stored in the primary storage device 104.

[0043] In S301, the mask generation unit 201 detects specific regions from the base image data that may change in size or decrease in value due to a combination of reduction and enlargement processes. Specific regions are regions such as point light source regions and edge regions, where the difference in brightness between adjacent pixels is large and the number of pixels in one or more directions is small (for example, about one to several pixels). Such regions may become darker than their original brightness due to a decrease in contrast, or may be enlarged beyond their original size when enlarged.

[0044] The mask generation unit 201 detects point light sources and edge regions by applying an edge detection or emphasis filter, such as a Laplacian filter, to the base image data. Typical examples of point light sources and edge regions include stars, small light sources, and thin objects with high contrast with the background. There are no particular limitations on the method for detecting point light sources and edge regions from the base image, and any other known method can be used.

[0045] Then, based on the detected region, the mask generation unit 201 generates mask image data to be used for sharpness processing, which will be described later. The mask image is an image that defines the region to which sharpness processing is applied. Specifically, the mask image is made up of pixels with values ​​ranging from 0 to 1. When the mask image is applied to a target image, sharpness processing is applied only to pixels in the target image at positions corresponding to pixels with a value of 1 in the mask image.

[0046] In this embodiment, sharpness processing is applied to a gain map that has been restored to the original resolution (= the resolution of the base image). Therefore, a mask image can be generated at the same resolution as the base image. The mask generation unit 201 can generate a mask image in which, for example, the pixel values ​​of the detected region and its surrounding region are set to 0, and the pixel values ​​of the other regions are set to 1. The surrounding region may be a region that circumscribes the detected region and has a width (thickness) of a predetermined number of pixels (for example, 1 to several pixels). The surrounding region may also be determined based on the range used in the calculation of sharpness processing. For example, when sharpness processing is applied using a two-dimensional spatial filter, mask image data can be generated so that the region where the pixel value is 0 becomes a rectangular region having a size equal to or larger than the filter size.

[0047] For example, consider a case where a point light source area of ​​one pixel is detected and a two-dimensional spatial filter of 3 × 3 pixels is used for sharpness processing. In this case, the mask generation unit 201 generates a mask image in which the value of the 3 × 3 pixels centered on the point light source area is set to 0 and the surrounding pixels are set to 1, so that the 3 × 3 pixels are the target of sharpness processing. The size of the entire mask image can be, for example, the size of a rectangular area circumscribing the area to be processed, surrounded by a frame-shaped pixel area of ​​a predetermined width.

[0048] In S302, the map information acquisition unit 202 acquires metadata of the gain map. As described above, the metadata includes the reduction ratio of the gain map. The map information acquisition unit 202 stores the acquired reduction ratio in the primary storage device 104.

[0049] In S303, the resizing unit 203 applies enlargement processing to the gain map based on the reduction ratio acquired in S302 so that the resolution is the same as before reduction. The resolution of the gain map before reduction is usually equal to the resolution of the base image. Therefore, instead of using the reduction ratio, enlargement processing may be applied to the gain map so that the resolution is the same as the base image. The resizing unit 203 stores the enlarged gain map in the primary storage device 104.

[0050] In S304, the sharpness processing unit 204 (correction means) performs sharpness processing on the gain map enlarged in S303 using the mask image generated in S301. The sharpness processing corresponds to a process of correcting a decrease in contrast (decrease in gain value) in the gain map caused by the combination of the reduction processing and the enlargement processing, and bringing the value closer to the gain map value before reduction. Specifically, the sharpness processing unit 204 applies sharpness processing to areas of the mask image where the pixel value is 0. The sharpness processing unit 204 does not apply sharpness processing to areas of the mask image where the pixel value is 1 and areas outside the mask image.

[0051] If a mask image is not used, the sharpness processor 204 applies sharpness processing to the entire image. In this case, the amount of noise increases in areas that do not require sharpness processing. Therefore, if the increase in noise due to sharpness processing is not a problem, there is no need to generate and use a mask image.

[0052] The sharpness processing unit 204 can vary the strength of the sharpness processing depending on the reduction ratio of the gain map. As shown in Fig. 4, for example, the sharpness processing unit 204 can control the strength of the sharpness processing so that the greater the reduction ratio (closer to 100%), the weaker the strength of the sharpness processing, and the smaller the reduction ratio, the stronger the strength of the sharpness processing. This is the same as controlling the strength of the sharpness processing so that the greater the difference between the resolution of the gain map and the resolution of the base image, the stronger the strength of the sharpness processing. The smaller the reduction ratio, the greater the difference between the enlarged gain map and the gain map before reduction, so the strength of the sharpness processing (correction strength) is increased.

[0053] The strength of the sharpness processing can be controlled based on the shooting sensitivity (ISO sensitivity) of the base image, the shape of the gamma applied to the base image, and the magnitude of the variance value (amount of noise) in the flat part of the base image, instead of or in addition to the reduction ratio. Basically, the strength of the sharpness processing can be controlled so that the greater the amount of noise, the stronger the sharpness processing. Furthermore, the relationship between the reduction ratio and the strength of the sharpness processing does not have to be linear. Furthermore, parameters related to the sharpness processing may be user-configurable.

[0054] The sharpness processor 204 stores the enlarged gain map to which the sharpness processing has been applied in the primary storage device 104 .

[0055] In S305, the gain map application unit 205 applies the gain map to which the sharpness processing has been applied in S304 to the base image. The gain map may be applied to the base image as is, or may be applied to the base image with a weighting according to the capabilities of the display device, etc. This embodiment does not depend on the method of applying the gain map.

[0056] As described above, according to this embodiment, when the gain map recorded together with the base image is reduced, the change in the gain value before reduction caused by the reduction process and the enlargement process during use is corrected. Therefore, even when a reduced gain map is recorded, it is possible to achieve an expression close to that obtained when the gain map before reduction is applied.

[0057] (Variation) In this embodiment, a specific region whose size or value may change due to a combination of reduction and enlargement processes is detected from the base image data. However, a specific region may also be detected from a reduced gain map or an enlarged gain map. When a specific region is detected from a gain map, the detection accuracy of the specific region is lower than when the specific region is detected from the base image data. However, it is possible to obtain the effect of improving the decrease in gain value that occurs when a reduced gain map is simply enlarged and applied.

[0058] <Second embodiment> Next, a second embodiment of the present invention will be described. This embodiment differs from the first embodiment in the operation of the image processing unit 107 for applying the gain map. Since the other configurations and operations of the imaging device 100 are the same as those of the first embodiment, the following description will focus on the differences from the first embodiment.

[0059] 5 is a diagram showing, as functional blocks, the processing steps for applying a reduced gain map to a base image, which are executed by the image processing unit 107 in this embodiment. Since the functions of the functional blocks are the same as those in the first embodiment, the same reference numerals as those in FIG. 2 are used.

[0060] In this embodiment, the sharpness processing is not applied to the gain map after enlargement, but is applied to the image data obtained by applying the gain map after enlargement.

[0061] The operation of the image processing unit 107 in this embodiment will be described using Fig. 5 and the flowchart shown in Fig. 6. Here, it is assumed that data of the base image to be processed (SDL image data or HDL image data), data of the reduced gain map, and metadata of the gain map are stored in the primary storage device 104. In Fig. 6, the same reference numerals as in Fig. 3 are used to designate steps that perform the same operations as in the first embodiment.

[0062] Steps S301 to S303 are the same as those in the first embodiment, and therefore their explanation will be omitted. In this embodiment, sharpness processing is applied to image data after the gain map has been applied, so step S305 is executed after step S303. Step S305 is the same processing as in the first embodiment, except that the gain map used is not subjected to sharpness processing.

[0063] In S601, the sharpness processing unit 204 (correction means) performs sharpness processing on the image data generated in S305 using the mask image generated in S301. The sharpness processing corresponds to a process of correcting the influence on the image data of a decrease in contrast (decrease in gain value) of the gain map caused by the combination of the reduction processing and the enlargement processing, and bringing the image data closer to the value when the gain map before reduction is applied. The content of the sharpness processing may be the same as in the first embodiment.

[0064] This embodiment can also provide the same effects as those of the first embodiment. Also in this embodiment, a specific region may be detected from a reduced gain map or an enlarged gain map, as in the modified example of the first embodiment.

[0065] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0066] The disclosure of the present embodiment includes the following image processing device, image processing method, and program. (Item 1) an acquisition means for acquiring a gain map that can be applied to one of SDR image data and HDR image data based on the same image to generate the other, and base image data that is one of the SDR image data and the HDR image data; an enlarging means for enlarging the gain map so that the resolution of the gain map becomes equal to the resolution of the image represented by the base image data; applying means for applying the expanded gain map to the base image data to obtain image data to which the expanded gain map has been applied; a detection means for detecting an area whose size or value changes depending on a combination of reduction processing and enlargement processing from the base image data, the gain map acquired by the acquisition means, or the enlarged gain map; a correction unit that applies sharpness processing to an area of ​​the enlarged gain map or image data to which the enlarged gain map has been applied, the area corresponding to the area detected by the detection unit; When the correction means applies the sharpness processing to the enlarged gain map, the correction means applies the sharpness processing before the enlarged gain map is applied to the base image data. 1. An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the detection means detects at least one of a point light source area and an edge area as the area. (Item 3) the detection means generates a mask image for defining a range to which the sharpness processing is to be applied with respect to the detected region; the correction means applies the sharpness processing to a range defined by the mask image. 3. The image processing device according to item 1 or 2, characterized in that: (Item 4) 4. The image processing device according to item 3, wherein the size of the range to which the sharpness processing is applied is based on the filter size used in the sharpness processing. (Item 5) The acquisition means further acquires metadata of the gain map; 5. The image processing device according to any one of items 1 to 4, wherein the enlargement means enlarges the gain map based on a reduction ratio included in the metadata. (Item 6) 6. The image processing device according to any one of items 1 to 5, characterized in that the correction means controls the intensity of the sharpness processing based on one or more of the reduction ratio of the gain map, the shooting sensitivity of the base image data, the shape of the gamma applied to the base image data, and the magnitude of the variance value in the flat part of the base image data. (Item 7) 6. The image processing device according to any one of items 1 to 5, characterized in that the correction means controls the strength of the sharpness processing to be stronger as the difference between the resolution of the gain map and the resolution of the base image data becomes greater. (Item 8) An image processing method executed by an image processing device, Obtaining a gain map that can be applied to one of SDR image data and HDR image data based on the same image to generate the other, and base image data that is one of the SDR image data and the HDR image data; scaling up the gain map so that the resolution of the gain map is equal to the resolution of the image represented by the base image data; Detecting an area whose size or value changes due to a combination of a reduction process and an enlargement process from the base image data, the gain map, or the enlarged gain map; applying sharpening to an area of ​​the expanded gain map that corresponds to the detected area; applying the expanded gain map to the base image data to obtain expanded gain mapped image data; An image processing method comprising: (Item 9) An image processing method executed by an image processing device, Obtaining a gain map that can be applied to one of SDR image data and HDR image data based on the same image to generate the other, and base image data that is one of the SDR image data and the HDR image data; scaling up the gain map so that the resolution of the gain map is equal to the resolution of the image represented by the base image data; Detecting an area whose size or value changes due to a combination of a reduction process and an enlargement process from the base image data, the gain map, or the enlarged gain map; applying the expanded gain map to the base image data to obtain expanded gain mapped image data; applying sharpness processing to a region of the image data to which the enlarged gain map has been applied, the region corresponding to the detected region; An image processing method comprising: (Item 10) 10. A program for causing a computer to execute each step of the image processing method according to item 8 or 9.

[0067] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention. [Explanation of symbols]

[0068] 100...imaging device, 101...optical system, 102...imaging element, 103...CPU, 104...primary storage device, 107...image processing device, 108...storage medium, 109...secondary storage device, 110...display unit, 111...operation unit

Claims

1. an acquisition means for acquiring a gain map that can be applied to one of SDR image data and HDR image data based on the same image to generate the other, and base image data that is one of the SDR image data and the HDR image data; an enlarging means for enlarging the gain map so that the resolution of the gain map becomes equal to the resolution of the image represented by the base image data; applying means for applying the expanded gain map to the base image data to obtain image data to which the expanded gain map has been applied; a detection means for detecting an area whose size or value changes depending on a combination of reduction processing and enlargement processing from the base image data, the gain map acquired by the acquisition means, or the enlarged gain map; a correction unit that applies sharpness processing to an area of ​​the enlarged gain map or image data to which the enlarged gain map has been applied, the area corresponding to the area detected by the detection unit; When the correction means applies the sharpness processing to the enlarged gain map, the correction means applies the sharpness processing before the enlarged gain map is applied to the base image data.

1. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein said detecting means detects at least one of a point light source area and an edge area as said area.

3. the detection means generates a mask image for the detected region that defines a range to which the sharpness processing is to be applied; the correction means applies the sharpness processing to a range defined by the mask image.

2. The image processing device according to claim 1, wherein:

4. 4. The image processing device according to claim 3, wherein the size of the range to which the sharpness processing is applied is based on a filter size used in the sharpness processing.

5. The acquisition means further acquires metadata of the gain map; 2. The image processing apparatus according to claim 1, wherein the enlarging means enlarges the gain map based on a reduction ratio included in the metadata.

6. The image processing device according to claim 1, characterized in that the correction means controls the intensity of the sharpness processing based on one or more of the reduction ratio of the gain map, the shooting sensitivity of the base image data, the shape of the gamma applied to the base image data, and the magnitude of the variance value in the flat portion of the base image data.

7. 2. The image processing device according to claim 1, wherein the correction means controls the sharpness processing so that the greater the difference between the resolution of the gain map and the resolution of the base image data, the stronger the intensity of the sharpness processing.

8. An image processing method executed by an image processing device, Obtaining a gain map that can be applied to one of SDR image data and HDR image data based on the same image to generate the other, and base image data that is one of the SDR image data and the HDR image data; scaling up the gain map so that the resolution of the gain map is equal to the resolution of the image represented by the base image data; Detecting an area whose size or value changes due to a combination of a reduction process and an enlargement process from the base image data, the gain map, or the enlarged gain map; applying sharpening to an area of ​​the expanded gain map that corresponds to the detected area; applying the expanded gain map to the base image data to obtain expanded gain mapped image data; An image processing method comprising:

9. An image processing method executed by an image processing device, Obtaining a gain map that can be applied to one of SDR image data and HDR image data based on the same image to generate the other, and base image data that is one of the SDR image data and the HDR image data; scaling up the gain map so that the resolution of the gain map is equal to the resolution of the image represented by the base image data; Detecting an area whose size or value changes due to a combination of a reduction process and an enlargement process from the base image data, the gain map, or the enlarged gain map; applying the expanded gain map to the base image data to obtain expanded gain mapped image data; applying sharpness processing to a region of the image data to which the enlarged gain map has been applied, the region corresponding to the detected region; An image processing method comprising:

10. A program for causing a computer to execute each step of the image processing method according to claim 8 or 9.

Images