Image processing device, image processing method, and program

Abstract

To enable desirable expression of specific color areas while maintaining the overall tonal quality of an image.SOLUTION: An image processing device includes decision means (203, 211) for deciding whether to perform a correction and correction means (203, 211) for making a specific frequency band a target of correction on the basis of a decision by the decision means. The correction means (203, 211) targets for correction a luminance range in which a slope of a tonal characteristic of a predetermined chromatic color is greater than that of a tonal characteristic of an achromatic color.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing technique for performing gradation processing and the like on a captured image. [Background technology]

[0002] In recent years, high-definition technology has improved elements such as brightness range, resolution, and color gamut, making it possible to reproduce the colors and details of objects more faithfully. However, in terms of the expression of human skin in photographed images, since women and children in particular tend to prefer uniform skin with few shadows and unevenness, the image expression that accompanies such high image quality may not be desirable as a picture. In particular, HDR (High Dynamic Range), a technology that expands the brightness range, has gradation characteristics with a higher peak brightness as the upper limit value than the conventional SDR (Standard Dynamic Range), and therefore the gradation on the high brightness side is significantly improved. As a result, the shadows and unevenness that are considered undesirable for human skin as mentioned above tend to be more pronounced.

[0003] Patent Document 1 discloses a method for naturally reducing skin noise such as blemishes, freckles, and acne while preserving fine irregularities such as skin texture. Patent Document 2 discloses a method for reducing facial shine. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-111595 A [Patent Document 2] JP 2005-327009 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, there are also disadvantages when correcting a specific color region. For example, if the contrast of an image is lowered in order to reduce the undesirable phenomenon occurring in an image of human skin as described above, the contrast of the entire image, including areas other than the skin, will decrease, and especially in the case of an HDR image, the high gradation will be lost. Even if the correction target is limited to the face area, the entire face may have a flat and unnatural impression. On the other hand, contrary to the case of human skin, there may be cases where more gradation is desired. For example, wood grain and leaf veins often appear dark and tend to be difficult to bring out details, but if the contrast is adjusted to match these subjects, there is a problem that the contrast will be too strong in other areas. In the case of the technology disclosed in Patent Document 1, the method is limited to the specific skin noise (noise containing red components) described above, and cannot handle the shadows and unevenness of the skin. In addition, in the case of the technology disclosed in Patent Document 2, the correction process is limited to the high luminance range, so it cannot handle other luminance ranges.

[0006] Therefore, an object of the present invention is to enable preferable expression of a specific color region while maintaining the gradation of the entire image. [Means for solving the problem]

[0007] The image processing apparatus of the present invention includes a determination unit for determining whether to perform correction, and a step of determining whether to perform correction based on the determination by the determination unit. The closest color to the subject is skin color. and a correction means for correcting a specific frequency band, the correction means being characterized in that the correction means corrects a luminance range in which the gradient of the gradation characteristics of flesh color is greater than the gradient of the gradation characteristics of achromatic colors. Effect of the Invention

[0008] According to the present invention, it is possible to preferably express a specific color region while maintaining the gradation of the entire image. [Brief description of the drawings]

[0009] [Figure 1]FIG. 1 is a diagram illustrating the configuration of a digital camera to which an image processing device according to an embodiment of the present invention is applied. [Diagram 2] 1 is a block diagram showing an example of a functional configuration of an image processing device; [Diagram 3] 4 is a flowchart of image processing according to the present embodiment. [Figure 4] 4A to 4C are diagrams used to explain gradation characteristics and slope changes in the first embodiment. [Diagram 5] FIG. 2 is a block diagram showing a functional configuration for performing local contrast correction. [Figure 6] 13 is a flowchart of a local gain map generation process. [Figure 7] 1A and 1B are diagrams used to explain a mechanism of local contrast correction. [Figure 8] 13A to 13C are diagrams used to explain gradation characteristics and slope changes in the second embodiment. [Figure 9] FIG. 1 is a diagram showing the EOTF characteristics of PQ in SMPTE ST2084. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings. The same configurations or processes will be described with the same reference symbols. 1 is a diagram showing a schematic configuration example of an imaging device (hereinafter, referred to as a digital camera or simply a camera) which is an application example of the image processing device of this embodiment. The digital camera of this embodiment is a camera with interchangeable lenses. The camera body has a mount part (not shown) that allows an interchangeable lens to be attached and detached, and the camera body and the interchangeable lens can be electrically connected via a contact part 111 provided on the mount part and a contact part 113 on the interchangeable lens side.

[0011] The control unit 101 on the camera body side is a one-chip microcomputer that incorporates an ALU (ARITHMETIC and Logic Unit), ROM, RAM, an A / D converter, a timer, and a serial communication port (SPI). The control unit 101 controls the operation of the camera body and the interchangeable lens by executing a program stored in, for example, the ROM. The specific operation of the control unit 101 will be described later.

[0012] An output signal from an AF sensor 102 for focus detection and an output signal from an AE sensor 103 for exposure setting are sent to an A / D converter input terminal of a control unit 101 . The signal processing circuit 104 controls the image sensor 105 according to instructions from the control unit 101, and applies A / D conversion and image signal processing to the image signal output by the image sensor 105 to obtain image data. The signal processing circuit 104 also performs necessary image processing such as compression and synthesis when recording the obtained image data. The function and operation of the signal processing circuit 104 will be described in detail later.

[0013] The memory 106 is a DRAM or the like, and is used as a work memory when the signal processing circuit 104 performs various signal processes, and as a VRAM when displaying images on a display device 107 (described later). The storage unit 108 is a non-volatile memory such as a flash memory, and receives image data obtained by image signal processing in the signal processing circuit 104 .

[0014] The display unit 107 is intended to be connected to a rear liquid crystal display or an external display using standards such as HDMI (registered trademark), and displays camera setting values, messages, information such as menu screens, and captured images. The display unit 107 is controlled by instructions from the control unit 101.

[0015] A motor 109, under the control of the control unit 101, moves the mirror inside the camera body up and down and charges the mechanical shutter. The operation unit 110 is a group of input devices such as switches used by the user to operate the camera. The operation unit 110 includes a release switch for instructing the start of preparations for shooting and the start of shooting, a shooting mode selection switch for selecting a shooting mode, direction keys, an enter key, and the like. The contact section 111 is a contact for communicating with an interchangeable lens, and input / output signals of a serial communication port of the control section 101 are connected thereto. The shutter driver 112 is connected to an output terminal of the controller 101 and drives the mechanical shutter.

[0016] The interchangeable lens is provided with a contact section 113 that forms a pair with the contact section 111 on the camera body side. A lens control section 114, which is a one-chip microcomputer similar to the control section 101, is connected to the contact section 113, thereby enabling communication between the lens control section 114 and the control section 101. The lens control unit 114 executes a program stored in, for example, a ROM, and controls the operation of the interchangeable lens based on instructions from the control unit 101. The lens control unit 114 also notifies the control unit 101 of the camera body of information such as the state of the interchangeable lens.

[0017] The focus lens driver 115 is connected to the output terminal of the lens controller 114 and drives the focus lens. A zoom driver 116 drives the zoom lens under the control of the lens controller 114 to change the angle of view when photographing with the digital camera. The aperture driver 117 adjusts the aperture opening size under the control of the lens controller 114 .

[0018] When the interchangeable lens is attached to the camera body, data communication is possible between the lens control unit 114 and the control unit 101 of the camera body via the contact unit 111 and the contact unit 113. In addition, power for driving the motors and actuators in the interchangeable lens is also supplied from the camera body via the contact unit 111 and the contact unit 113. Lens-specific optical information required for the control unit 101 of the camera body to perform focus detection and exposure calculation, information on the subject distance based on a distance encoder, and the like are sent from the lens control unit 114 of the interchangeable lens to the control unit 101 of the camera body by data communication. In addition, focus adjustment information and aperture information obtained as a result of the control unit 101 of the camera body performing focus detection and exposure calculation are sent from the control unit 101 of the camera body to the lens control unit 114 of the interchangeable lens by data communication. The lens control unit 114 of the interchangeable lens drives the focus lens according to the focus adjustment information, and drives the aperture according to the aperture information.

[0019] FIG. 2 is a functional block diagram showing the image processing performed by the image processing device of this embodiment by function. In this embodiment, the signal processing circuit 104 of FIG. 1 performs image processing corresponding to each functional unit shown in FIG. 2. The image processing by each functional unit may be realized by a hardware configuration such as a circuit, or may be realized by a CPU or the like executing a program related to the image processing of this embodiment. The image processing program may be prepared in advance in a ROM, or may be acquired via a removable memory (not shown) or a communication unit (not shown). In this case, the CPU performs the image processing according to this embodiment by executing the image processing program. The control unit 101 may be responsible for executing the image processing program according to this embodiment.

[0020] The RAW image 201 is RAW image data obtained by A / D converting the image signal of the image sensor 105, and is an image in which each pixel is composed of pixel values ​​(intensities) corresponding to a single color, R (red), G (green), or B (blue). An image composed of pixel values ​​corresponding to a single color is also called a color plane.

[0021] A white balance correction unit (hereinafter, WB correction unit 202) performs white balance correction processing on the RAW image 201 to correct color casts caused by a light source and reproduce white. Specifically, the WB correction unit 202 first plots the RGB data of each pixel constituting the RAW image 201 in a predetermined color space, such as an xy color space, and integrates the RGB data of data plotted near a blackbody radiation locus that is likely to be the light source color in the color space. Furthermore, the WB correction unit 202 obtains white balance coefficients G / R and G / B of the R and B components from the integral value. The WB correction unit 202 then performs white balance correction processing using the white balance coefficients generated by the above processing.

[0022] The luminance range selection unit 211 acquires the shooting conditions 210 when an image is shot, and determines what shooting mode the digital camera of this embodiment is in. There are multiple shooting modes in a digital camera, and the shooting modes to be determined in this embodiment include whether the shooting mode is SDR shooting or HDR shooting, or whether the shooting mode is one in which a function that prioritizes gradation on the high luminance side is ON or OFF. Then, the luminance range selection unit 211 performs a luminance range setting process to set a luminance range to be corrected according to the shooting mode determined from the shooting conditions 210. The details of the luminance range setting process based on the shooting conditions 210 in the luminance range selection unit 211 will be described later.

[0023] The luminance tone correction unit 203 performs tone correction processing on the RAW image after the white balance correction processing. In the present embodiment, the luminance tone correction unit 203 performs tone correction processing such as raising the brightness of a region of a specific subject, such as a face, that has become dark in the image. For example, the luminance tone correction unit 203 judges the presence or absence of a subject, such as a face, to be corrected, or the presence or absence of a pixel having color information similar to that of a subject, such as a face, to be corrected, for the input image, and determines whether or not to perform tone correction based on the judgment result. That is, the luminance tone correction unit 203 determines whether or not to activate tone correction processing based on the presence or absence of a subject to be corrected. Then, when gradation correction is activated, the luminance tone correction unit 203 sets the luminance range set by the luminance range selection unit 211 as the luminance range to be corrected, and further sets a frequency band corresponding to the correction target, such as a face, as the frequency band to be corrected, and executes tone correction limited to those regions. In this manner, in this embodiment, the luminance tone correction unit 203 performs tone correction limited to the luminance range to be corrected and the frequency range to be corrected, thereby making it possible to obtain an image with favorable skin expression, etc., while maintaining the gradation of the entire image. The details of the correction target frequency band setting process in the luminance tone correction unit 203 and the tone correction process limited to the correction target frequency band and the correction target luminance range will be described later.

[0024] A color interpolation unit 204 performs noise reduction processing and pixel interpolation processing on the image after the above-mentioned luminance tone correction processing, and generates an image in which R, G, and B color information is complete for all pixels.

[0025] The development processing unit 205 performs development processing on an image in which R, G, and B color information is completed for all pixels by the color interpolation unit 204, to generate a color image. The development processing unit 205 performs processing such as matrix conversion and gamma conversion. In the gamma conversion processing, in the case of development processing compatible with SDR (hereinafter referred to as SDR development), the gamma characteristic of sRGB is mainly used. On the other hand, in the case of development processing compatible with HDR (hereinafter referred to as HDR development), for example, the inverse characteristic of EOTF (Electro-Optical Transfer Function) of PQ (Perceptual Quantization) is used as the gamma characteristic. Note that the gamma characteristic in HDR development may be a combination of EOTF and OOTF (Opto-Optical Transfer Function) characteristics. FIG. 9 shows the EOTF characteristic of PQ standardized in SMPTE ST2084 (Society Motion Picture and Television Engineers ST2084).

[0026] The color luminance adjustment unit 206 performs color luminance adjustment processing to further improve the appearance of the color image after the development processing by the development processing unit 205. For example, the color luminance adjustment unit 206 performs image correction processing to increase the brightness in the case of portrait photography, and performs image correction processing to emphasize the saturation of green and blue sky in the case of landscape photography. These image correction processes are performed by applying a lookup table (LUT) for color luminance adjustment to color signal values ​​such as RGB.

[0027] The compression unit 207 compresses the high-resolution image after the color and luminance adjustment processing in the color and luminance adjustment unit 206 using a method such as HEVC (High Efficiency Video Codec). Thereafter, a recording unit 208 generates image data (developed image 209) to be recorded on a recording medium such as a flash memory.

[0028] In this embodiment, as described above, the luminance tone correction unit 203 performs tone correction processing by limiting the luminance range and frequency band to be corrected, thereby making it possible to realize preferable skin expression, etc., while maintaining a certain degree of the gradation of the entire image. In the functional block diagram of Fig. 2, the luminance tone correction unit 203 and the luminance range selection unit 211 are listed as separate functional units, but it is also possible to provide a determination and correction unit that combines these two functional units into one. In this case, the output of the WB correction unit 202 and the shooting conditions 210 are input to the determination and correction unit, which performs a determination process to determine whether or not to perform correction, and a gradation correction process to set the correction target frequency and correction target luminance range as the correction targets, respectively, based on the decision. The output of the determination and correction unit is then sent to the color interpolation unit 204. Here, examples of the correction target luminance range and whether or not to execute (invoke) luminance correction processing for the correction target luminance range include two methods as described in the first and second embodiments below.

[0029] <First embodiment> First, the luminance range to be corrected in the first embodiment will be described. In the first embodiment, a skin image correction method that can be applied in the same manner not only in HDR development but also in SDR development will be described. Image processing in the first embodiment will be described in detail below with reference to the functional blocks shown in FIG. 2 and the flowchart shown in FIG.

[0030] First, in step S301, the luminance tone correction unit 203 uses a face detection method to determine whether or not a human face area is included in the input image. The face detection method is a known method. For example, a face detection method using learning represented by a neural network, and a face detection method that extracts a part having a characteristic physical shape such as eyes and a nose from a captured image by a template matching process or the like are known. In addition, a face detection method that detects a face by detecting image features such as skin color and eye shape from an input image and performing an analysis using a statistical method is known. When it is determined that a face area is included in the input image by these methods, the luminance tone correction unit 203 decides to perform the luminance tone correction process in this embodiment (i.e., decides to activate the luminance tone correction process).

[0031] Next, in step S302, the luminance range selection unit 211 acquires the shooting conditions 210 when the image was shot, and determines what shooting mode the digital camera of this embodiment is in. The shooting mode to be determined in this embodiment is, as described above, whether the shooting mode is SDR shooting or HDR shooting, or whether the function that prioritizes the gradation on the high luminance side is ON or OFF.

[0032] Then, in the next step S303, the luminance range according to the shooting mode determined in step S302 is set as the luminance range to be corrected. In this embodiment, the luminance range selection unit 211 holds information on the luminance range to be corrected calculated in advance for each shooting mode, and in step S303, the luminance range to be corrected according to the shooting mode determined in step S302 is selected from the held luminance ranges to be corrected.

[0033] Hereinafter, a method for calculating the "luminance range to be corrected" will be described in detail with reference to FIG. Graph 410 in FIG. 4 is a graph showing a chromatic gradation characteristic 401 and an achromatic gradation characteristic 402 in a shooting mode determined based on shooting conditions 210 in step S302. The horizontal axis of graph 410 in FIG. 4 shows the input signal value of a RAW image, and the I value on the vertical axis is a luminance evaluation value calculated from an ICtCp color space that can correctly evaluate up to a high luminance range expressed by HDR. Graph 420 in FIG. 4 is a graph showing a change characteristic 403 of the slope of the output I value corresponding to the chromatic gradation characteristic 401 shown in graph 410, and a change characteristic 404 of the slope of the output I value corresponding to the achromatic gradation characteristic 402. The horizontal axis of graph 420 shows the input signal value of a RAW image, and the vertical axis shows the slope of the output I value. The gradation characteristic indicates the correspondence relationship of the output I value to the input signal value (RAW value). The ICtCp color space is a color space defined in ITU-R BT.2100. I (intensity) is the luminance component that represents brightness, and Ct and Cp are color difference components that represent blue-yellow (derived from tritan=type 3 color vision) and red-green (derived from protan=type 1 color vision), respectively. In other words, in graphs 410 and 420 in Figure 4, the gradation characteristics are characteristics that represent the output I value of the ICtCp color space relative to the input signal value.

[0034] In this embodiment, the "predetermined chromatic color" is a color having saturation determined by any saturation and hue, and the "achromatic color" is a gray color having no saturation. In this embodiment, the predetermined chromatic color is a color set in advance, for example, human skin color. However, since human skin color can have various saturations and hues, it is not uniquely determined. In this regard, a reference value for human skin may be determined in advance, or may be determined based on color information detected from a captured image, for example, by taking an average from a skin color area. In this embodiment, the hue and saturation of the reference skin color are determined in advance, and the gradation characteristics corresponding to the hue and saturation are acquired.

[0035] Here, when comparing the achromatic gradation characteristics with the skin tone characteristics, it can be seen from the change characteristics 403 and 404 that the chromatic (skin tone) gradation characteristics 401 have a steeper slope than the achromatic gradation characteristics 402 in the luminance range 405 shown in FIG. 4. In the first embodiment, the gradation correction is performed on the luminance range to be corrected, which is the "luminance range in which the skin tone characteristics have a steeper slope than the achromatic gradation characteristics" as shown in the luminance range 405. The steeper the slope of the output I value is, the more contrast there is in the gradation characteristics, and the finer light and dark areas are more clearly expressed. Therefore, the luminance range in which the skin tone characteristics have a steeper slope is the luminance range in which the unevenness of the skin is more noticeable than in other colors. Therefore, it can be said that by performing the gradation correction only in this luminance range, the skin can be expressed in a favorable manner without impairing the contrast of the entire image. That is, in the first embodiment, the luminance range of the input image in which the slope of the gradation characteristics of a specified chromatic color is greater than the slope of the gradation characteristics of an achromatic color is set as the luminance range to be corrected, and a specified luminance correction is performed on that luminance range to be corrected.

[0036] As described above, the gradation characteristics of flesh tones and achromatic colors are known as design values, and therefore the luminance range selection unit 211 holds information on the luminance range corresponding to the results of comparing these in advance for each shooting mode. Then, the luminance range selection unit 211 selects the luminance range corresponding to the shooting mode determined in step S303 as the luminance range to be corrected.

[0037] In the above description, the saturation and hue for determining the gradation characteristics of skin color are determined as a reference, but the reference saturation and hue may be single or multiple. When there are multiple reference saturations and hues, the luminance range selection unit 211 may hold information on the luminance range obtained for each saturation and hue, and the luminance gradation correction unit 203 may adopt the luminance range corresponding to the color (skin color) closest to the face area detected from the captured image as the luminance range to be corrected. Alternatively, the luminance range selection unit 211 may calculate the luminance range from the gradation characteristics of achromatic colors and the gradation characteristics corresponding to a predetermined chromatic color obtained from the captured image, and output the luminance range to the luminance gradation correction unit 203 as the luminance range to be corrected.

[0038] Furthermore, in the first embodiment, the frequency band to be subjected to gradation correction for the input image is set to an arbitrary band, and correction is performed limited to a frequency range in which unevenness and shading of the skin are of concern. Possible methods of determining the frequency band include setting a frequency band suitable for shading and unevenness of the skin, or determining (or changing) the frequency band based on the face area detected as the subject to be corrected (based on the size of the face area, for example).

[0039] In this way, in the first embodiment, the luminance range obtained by comparing the gradation characteristics of a predetermined chromatic color with the gradation characteristics of achromatic colors is set as the luminance range to be corrected, and a correction target frequency band is set according to the target of the gradation correction, and then gradation correction limited to that range is performed. Gradation correction limited to the frequency band to be corrected is also called local contrast correction processing.

[0040] As described above, the tone correction in this embodiment is assumed to be local contrast correction. Therefore, in step S304 in Fig. 3, the luminance tone correction unit 203 generates a local contrast gain map in the luminance range to be corrected selected in step S303 so that the correction is limited to the frequency band to be corrected set as described above. A gain map is a map in which gains for adjusting the contrast of an image for each pixel are two-dimensionally arranged. That is, the local contrast gain map in this embodiment is a gain map in which each gain is arranged so that the correction is limited to the frequency band to be corrected, and is hereinafter referred to as a local gain map for simplicity. Details of the local gain map generation process according to this embodiment will be described later.

[0041] Next, in step S305, the luminance tone correction unit 203 performs local contrast correction by applying a gain to the input image using the local gain map generated in step S304. Here, if the luminance signal value at coordinates (x, y) of the input image is IN(x, y) and the value of the local gain map at coordinates (x, y) is GMlocal(x, y), the luminance signal value OUT(x, y) of the output image after gain processing is expressed as in the following equation (1).

[0042] OUT(x,y)=GMlocal(x,y)×IN(x,y) Equation (1)

[0043] The local gain map generation process in step S304 will be described in detail below with reference to Fig. 5 and Fig. 6. Fig. 5 is a block diagram showing an example of the configuration for the local gain map generation process, and Fig. 6 is a flowchart showing the flow of the local gain map generation process. In the case of this embodiment, it is assumed that the configuration shown in Fig. 5 is formed in the luminance tone correction unit 203 in Fig. 2.

[0044] The local gain map generating unit 501 is a unit that generates a local gain map for adjusting the local contrast of an image. In the first embodiment, the local gain map generating unit 501 generates a low-frequency image from an input image, and applies gain processing to the low-frequency image to generate a local gain map. The local gain map generating unit 501 includes a first low-frequency image generating unit 502, a second low-frequency image generating unit 503, a correction gain converting unit 504, a gain map converting unit 505, a gain map synthesizing unit 506, and a local gain map calculating unit 507.

[0045] In step S601 of FIG. 6, the first low-frequency image generating unit 502 performs a reduction process on the input image to generate a first reduced image. The second low-frequency image generating unit 503 performs a further reduction process on the first reduced image to generate a second reduced image. In this way, the first reduced image is generated by reducing the input image, and the second reduced image is generated by further reducing the first reduced image. Furthermore, the first low-frequency image generating unit 502 enlarges the first reduced image to the size of the original image to generate a first low-frequency image having a different frequency level from the original image. Moreover, the second low-frequency image generating unit 503 enlarges the second reduced image to the size of the original image to generate a second low-frequency image having a different frequency level from the original image and the first low-frequency image. As a method for reducing and enlarging such an image, a known method such as a process using a bilinear method can be used. Moreover, as a method for obtaining a low-frequency image, a known method such as a low-pass filter process can be used.

[0046] In this embodiment, the input image is a luminance image, and hereinafter, the input image is referred to as an input luminance image, the luminance image obtained by enlarging the first reduced image is referred to as a first layer luminance image, and the luminance image obtained by enlarging the second reduced image is referred to as a second layer luminance image. Note that in Fig. 5, the input luminance image at coordinates (x, y) is referred to as IN(x, y), the first layer luminance image is referred to as INM(x, y), and the second layer luminance image is referred to as INS(x, y).

[0047] In the next step S602, the correction gain conversion unit 504 generates a correction gain characteristic. The generation of the correction gain characteristic will be described in detail with reference to FIG. 7. Graphs 710 and 712 in FIG. 7(a) and graphs 720 and 722 in FIG. 7(b) are diagrams in which the signal values ​​of the image are plotted in the x direction, with the horizontal axis representing the position in the x direction and the vertical axis representing the value of the luminance signal. Graphs 711 in FIG. 7(a) and graph 721 in FIG. 7(b) are diagrams in which the signal values ​​of the image are plotted in the x direction, with the horizontal axis representing the position in the x direction and the vertical axis representing the value of the gain signal. Graphs 710 to 712 in FIG. 7(a) are diagrams for explaining local contrast correction when the gradation characteristic is such that the gain decreases monotonically with respect to the luminance (the higher the luminance, the greater the gain). Graphs 720 to 722 in FIG. 7(b) are diagrams for explaining local contrast correction when the gradation characteristic is such that the gain increases monotonically with respect to the luminance (the higher the luminance, the greater the gain).

[0048] In the graph 710 of FIG. 7(a), the solid line indicates the input luminance signal 701, and the dotted line indicates the luminance signal 702 that is reduced in frequency by filtering and reducing the input luminance signal. The gain signal 703 (gain map) generated by local tone mapping the signal values ​​shown in the graph 710 of FIG. 7(a) with a gradation characteristic in which the gain decreases monotonically with respect to the luminance is represented by the dotted line in the graph 711 of FIG. 7(a). As shown in the graphs 710 and 711, the gain map (gain signal 703) is generated based on the low-frequency luminance signal 702, so that it changes gradually with respect to the change in the input luminance signal 701. That is, the input luminance signal 701 decreases rapidly from left to right on the graph, whereas the gain signal 703 increases gradually from left to right on the graph. As a result, as shown in the graph 712 of FIG. 7(a), the signal portion 705 in the luminance signal 704 after gain processing can produce a contrast improvement effect.

[0049] In FIG. 7B, where the gain has a monotonically increasing gradation characteristic with respect to luminance, the graph 720 represents the input luminance signal 701 and the low-frequency luminance signal 702, as in the graph 710 in FIG. 7A. In addition, the graph 721 in FIG. 7B shows the gain signal 706 generated by local tone mapping the signal value with a gradation characteristic in which the gain increases monotonically with respect to luminance, and the gain map in this case is shown by the dotted line in the figure. As shown in the graphs 720 and 721, the gain map (gain signal 706) is generated based on the low-frequency luminance signal 702, so that it changes gradually with respect to the change in the input luminance signal 701. That is, the input luminance signal 701 decreases rapidly from left to right on the graph, whereas the gain signal 706 decreases gradually from left to right on the graph. As such, it can be seen that the graph 721 in FIG. 7B has the opposite characteristics to the graph 711 in FIG. 7A. As a result, as shown in graph 722 of FIG. 7(b), the signal portion 708 in the luminance signal 707 after gain processing can produce a contrast reduction effect.

[0050] The above is how contrast can be improved or reduced by varying the correction gain characteristics. Here, in this embodiment, since it is necessary to apply a correction that improves the appearance of the skin, it is necessary to set the local gain in a direction that reduces the local contrast. Therefore, in this embodiment, the correction gain characteristic is set to a monotonically increasing characteristic. Furthermore, in this embodiment, it is necessary to apply local contrast correction limited to the correction target luminance range selected in step S303. Also, it is necessary to apply correction to the frequency band that is the target of gradation correction in an arbitrarily set frequency band. Therefore, in step S602, the correction gain characteristic is determined taking these limiting factors of the correction target into account.

[0051] Next, in S603, the gain map conversion unit 505 generates a gain map for each layer by applying the correction gain characteristics generated as described above to the input luminance image. Hereinafter, the gain map generated by applying the correction gain characteristics to the input luminance image that has not been subjected to the reduction process is referred to as a single-magnification layer gain map. Also, the gain map generated by applying the correction gain characteristics to the first layer luminance image INM generated in step S601 is referred to as a first layer gain map. Also, the gain map generated by applying the correction gain characteristics to the second layer luminance image INS is referred to as a second layer gain map.

[0052] 5, the equal-magnification hierarchical gain map obtained by applying the above-mentioned correction gain characteristics to the input luminance image IN(x,y) is denoted as GML(x,y). The first hierarchical gain map obtained by applying the correction gain characteristics to the first hierarchical luminance image INM(x,y) is denoted as GMM(x,y), and the second hierarchical gain map obtained by applying the correction gain characteristics to the second hierarchical luminance image INS is denoted as GMS(x,y). The equal-magnification hierarchical gain map is then sent to a local gain map calculation unit 507, and the first hierarchical gain map and the second hierarchical gain map are sent to a gain map synthesis unit 506.

[0053] Next, in step S604, the gain map synthesis unit 506 synthesizes the first hierarchical gain map and the second hierarchical gain map to generate a synthetic gain map. The synthetic gain map is generated using a known method (see JP 2014-154108 A) in which weighted addition is performed according to a gain signal difference between a gain signal of a gain map with a large image size and a gain signal of a gain map with a small image size.

[0054] Next, in step S605, the local gain map calculation unit 507 generates a final local gain map to be used for local contrast correction from the equal-magnification hierarchical gain map and the composite gain map generated in step S604. If the signal of the composite gain map at coordinates (x, y) is GMC(x, y) and the final local gain map at coordinates (x, y) is GMlocal(x, y), then GMlocal(x, y) is expressed by the following formula (2).

[0055] GMlocal(x,y)=GMC(x,y)-GML(x,y) Equation (2)

[0056] This concludes the description of the local gain map generation process in step S304 of FIG. Then, in step S305, the gain processing unit 508 applies the final local gain map to the input luminance image IN(x,y) to generate an output luminance image OUT(x,y).

[0057] <Second embodiment> In the first embodiment, an example in which the development method is not particularly limited is given, but in the second embodiment, a skin correction method assuming HDR development (development having gradation characteristics with a peak luminance higher than that of conventional SDR development) will be described. In other words, if the gradation characteristics in the case of HDR development are gradation characteristics with a first peak luminance as the upper limit, the gradation characteristics in the case of SDR development are gradation characteristics with a second peak luminance as the upper limit, and the first peak luminance is a luminance value greater than the second peak luminance. In addition, the first peak luminance and the second peak luminance are maximum luminance values ​​based on the output dynamic range.

[0058] The overall flow of image processing in the second embodiment is basically the same as the flow of the first embodiment described in Fig. 3, so illustration and description thereof will be omitted. The processing in the second embodiment that differs from the first embodiment is the method of calculating the "luminance range to be corrected" that is selected according to the shooting mode in step S303 in Fig. 3. The method of calculating the "luminance range to be corrected" in the second embodiment will be described below.

[0059] In the second embodiment, the skin color is determined by an arbitrary saturation and hue, as in the first embodiment, and this arbitrary saturation and hue may be determined by predetermining a reference value for human skin, or by taking an average from the skin area of ​​a captured image. In this embodiment, the hue and saturation of the reference skin color are determined in advance, and the gradation characteristics for SDR development and the gradation characteristics for HDR development are obtained for that skin color.

[0060] Graph 810 in Fig. 8 shows gradation characteristics 801 in the case of HDR development and gradation characteristics 802 in the case of SDR development (development with gradation characteristics with a peak luminance lower than HDR as the upper limit value) for skin color in the shooting mode determined in step S302. The horizontal axis of graph 810 indicates the input signal value of the RAW image, and the vertical axis indicates the output I value. Graph 820 in Fig. 8 is a graph showing change characteristics 803 and 804 of the slope of the output I value corresponding to gradation characteristics 801 and 802, respectively. The horizontal axis of graph 820 indicates the input signal value of the RAW image, and the vertical axis indicates the slope of the output I value.

[0061] Here, when comparing the gradation characteristics in the case of HDR development with the gradation characteristics in the case of SDR development, it can be confirmed from the change characteristics 803 and 804 that the gradation characteristics in the case of HDR development have a steeper slope than the gradation characteristics in the case of SDR development in the luminance range 805. In the second embodiment, gradation correction is performed with the luminance range to be corrected, which is the "luminance range in which the gradation characteristics in the case of HDR development have a steeper slope than the gradation characteristics in the case of SDR development" as shown in the luminance range 805. That is, in the second embodiment, gradation correction is performed with the luminance range to be corrected, in which the slope of the gradation characteristics of a predetermined chromatic color in the gradation characteristics having the first peak luminance as an upper limit value is steeper than the slope of the gradation characteristics of a predetermined chromatic color in the gradation characteristics having the second peak luminance as an upper limit value.

[0062] As explained in the first embodiment, the greater the gradient of the gradation characteristics, the greater the contrast, and the clearer the finer the light and dark areas. Therefore, the luminance range in which the gradient of the gradation characteristics in HDR development is greater is the luminance range in which the unevenness of the skin is more noticeable in HDR development than in SDR development. Therefore, by performing gradation correction limited to this luminance range, it can be said that the skin can be expressed favorably without impairing the contrast of the entire image. Also, as explained in the first embodiment, the gradation characteristics of the skin color in SDR development and the gradation characteristics of the skin color in HDR development can be obtained in advance as design values, so that they can be compared in advance for each shooting mode.

[0063] Therefore, in the second embodiment, the luminance range selection unit 211 also holds information on the luminance range corresponding to the result of comparing these in advance for each shooting mode. Then, the luminance range selection unit 211 selects the luminance range corresponding to the shooting mode determined in step S303 as the luminance range to be corrected. Note that in the second embodiment, as in the case of the first embodiment described above, the saturation and hue for determining the gradation characteristics of the skin color are determined as references, but the reference saturation and hue may be single or multiple. When there are multiple reference saturations and hues, information on the luminance range obtained for each saturation and hue may be held, and the luminance gradation correction unit 203 may adopt the luminance range corresponding to the skin color closest to the face area detected from the captured image as the luminance range to be corrected.

[0064] In the second embodiment, the contents of the subsequent processes in steps S304 and S305 are similar to those in the first embodiment, and therefore a description thereof will be omitted. The above is a description of the image processing in the image processing device of the second embodiment.

[0065] As described above, according to the first and second embodiments, it is possible to reduce the shadows and unevenness of human skin while maintaining the gradation of the entire image. In particular, when HDR shooting or HDR development is performed, it is possible to express human skin in a good condition while suppressing the loss of HDR effects. In the second embodiment, a determination and correction unit may also be provided that integrates both the luminance tone correction unit 203 and the luminance range selection unit 211 in the functional blocks of FIG.

[0066] <Other embodiments> In the first embodiment described above, an example was described in which a luminance range in which the gradation characteristics of a predetermined chromatic color have a greater slope than the gradation characteristics of achromatic colors is determined as the luminance range to be corrected. In the second embodiment, a method was described in which a luminance range in which the gradation characteristics of a predetermined chromatic color in HDR development have a greater slope than the gradation characteristics of a predetermined chromatic color in SDR development is determined as the luminance range to be corrected.

[0067] As yet another embodiment, it is also possible to determine a luminance range in which the magnitude relationship of the gradient of the gradation characteristics is reversed as the luminance range to be corrected. For example, in the first embodiment, when the predetermined chromatic color is green, the luminance range in which the green gradation characteristics have a smaller slope than the achromatic gradation characteristics is set as the luminance range to be corrected. That is, the luminance range in which the gradation characteristics of the predetermined chromatic color have a smaller slope than the achromatic gradation characteristics is set as the luminance range to be corrected, among the luminance ranges of the input image. Then, luminance correction processing is performed on the luminance range to be corrected. Then, by generating a local gain map in a direction that improves contrast, it is possible to improve areas where the details of the veins are difficult to see.

[0068] Furthermore, it is possible to combine correction gains for color regions where contrast is desired to be strengthened and color regions where contrast is desired to be weakened. For example, in a photographed image in which both leaves and people are captured, based on the flow of the flowchart in Fig. 3, a correction gain is set in the direction of reducing contrast in a luminance region in which the gradation characteristics of skin tones have a greater slope than the gradation characteristics of achromatic colors. Also, a correction gain is set in the direction of improving contrast in a luminance region in which the gradation characteristics of green tones have a smaller slope than the gradation characteristics of achromatic colors. In this case, it is possible to express both subjects favorably.

[0069] Also, for example, as an example in which the magnitude relationship of the gradient of the gradation characteristics is reversed from that of the second embodiment, a luminance range in which the gradient of the gradation characteristics of a predetermined chromatic color in HDR development is smaller than that in SDR development may be set as the luminance range to be corrected. In other words, the luminance range in which the gradient of the gradation characteristics of a predetermined chromatic color in the gradation characteristics having an upper limit value of a first peak luminance is smaller than the gradient of the gradation characteristics of a predetermined chromatic color in the gradation characteristics having an upper limit value of a second peak luminance is set as the luminance range to be corrected and gradation correction is performed.

[0070] In the above-described embodiment, the luminance tone correction unit 203 performs local contrast correction such that luminance correction is performed with a different correction amount for each frequency band. However, luminance correction may be performed with the same correction amount for all frequency bands. Furthermore, in each of the above-described embodiments, an example was given in which the input and output in the local contrast correction process were luminance images, but the input and output images may be images in other formats, such as so-called YUV.

[0071] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0072] 101: control unit, 104: signal processing circuit, 202: WB correction unit, 203: luminance tone correction unit, 204: color interpolation unit, 205: development processing unit, 206: color luminance adjustment unit, 207: compression unit, 208: recording unit, 211: luminance range selection unit

Claims

1. A determination means for determining whether to perform a correction; a correction means for correcting a specific frequency band that is adapted to a subject having a skin color as a closest color based on the determination by the determination means; having The image processing device according to claim 1, wherein the correction means corrects a luminance range in which a gradient of the gradation characteristics of flesh color is larger than a gradient of the gradation characteristics of achromatic colors.

2. The correction means is a band setting means for setting a specific frequency band of an input image as the frequency band to be corrected; a luminance range setting means for setting a luminance range in which the gradient of the gradation characteristic of the skin color is larger than that of the gradation characteristic of an achromatic color, as a luminance range to be corrected, from among the luminance ranges of the input image; having 2. The image processing apparatus according to claim 1, wherein a gradation correction is performed on the input image based on the set frequency band and the set luminance range.

3. A determination means for determining whether to perform a correction; a correction means for correcting a specific frequency band that is adapted to a subject having a color that is closest to green based on the determination by the determination means; having The image processing device according to claim 1, wherein the correction means corrects a luminance range in which a gradient of the green gradation characteristic is smaller than a gradient of the achromatic gradation characteristic.

4. The correction means is a band setting means for setting a specific frequency band of an input image as the frequency band to be corrected; a luminance range setting means for setting a luminance range in which the gradient of the green gradation characteristic is smaller than the gradient of the achromatic gradation characteristic as a luminance range to be corrected, from among the luminance ranges of the input image; having 4. The image processing apparatus according to claim 3, wherein a gradation correction is performed on the input image based on the set frequency band and the set luminance range.

5. A determination means for determining whether to perform a correction; a correction means for correcting a specific frequency band that is adapted to a subject having a skin color as a closest color based on the determination by the determination means; having The image processing device is characterized in that the correction means corrects a luminance range in which a slope of the skin color gradation characteristic in the gradation characteristic having an upper limit value of a first peak luminance is larger than a slope of the skin color gradation characteristic in the gradation characteristic having an upper limit value of a second peak luminance.

6. The correction means is a band setting means for setting a specific frequency band of an input image as the frequency band to be corrected; a luminance range setting means for setting, from among the luminance ranges of the input image, a luminance range in which a gradient of a skin tone characteristic in a gradation characteristic having an upper limit value of a first peak luminance is greater than a gradient of the skin tone characteristic in a gradation characteristic having an upper limit value of a second peak luminance, as a luminance range to be corrected; having 6. The image processing apparatus according to claim 5, wherein a gradation correction is performed on the input image based on the set frequency band and the set luminance range.

7. A determination means for determining whether to perform a correction; a correction means for correcting a specific frequency band that is adapted to a subject having a color that is closest to green based on the determination by the determination means; having The image processing device is characterized in that the correction means corrects a luminance range in which a slope of the green gradation characteristic in the gradation characteristic having an upper limit value of a first peak luminance is smaller than a slope of the green gradation characteristic in the gradation characteristic having an upper limit value of a second peak luminance.

8. The correction means is a band setting means for setting a specific frequency band of an input image as the frequency band to be corrected; a luminance range setting means for setting, from among the luminance ranges of the input image, a luminance range in which a gradient of a green gradation characteristic in a gradation characteristic having an upper limit value of a first peak luminance is smaller than a gradient of the green gradation characteristic in a gradation characteristic having an upper limit value of a second peak luminance, as a luminance range to be corrected; having 8. The image processing apparatus according to claim 7, wherein a gradation correction is performed on the input image based on the set frequency band and the set luminance range.

9. 9. The image processing device according to claim 5, wherein the first and second peak luminances are maximum values ​​of luminance based on an output dynamic range.

10. 10. The image processing device according to claim 5, wherein the first peak luminance is a luminance value greater than the second peak luminance.

11. 5. The image processing apparatus according to claim 1, wherein the correction means determines the specific frequency band based on a subject to be corrected in the input image.

12. 12. The image processing device according to claim 1, wherein the determination means determines whether or not to perform the correction based on the presence or absence of a subject to be corrected in the input image, or the presence or absence of a pixel having a color similar to the color of the subject to be corrected.

13. 13. The image processing apparatus according to claim 1, wherein the correction means performs the correction with a different correction amount for each frequency band or with the same correction amount for all frequency bands.

14. 14. The image processing device according to claim 1, wherein the gradation characteristic is a characteristic that represents an output in an ICtCp color space with respect to an input signal value.

15. 7. The image processing apparatus according to claim 1, wherein the skin color is determined based on a preset skin color or a skin color obtained from an input image.

16. The image processing device according to any one of claims 1, 2, 5, 6 and 15, characterized in that the correction means sets, as the luminance range to be corrected, a luminance range selected from a plurality of luminance ranges prepared in advance according to the shooting mode when the input image was obtained by shooting, or a luminance range calculated based on a gradation characteristic according to the gradation characteristic of achromatic colors and skin color obtained from the input image.

17. The image processing device according to claim 16, characterized in that when a luminance range selected from the plurality of luminance ranges prepared in advance according to the shooting mode is set as the luminance range to be corrected, the correction means selects a luminance range corresponding to a color closest to a skin color obtained from the input image.

18. An image processing method executed by an image processing device, comprising: a determination step for determining whether to perform a correction; a correction step of selecting a specific frequency band that is adapted to a subject having a skin color as a closest color based on the determination step; having The image processing method according to the present invention, wherein the correction step targets a luminance range in which the gradient of the gradation characteristics of flesh color is greater than the gradient of the gradation characteristics of achromatic colors.

19. An image processing method executed by an image processing device, comprising: a determination step for determining whether to perform a correction; a correction step of selecting a specific frequency band that is matched with a subject having a closest color of green as a correction target based on the determination step; having The image processing method according to the present invention, wherein the correction step targets a luminance range in which the gradient of the green gradation characteristic is smaller than the gradient of the achromatic gradation characteristic.

20. An image processing method executed by an image processing device, comprising: a determination step for determining whether to perform a correction; a correction step of selecting a specific frequency band that is adapted to a subject having a skin color as a closest color based on the determination step; having an upper limit of the first peak luminance, and a luminance range in which a slope of the skin color gradation characteristic in the gradation characteristic having an upper limit of the second peak luminance is larger than a slope of the skin color gradation characteristic in the gradation characteristic having an upper limit of the second peak luminance, the correction step being targeted for the luminance range.

21. An image processing method executed by an image processing device, comprising: a determination step for determining whether to perform a correction; a correction step of selecting a specific frequency band that is matched with a subject having a closest color of green as a correction target based on the determination step; having The image processing method according to the present invention, characterized in that in the correction step, a luminance range in which a slope of the green gradation characteristic in the gradation characteristic having an upper limit value equal to a first peak luminance is smaller than a slope of the green gradation characteristic in the gradation characteristic having an upper limit value equal to a second peak luminance is targeted for correction.

22. A program for causing a computer to function as the image processing device according to any one of claims 1 to 17.

Images