Image Processing Apparatus, Image Processing Method, and Storage Medium

Abstract

Image processing device, image processing method, and storage medium. The image processing device includes: a detection unit configured to detect a subject area from an image; a first image processing unit configured to perform first image processing on the subject area; a second image processing unit configured to perform second image processing on an area other than the subject area; a first determination unit configured to determine, in the subject area, a first reference area that is to undergo the first image processing and has a size close to that of the subject area; and a second determination unit configured to determine a second reference area that has a size close to that of the subject area and is different in size from the first reference area, so as to determine an area to undergo the second image processing.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus that performs image processing on each region of an image captured by a digital camera or other such device. Background Art

[0002] Heretofore, a correction method for extracting a subject region and correcting the brightness of the subject region, and an image processing apparatus for correcting the gray level of a region other than the subject region have been proposed.

[0003] For example, Japanese Unexamined Patent Application Publication No. 2018-182700 discloses a technique for determining a subject region and correcting the illumination of the subject region. By performing relighting processing using a virtual light source, it is possible to brighten a dark region such as a shadow caused by ambient light, and a desired image can be obtained.

[0004] In this case, known methods for detecting a subject region involve acquiring distance information, and detecting a region having a distance value within a predetermined range with respect to the main subject as the subject region.

[0005] Although there are various methods for acquiring distance information, a representative technique is the following stereo method, which involves: acquiring images from a plurality of viewpoint positions, and calculating distances based on triangulation using disparities calculated from the correspondence of pixels in these images. By acquiring distance information and detecting the subject region, it is possible to distinguish between a subject region located nearby and a subject region located far away.

[0006] For example, between the case where a virtual light source is directed at a nearby subject and the case where a virtual light source is directed at a faraway subject, control can be performed such that the irradiation amount (intensity) of the virtual light source is changed according to the distance difference from the virtual light source to the subject.

[0007] In addition, Japanese Unexamined Patent Application Publication No. 2014-153959 discloses a method that involves: detecting a plurality of subject regions, and performing gray level correction on each subject region (for example, the region of the main subject and regions other than the main subject).

[0008] The methods for subject region estimation used in the above methods include a method of estimating a human body region from the result of face detection, a technique of using machine learning to estimate a subject region, and a method of detecting a subject region by acquiring distance information.

[0009] However, in the case where the subject area is not properly estimated, unnatural correction may be performed. For example, when correcting the brightness of a person area, correction is also performed to brighten the background area, or when correcting the contrast by correcting the background area, not only the contrast of the background area is improved, but also the contrast of the main subject area is improved.

[0010] To solve these problems, Japanese Patent Application Laid-Open No. 2018-182700 discloses the following method: based on the relationship between the shooting conditions and the subject conditions and the result of image analysis of the subject, switching the means of obtaining subject area information from subject distance information and the means of obtaining subject area information from an image.

[0011] However, for a subject area estimation method using image information such as subject continuity and face detection information, it is difficult to correctly estimate the subject area when the boundary between the background color and the skin and clothing colors is unclear. A similar problem also exists for a subject area detection method using machine learning. For a technique using distance information, there is the following problem: it is impossible to correctly estimate the subject area for a subject located close to the main subject, etc. Summary of the Invention

[0012] The present invention has been made in view of the above problems, and the present invention provides an image processing apparatus capable of performing more natural correction when performing image correction on each subject area.

[0013] According to a first aspect of the present invention, there is provided an image processing apparatus including: a detection unit configured to detect a subject area from an image; a first image processing unit configured to perform first image processing on the subject area; a second image processing unit configured to perform second image processing on an area other than the subject area; a first determination unit configured to determine, in the subject area, a first reference area that is to undergo the first image processing and has a size close to the subject area; and a second determination unit configured to determine a second reference area that has a size close to the subject area and is different in size from the first reference area, so as to determine an area to undergo the second image processing.

[0014] According to a second aspect of the present invention, there is provided an image processing method including: detecting a subject area from an image; performing first image processing on the subject area; performing second image processing on an area other than the subject area; determining, in the subject area, a first reference area that is to undergo the first image processing and has a size close to the subject area; and determining a second reference area that has a size close to the subject area and is different in size from the first reference area, so as to determine an area to undergo the second image processing.

[0015] According to a third aspect of the present invention, there is provided a non-transitory computer-readable storage medium storing a program for causing a computer to execute the processing of the above-described image processing method.

[0016] Other features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a block diagram showing a configuration of a digital camera according to an embodiment of the present invention.

[0018] Figure 2 is a block diagram showing a flow of processing in the embodiment.

[0019] Figure 3 is a block diagram showing a flow of background area correction processing in the embodiment.

[0020] Figure 4 is a block diagram showing a flow of person area correction processing in the embodiment.

[0021] Figures 5A to 5C is a conceptual diagram of a shooting scene assumed to be applicable to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. Multiple features are described in the embodiments, but the invention is not limited to requiring all of these features, and multiple such features can be combined appropriately. Further, in the drawings, the same or similar configurations are given the same reference numerals and their repeated description is omitted.

[0023] Hereinafter, a digital camera which is an embodiment of an image processing apparatus according to the present invention will be described. In the present embodiment, a digital camera (imaging apparatus) will be described as an example of the image processing apparatus, but the image processing apparatus of the present invention is not limited to an imaging apparatus and may be, for example, a personal computer (PC) or other such device.

[0024] Generally, when taking an image of a person in a backlit scene, when shooting such that the person subject is properly exposed, the background area will be overexposed. On the contrary, when shooting such that the background area is properly exposed, the person area will be underexposed. In this case, shooting is performed under a shooting condition that is somewhere between the shooting condition for properly exposing the person area and the shooting condition for properly exposing the background area. However, an image taken in this way will ultimately be an image with a dark person and insufficient contrast in the background. Figure 5AIt is a conceptual diagram envisioned for the shooting scene applicable to the present invention. It shows a situation where the contrast of the background is low and the human subject appears dark.

[0025] To correct the appearance of such an image, it is desirable to perform a process for making the brightness of the subject area brighter and for improving the contrast of the background area. In a landscape scene such as a distant view of a mountain, sea, or lake, the contrast is usually reduced due to the influence of fog or water vapor in the air, and thus there is a high demand for improving the contrast of the background area.

[0026] In the present embodiment, a correction process for improving the contrast is performed in the background area correction, and a relighting correction process for correcting the illumination such as the brightness of the subject or the direction of the light source illuminating the subject is performed in the subject area.

[0027] <Configuration of the digital camera>

[0028] Figure 1 It is a block diagram showing the configuration of the digital camera 100 of the present embodiment.

[0029] In Figure 1 In the digital camera 100 shown, the light that enters through the lens unit 101 (imaging optical system) including a zoom lens and a focusing lens and the shutter 102 that provides an aperture function is photoelectrically converted in the imaging unit 103. The imaging unit 103 is configured with an image sensor such as a CCD or CMOS sensor, and the electrical signal obtained through photoelectric conversion is output as an image signal to the A / D converter 104. The A / D converter 104 converts the analog image signal output by the imaging unit 103 into a digital image signal (image data), and outputs the digital image signal to the image processing unit 105.

[0030] The image processing unit 105 performs various types of image processing such as white balance and other color conversion processing, gamma processing, edge enhancement processing, and color correction processing on the image data from the A / D converter 104 or the image data read from the image memory 106 via the memory control unit 107. The image data output by the image processing unit 105 is written into the image memory 106 via the memory control unit 107. The image memory 106 stores the image data output by the image processing unit 105 and the image data for display on the display unit 109. The face / face organ detection unit 113 detects the face area and the face organ area where a human face or face organs exist from the captured image.

[0031] In the image processing unit 105, using the face detection result and the face organ detection result of the face / face organ detection unit 113 and the captured image data, a predetermined evaluation value calculation process is performed, and the system control unit 50 performs exposure control and focus control based on the obtained evaluation value. Thus, through-the-lens (TTL) autofocus (AF) processing, automatic exposure (AE) processing, automatic white balance (AWB) processing, and other such processes are performed.

[0032] In addition, the D / A converter 108 converts the digital image data for display stored in the image memory 106 into an analog signal, and supplies the analog signal to the display unit 109. The display unit 109 performs display depending on the analog signal from the D / A converter 108 on a display device such as an LCD.

[0033] The codec unit 110 compresses and encodes the image data stored in the image memory 106 based on standards such as JPEG or MPEG. The system control unit 50 stores the encoded image data in a recording medium 112 such as a memory card or a hard disk via an interface (I / F) 111. In addition, the image data read out from the recording medium 112 via the I / F 111 is decoded and decompressed by the codec unit 110, and stored in the image memory 106. By displaying the image data stored in the image memory 106 on the display unit 109 via the memory control unit 107 and the D / A converter 108, the image can be reproduced and displayed.

[0034] The relighting processing unit 114 performs relighting processing for directing a virtual light source at the captured image to correct the brightness. The image compositing processing unit 115 composites two types of images according to a compositing map. The compositing map represents the compositing ratio of the two types of images.

[0035] The system control unit 50 controls the entire system of the digital camera 100. The non-volatile memory 121 is composed of a memory such as an EEPROM, and stores programs, parameters, etc. required when the system control unit 50 performs processing. The system control unit 50 extracts and executes the programs recorded on the non-volatile memory 121 and constants and variables used when the system control unit 50 operates in the system memory 122, to implement various processes described later in the present embodiment. The operation unit 120 accepts operations of the user such as menu setting and image selection.

[0036] <Flow of Image Correction Processing>

[0037] Figure 2 is a block diagram showing the flow of the processing in the present embodiment.

[0038] exist Figure 2 , an input image 201 is input to the image processing unit 105, and a reference image 202 is generated. The reference image 202 generated here is used to estimate a face region or a human body region, and is an image for which image quality settings suitable for face detection, human body region detection, etc. have been configured, and the image is different from an image for which white balance, hue, and other such settings of the image have been configured by the user. Even for an image with many dark regions or an image with high saturation, it is advantageous to configure settings that facilitate detection of a face region and a human body region, such as setting automatic white balance, hue, and saturation to standard, setting dark region correction to strong, and setting brightness setting to bright.

[0039] In the face / facial part detection unit 113, facial region information 203 is generated using the reference image 202 generated by the image processing unit 105. The facial region information 203 is information such as edge information of the facial region, position information of facial parts such as eyes, nose, and mouth of the face and angles relative to the image, and face detection reliability indicating the detection accuracy of the position information of the facial parts, etc. For the face and facial part detection method, a technique using template matching or machine learning and other such techniques are generally employed.

[0040] The background contrast correction region determination unit 300 determines a region where contrast is to be improved based on the face region information 203 and the reference image 202. The contrast enhancement unit 310 improves the contrast of the contrast enhancement region (contrast correction processing region) determined by the background contrast correction region determination unit 300.

[0041] The re-illumination correction region determination unit 400 determines a region for performing illumination correction based on the face region information 203 and the reference image 202. The re-illumination processing unit 410 performs re-illumination processing on the illumination correction region determined by the re-illumination correction region determination unit 400.

[0042] The contrast enhancement processing performed by the contrast enhancement unit 310 may be a generally used method in which a gamma process and a sharpness enhancement process for improving a ratio of a dark region to a bright region as a grayscale conversion characteristic, a process for enhancing only a local contrast, and other such processes are used. In the present embodiment, a local contrast enhancement process will be applied.

[0043] Figure 3 2 is a block diagram showing the flow of processing by the background contrast correction region determination unit 300 and the contrast enhancement unit 310 in the present embodiment.

[0044] exist Figure 3Among them, the subject area setting unit 301 for background correction determines the subject area for background correction from the face area information 203. The person area specifying unit 302 for background correction specifies the background area from the reference image 202 and the subject area determined by the subject area setting unit 301 for background correction, and generates a background contrast correction area map 210. The image synthesis processing unit 115 determines the synthesis processing ratio for each area based on the area / level information of the background contrast correction area map 210, and performs the processing for synthesizing the input image 201 and the output image of the contrast enhancement processing unit 311.

[0045] The background contrast correction area map 210 generated by the person area specifying unit 302 for background correction is area information indicating the area determined as the background area, and is information on the part not determined as the person area. In other words, this information is area information in which the area determined as the person area has the determination level of the person reversed, and this information corresponds to the area information indicating the correction level of contrast correction in the synthesis processing unit 115.

[0046] Figure 4 It is a block diagram showing Figure 2 the process of relighting processing in the relighting correction area determination unit 400 and the relighting processing unit 410.

[0047] The subject area setting unit 401 sets the subject area based on the face area information 203. The subject area specifying unit 402 uses the subject area set by the subject area setting unit 401 and the reference image 202 to generate a relighting correction area map 220.

[0048] The virtual light source calculation unit 411 uses the input image 201 and the face area information 203 to generate the light source information of the virtual light source. The virtual light source component operation unit 412 uses the virtual light source information calculated by the virtual light source calculation unit 411 and the relighting correction area map 220 to generate a relighting gain map 221. The relighting gain map 221 is a gain map representing the correction gain for correcting the brightness of the subject area through the image area. The gain multiplication unit 413 performs gain multiplication on the contrast-corrected image 211 for each subject area according to the relighting gain map 221.

[0049] Note that since the method for generating light source information for virtual light sources in the virtual light source calculation unit 411 and the method for calculating virtual light source components in the virtual light source component operation unit 412 are described in detail in Japanese Patent Laid-Open No. 2018-182700, their descriptions will be omitted in the present embodiment. Although Japanese Patent Laid-Open No. 2018-182700 performs relighting processing during the processing by the signal processing circuit, in terms of the processing flow, the relighting processing is performed after the gamma correction processing, and substantially the same effect as that obtained when performing the relighting processing before the signal processing as in the present embodiment is obtained.

[0050] <Method for Applying Human Reference Image>

[0051] Next, the subject area setting method of the subject area setting unit 301 for background correction in Figure 3 and the subject area setting unit 401 in Figure 4 will be described.

[0052] In the present embodiment, a human reference image (human reference area) including a face and a human body area is set in advance. The face area of the human reference image has facial organ information such as eyes, nose, and mouth of the face. The human reference image is made to coincide with the area of the actual image by scaling and rotation so that the organ information of the human reference image coincides with the positions of the organs such as eyes, nose, and mouth of the face detected from the captured image. In addition, the human reference image is set to have a gray scale that reduces the determination level of the human body area in the direction toward the periphery of the human body area.

[0053] The human reference image for background area correction uses a model (subject model image) in which the human body part is set larger than a typical human body model, and the human reference image used by the subject area setting unit 401 uses a model in which the human body part is set smaller than a typical human body model. Similarly, for the area corresponding to the head of the human body area, a model in which the head is set larger than a typical human body model can be used for the human reference image for background area correction, and a model in which the head is set smaller than a typical human body model can be used for the human reference image used by the subject area setting unit 401.

[0054] Figures 5A to 5C is a conceptual diagram showing an exemplary application of the subject model in the present embodiment. Figure 5B shows the reference image applied (as a target) in the case of the relighting processing for correcting the brightness of the human body area, and for the image capturing of the backlit human in Figure 5A , the human reference image is set to be located inside the human body.

[0055] Figure 5C shows a reference image applicable to the case of correcting a background region. In relation to the image capture of the backlit person in Figure 5A , the person reference image is set to extend outside the person. In addition, since the facial region has a large amount of organ information such as eyes and mouth compared to the human body region, the detection accuracy of the facial region is high. Therefore, the size of the head region of the reference image can be set to be equivalent to the actual image, or the magnification / reduction ratio of the actual image is small compared to the human body region. Figure 5B and Figure 5C both show the case where the magnification / reduction ratio of the reference image of the head region to the actual image is set small compared to the human body region.

[0056] Figure 3 The subject region specifying method of the person region specifying unit 302 for background correction in Figure 4 and the subject region specifying unit 402 in

[0057] is related to performing a subject region specifying process by performing a shaping process with reference to the pixel values of the reference image 202 so that the subject reference image coincides with the edge of the subject, as described in Japanese Unexamined Patent Application Publication No. 2017 - 11652.

[0058] In this way, when the subject region specifying unit 402 applies the subject region specifying process that makes the subject reference image coincide with the edge of the subject, in the person region specifying unit 302 for background correction for correcting the contrast of the background region, the person specifying region is set to extend outside the person. In addition, since the person specifying level decreases from the inside to the outside of the person, the background contrast correction region mapping 210 with the specified region setting reversed is set, and an operation to increase the contrast is performed on the outside of the person. In other words, the less a region is determined to be a background region, the closer it is to the person region, and thus, the less the contrast enhancement process is applied, the closer it is to the person region.

[0059] In addition, the subject area designation process involves shaping the subject area based on the edge information of the image. However, since the person reference image is selected in such a way that it does not overlap with each correction area, even in cases where the edge of the subject area cannot be detected, correction failure can be effectively avoided.

[0060] The following configuration can be adopted: Prepare person reference images to be used in the subject area setting unit 301 for background correction and the subject area setting unit 401 in the present embodiment for each orientation in the left-right direction of the face. The best-angle person reference image can be selected based on the information related to the orientation of the face detected from the image. In addition, a three-dimensional model including a face area and a human body area can be prepared, and a human body reference image can be generated according to the angle of the face. In addition, in addition to preparing reference images for each angle of the face, reference images for the face and the human body area can be prepared separately for each angle. The model of the face angle can be selected based on the face angle detected from the image, multiple human body areas with different angles below the neck can be applied to the lower area of the face in the actual image, and the angle with the minimum dispersion of the person area or the angle with the highest similarity within the area can be set as the reference image of the person area. Figure 3 in the present embodiment Figure 4 The following configuration can be adopted: Prepare person reference images to be used in the subject area setting unit 301 for background correction and the subject area setting unit 401 in the present embodiment for each orientation in the left-right direction of the face. The best-angle person reference image can be selected based on the information related to the orientation of the face detected from the image. In addition, a three-dimensional model including a face area and a human body area can be prepared, and a human body reference image can be generated according to the angle of the face. In addition, in addition to preparing reference images for each angle of the face, reference images for the face and the human body area can be prepared separately for each angle. The model of the face angle can be selected based on the face angle detected from the image, multiple human body areas with different angles below the neck can be applied to the lower area of the face in the actual image, and the angle with the minimum dispersion of the person area or the angle with the highest similarity within the area can be set as the reference image of the person area.

[0061] In addition, the present embodiment describes a reference image for the case of correcting the subject area and a reference image for the case of correcting the area outside the subject area, that is, multiple reference images, and selecting the reference image to be used therefrom. However, the following configuration can be adopted: There is a common reference image for both the case of correcting the subject area and the case of correcting the area outside the subject area, and for the head area and the human body area, the scaling factor of the reference image changes according to the situation to which the reference image is applied.

[0062] In the present embodiment, a method of estimating the subject area from the size and direction of the face area is used in subject area estimation, but the present invention is not limited to this method, and there are area estimation methods using machine learning and other similar methods. Similarly, for these methods, in cases where the hue or pattern of the human body area is very similar to the background area, the human body area usually cannot be correctly extracted, and the present invention is still applicable in these cases.

[0063] <Other Embodiments>

[0064] Embodiments of the present invention can also be implemented by the following method, that is, software (program) that executes the functions of the above embodiments is provided to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0065] Although the present invention has been described with reference to exemplary embodiments, it should be understood that the present invention is not limited to the disclosed exemplary embodiments. The scope of the above claims is to be accorded the broadest interpretation so as to encompass all such modifications, equivalent structures and functions.

Claims

1. An image processing apparatus, comprising: a detection unit configured to detect a subject area from an image; a first image processing unit configured to perform first image processing on the subject area; a second image processing unit configured to perform second image processing on an area other than the subject area; a first determination unit configured to determine, in the subject area, a first reference area to undergo the first image processing, the first reference area having a size that is close to and smaller than the subject area; a second determination unit configured to determine a second reference area having a size that is close to and larger than the subject area to determine an area to undergo the second image processing; and an estimation unit configured to estimate the subject area from the size and orientation of the first reference area.

2. The image processing apparatus according to claim 1, wherein the detection unit detects a face area and a human body area as the subject area, and the first determination unit makes the degree to which the first reference area is smaller than the subject area different between the face area and the human body area.

3. The image processing apparatus according to claim 2, wherein compared with the face area, the first determination unit determines the degree to which the first reference area is smaller than the subject area such that the human body area is much smaller.

4. The image processing apparatus according to claim 1, wherein the detection unit detects a face area and a human body area as the subject area, and the second determination unit makes the degree to which the second reference area is larger than the subject area different between the face area and the human body area.

5. The image processing apparatus according to claim 4, wherein compared with the face area, the second determination unit determines the degree to which the second reference area is larger than the subject area such that the human body area is much larger.

6. The image processing apparatus according to claim 1, wherein, The first image processing is relighting processing.

7. The image processing apparatus according to claim 1, wherein, The second image processing is contrast correction processing.

8. The image processing apparatus according to claim 1, wherein, The first determination unit and the second determination unit deform a subject model image based on the image and a detection result of the face area, and determine the first reference area and the second reference area.

9. An image processing method, comprising: detecting a subject area from an image; performing first image processing on the subject area; performing second image processing on an area other than the subject area; determining, in the subject area, a first reference area to undergo the first image processing, the first reference area having a size that is close to and smaller than the subject area; determining a second reference area having a size that is close to and larger than the subject area to determine an area to undergo the second image processing; and estimating the subject area from the size and orientation of the first reference area.

10. A non-transitory computer-readable storage medium storing a program for causing a computer to execute the processing of the image processing method according to claim 9.

11. A computer program product comprising a program for causing a computer to execute processing of the image processing method according to claim 9.

Images