Image processing device and method, electronic device, program and storage medium

Abstract

To perform exposure control so that a subject has appropriate brightness regardless of the state of the subject.SOLUTION: An image processing apparatus detects a first area indicating a subject included in an input image, and a second area having a predetermined first feature included in the image, and determines the state of the detected subject. When determining that the state of the subject satisfies a predetermined condition, the image processing apparatus calculates a photometric value by using a luminance value of a first superposition area where the first area and the second area are superposed on each other, and determines an exposure value on the basis of the calculated photometric value.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus and method, an electronic device, a program, and a storage medium, and more particularly to a method for determining a photometric value used for exposure control. [Background technology]

[0002] A known exposure control technique for digital cameras and other devices involves dividing a captured image into grid-like blocks for photometry, acquiring the brightness value for each block, and then controlling the exposure based on an evaluation value calculated from the average of the acquired brightness values. By calculating an exposure compensation value that converges this evaluation value to the appropriate brightness and feeding it back to exposure control such as aperture, shutter, and ISO, it is possible to maintain the appropriate brightness of the image being captured. Another known technique involves detecting the subject's face or head and using the detected area to calculate an exposure compensation value that ensures the face has the appropriate brightness. However, if the detected face area includes background or hair, these may affect the brightness, resulting in a deviation from the appropriate brightness.

[0003] Patent Document 1 proposes a mechanism for detecting pixels that represent skin color in a subject detection area using skin color detection, and calculating exposure compensation parameters based on the luminance values ​​of the detected pixels. This makes it possible to calculate an exposure compensation value that provides appropriate luminance in the face area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2005-148915 Summary of the Invention [Problem to be solved by the invention]

[0005] However, even when using skin color detection as described in Patent Document 1, there is a possibility that exposure correction may not be performed in the intended area. For example, clothing or tree trunks that are close in color to the skin are likely to be misdetected. Furthermore, areas where the underlying skin is easily visible, such as the back of the head, are likely to be detected as skin color, which may result in inappropriate exposure correction.

[0006] Furthermore, when attempting to calculate an appropriate exposure value for a subject using the area obtained by subject detection, variations in the position and size of the detection frame can result in the background or hair being included in the detection frame, as shown in Figure 12(a), for example. In particular, if the frame shifts in a situation where the brightness differs, such as when the face is dark and the background is bright in a backlit scene, the brightness of the dark face to be extracted and the brightness of the bright background are mixed and averaged, resulting in a higher-than-expected brightness value being calculated. When the brightness value is calculated in this state with the frame shifted, the difference from the target brightness will be inaccurate, making it impossible to set an appropriate exposure value. Similar deviations can also occur when the frame overlaps with accessories such as masks or sunglasses.

[0007] In contrast, as shown in Figure 12(b), if exposure values ​​are calculated for only skin using skin detection, it is possible to avoid brightness discrepancies caused by the background or clothing. However, as shown in Figure 12(c), areas where the underlying skin is easily visible, such as the back of the head, are also likely to be detected as skin, raising the concern that an appropriate exposure value may not be set.

[0008] The present invention has been made in consideration of the above problems, and has as its object to control exposure so that the subject is illuminated with appropriate brightness regardless of the state of the subject. [Means for solving the problem]

[0009] In order to achieve the above object, the image processing device of the present invention comprises an input means for inputting an image, a first detection means for detecting a first region showing a subject included in the image, a second detection means for detecting a second region having a predetermined first characteristic included in the image, a determination means for determining a state of the subject, a calculation means for calculating a photometric value using a luminance value of a first overlapping region where the first region and the second region overlap when the determination means determines that the state of the subject satisfies a predetermined condition, and a determination means for determining an exposure value based on the photometric value. When the determining means determines that the state does not satisfy the condition, the determining means determines an exposure value using a photometric value calculated using a luminance value of the first area. do. [Effects of the Invention]

[0010] According to the present invention, exposure control can be performed so that the subject has appropriate brightness regardless of the state of the subject. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing the functional configuration of an imaging apparatus according to a first embodiment of the present invention. [Figure 2] 4 is a flowchart of an imaging process according to the first embodiment. [Figure 3] FIG. 3 is a diagram illustrating divided regions and brightness values ​​according to the first embodiment. [Figure 4] FIG. 10 is a block diagram showing the functional configuration of an imaging apparatus according to a second embodiment. [Figure 5] 10 is a flowchart of an imaging process according to the second embodiment. [Figure 6] 10A and 10B are diagrams showing distribution histograms of brightness of the skin region of the side of the face and the back of the head, and brightness of the hair region in the second embodiment. [Figure 7] 5A and 5B are diagrams showing a method for detecting a face angle in the first and second embodiments. [Figure 8] 4A and 4B are diagrams showing a method for detecting an attachment in the first and second embodiments. [Figure 9] 10 is a flowchart of an imaging process according to the third embodiment. [Figure 10]10 is a flowchart illustrating a method using a plurality of thresholds for determining the size of the area of ​​a skin region according to the third embodiment. [Figure 11] 10A and 10B are diagrams for explaining a method using a plurality of thresholds for determining the size of the area of ​​a skin region according to the third embodiment. [Figure 12] FIG. 10 is a diagram for explaining a problem in the related art. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] First Embodiment A first embodiment of the present invention will be described below. Imaging device configuration FIG. 1 is a block diagram showing the functional configuration of an image capturing apparatus 100 according to the first embodiment. 1, the configurations represented as blocks can be realized by integrated circuits (ICs) such as ASICs or FPGAs, by discrete circuits, or by a combination of a memory and a processor that executes a program stored in the memory. Also, one block may be realized by multiple integrated circuit packages, or multiple blocks may be realized by a single integrated circuit package. Furthermore, the same block may be implemented in different configurations depending on the operating environment, required capabilities, etc.

[0014] In the following embodiments, the present invention will be described as being implemented in an imaging device such as a digital camera, but the present invention can also be implemented in any electronic device having an imaging function. Such electronic devices include not only imaging devices but also computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, drive recorders, etc. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0015] The operation unit 101 is composed of various operation members such as switches and buttons that are operated by the operator of the imaging device 100 to input various instructions, and includes a shutter switch and a touch sensor (which can be operated by touching the display device). The shutter switch includes, for example, SW1, which is turned on during operation (e.g., halfway pressed) and instructs preparation for shooting, and SW2, which is turned on when operation is completed (e.g., fully pressed) and instructs the start of a series of imaging processing operations.

[0016] The control unit 102 includes a CPU, a nonvolatile memory, and a RAM, and can realize various functions by the CPU executing programs stored in the nonvolatile memory. For example, the control unit 102 controls the operation of each unit shown in FIG. 1 in response to instructions from the operation unit 101. The sensor unit 103 receives light incident thereon through an optical system including a mechanical mechanism 1091 such as a lens 1081 and an aperture, converts the light into an electric charge according to the amount of light, and outputs an analog image signal. The sensor unit 103, lens 1081, and mechanical mechanism 1091 constitute an imaging unit.

[0017] The A / D conversion unit 104 performs sampling, gain adjustment, A / D conversion, etc. on the analog image signal output from the sensor unit 103, and outputs the resulting digital image signal. The image processing unit 105 performs various image processing on the digital image signal output from the A / D conversion unit 104 and outputs the processed digital image signal (image data). For example, the image processing unit 105 converts the digital image signal output from the A / D conversion unit 104 into a YUV image signal and outputs it. The luminance calculation unit 106 calculates the luminance using the image data output from the image processing unit 105, and calculates the difference from the appropriate luminance.

[0018] The subject detection unit 107 detects subjects within an image using image data obtained from the image processing unit 105. The subject detected here is assumed to be a person, and the area of ​​the subject's face and head within the image is acquired. Subject detection can be performed using known methods, such as a method of extracting an area by pattern matching from the contour shape of the human body, a method of detecting important organs with distinctive features such as the eyes, nose, and mouth and then detecting the head area including these, or a method using an algorithm that learns a person's facial area through machine learning. For example, in a machine learning method, learning is performed by relating concepts of each granularity, from the overall image of the object to the details, in a hierarchical structure. When learning about people, images of people of various races, ages, genders, facial orientations, and hair types are used for learning. In addition to faces, the detection area can also be divided into categories such as head, torso, limbs, upper body, lower body, and whole body.

[0019] The subject state determination unit 117 determines the facial state of the subject detected by the subject detection unit 107. The facial state to be determined here includes, for example, the orientation of the face (up, down, left, or right), the presence or absence of hair or beard, and the presence or absence of accessories such as a mask or sunglasses.

[0020] The skin detection unit 118 detects skin within an image and acquires its area. Skin detection can be performed using known methods, such as extracting a predetermined color gamut defined as skin color or using an algorithm that learns skin areas through machine learning. For example, in a machine learning technique, concepts of each granularity, from the overall image of the object to the details, are related to each other in a hierarchical structure for learning. By using images of people of various races, ages, and genders to learn skin, it becomes possible to detect skin even if there are differences in the shade of skin color between people.

[0021] The display unit 115 is configured with a liquid crystal display or the like, and displays an image based on the digital image signal output from the A / D conversion unit 104 . The external connection unit 114 is a connection unit for connecting an external monitor, a personal computer, etc. For example, by connecting an external monitor via the external connection unit 114, it becomes possible to display an image displayed on the display unit 115 on the external monitor.

[0022] The AF processing unit 108 determines the focus state based on the image data output from the image processing unit 105, and controls the lens 1081 to adjust the focus.

[0023] The AE processing unit 109 controls the mechanical mechanism 1091 based on the difference in brightness between the appropriate brightness and the brightness calculated by the brightness calculation unit 106 using the image data output from the image processing unit 105. This controls, for example, the aperture value and the shutter speed. The AE processing unit 109 may also be configured to control the gain used in the gain adjustment performed by the A / D conversion unit 104. If the sensor unit 103 has an electronic shutter function, the shutter speed may be controlled by controlling the charge accumulation time in the sensor unit 103. Furthermore, it may be possible to determine whether or not to emit light from the flash unit 111 depending on the difference in brightness, and if it is determined that light should be emitted, to issue a light emission instruction to the EF processing unit 110. Note that the determination of light emission and the light emission instruction may be performed by the control unit 102 in cooperation with the AE processing unit 109. If it is determined that light should be emitted, the EF processing unit 110 causes the flash unit 111 to emit light in an amount that will provide appropriate brightness for the subject in response to the light emission instruction.

[0024] The encoder unit 112 converts the format of the image data output from the image processing unit 105 into a format such as JPEG, and outputs the converted data to the image recording unit 113 . The image recording unit 113 performs a process of recording the format-converted image data output from the encoder unit 112 in a memory (not shown) within the imaging device 100 or an external memory (not shown) inserted into the imaging device 100. The memory unit 116 temporarily stores image data being processed by the control unit 102, the image processing unit 105, the encoder unit 112, and the like. Based on the determination result of the subject state determination unit 117, the exposure compensation value calculation unit 119 calculates an exposure compensation value using the image signal of the area where the subject area (face area or head area) detected by the subject detection unit 107 and the skin area detected by the skin detection unit 118 overlap.

[0025] The image processing unit 105, brightness calculation unit 106, subject detection unit 107, subject state determination unit 117, skin detection unit 118, and exposure compensation value calculation unit 119 may each be realized by a processor executing software, or may be realized by dedicated hardware. Also, although the image processing unit 105, brightness calculation unit 106, subject detection unit 107, subject state determination unit 117, skin detection unit 118, and exposure compensation value calculation unit 119 are shown as being separate from the control unit 102, this is not limitative. At least some of the functions of the image processing unit 105, brightness calculation unit 106, subject detection unit 107, subject state determination unit 117, skin detection unit 118, and exposure compensation value calculation unit 119 may be included in the control unit 102. In that case, the functions included in the control unit 102 may be, for example, a CP U Non-volatile memory Li This is achieved by executing the stored program.

[0026] ●Exposure control Next, exposure control in the first embodiment by the imaging device 100 having the above configuration will be described along the flow of the photographing operation with reference to the flowchart in FIG.

[0027] First, in step S201, when the operator of the imaging device 100 turns on the power switch included in the operation unit 101, the control unit 102 detects this and starts supplying power to each unit that makes up the imaging device 100.

[0028] When power is supplied to each component of the imaging device 100, the shutter opens in step S202, and light enters the sensor unit 103 via the lens 1081 and mechanical mechanism 1091 arranged in front of the camera. The sensor unit 103 reads out charges accumulated by photoelectric conversion according to the amount of incident light, and outputs the charges to the A / D conversion unit 104 as an analog image signal.

[0029] The A / D conversion unit 104 performs sampling, gain adjustment, A / D conversion, etc. on the analog image signal output from the sensor unit 103 and outputs it as a digital image signal. The image processing unit 105 then performs various image processing on the digital image signal output from the A / D conversion unit 104 and outputs the processed digital image signal (hereinafter referred to as a "live image").

[0030] Next, in step S203, the luminance calculation unit 106 calculates the luminance value (image luminance value) of the entire live image output from the image processing unit 105. In this embodiment, first, the entire image is divided into lattice-shaped blocks, and the luminance values ​​of the pixels in each block are averaged to determine the luminance value of each block. Then, the calculated luminance value of each block is multiplied by a weight determined in advance for each block to determine the average luminance value, which is used as the image luminance value.

[0031] The image luminance value calculated here will be described with reference to FIG. Fig. 3(a) shows an example of a live image output from the image processing unit 105. Fig. 3(b) shows an example of dividing the live image shown in Fig. 3(a) into blocks, and Fig. 3(c) shows an example of the luminance value of each block shown in Fig. 3(b). The image luminance value is obtained by weighting and adding the luminance of each block shown in Fig. 3(c).

[0032] Next, in step S204, the subject detection unit 107 detects a subject. Here, a person is detected in the live image, and the face or head area of ​​the person is detected as the subject.

[0033] Next, in step S205, the state of the subject is determined by subject state determination unit 117. The state of the subject to be determined here refers to the face region, whole body region, face direction, the presence and type of accessories such as a mask or sunglasses, the presence and color of hair, the presence or absence of a beard, etc. In this embodiment, subject state determination unit 117 acquires information on important organs such as the eyes, nose, and mouth from the face region detected by subject detection unit 107, and determines the direction of the face based on the positions of these organs in the face region.

[0034] Specifically, as shown in FIG. 7, subject state determination section 117 sets feature point coordinates for important organs within the face region. For example, in the case of a face facing forward as shown in FIG. 7(a), as shown in FIG. 7(b), the eyes are positioned one by one relative to the center line of the detected face area, the nose is slightly below the center of the screen, and the mouth is located at the bottom of the screen relative to the center line. In this case, the subject state determination unit 117 determines that the face is facing forward (face angle is 0°). In FIG. 7(c), the eyes and nose are shifted slightly to the left overall compared to FIG. 7(b), so the subject state determination unit 117 determines the face angle to be 30°. In FIG. 7(d), the face is shifted further to the left, so it is determined to be 60°. In FIG. 7(e), the eyes are combined, and the nose and mouth are also shifted to the right edge of the frame, so it is determined to be 90°. Note that the angles shown in FIGS. 7(b) to 7(e) are merely examples for indicating how far the face is facing sideways, and any other indicators can be used as long as they allow the subject state determination unit 117 to grasp the relationship between the magnitude of the face direction.

[0035] The subject state determination unit 117 also determines whether or not an object such as a mask is being worn based on the presence or absence of organ information. As shown in FIG. 8(a), if only the eyes are detected as important organs within the face detection frame and the nose and mouth are covered by something other than skin, it is determined that the subject is wearing a mask. As shown in FIG. 8(b), if only the mouth and nose are detected as important organs within the face detection frame and the eyes are covered by something other than skin, it is determined that the subject is wearing sunglasses or eyeglasses. As for hair, the subject state determination unit 117 determines whether or not there is an object with a color other than skin above the head region, and classifies the hair into black hair, chestnut hair, blonde hair, red hair, white hair, etc. depending on the color.

[0036] Next, in step S206, skin present in the live image is detected by skin detection unit 118. Here, the detectable skin area refers to human skin regardless of differences in skin color or brightness, and is capable of detecting not only facial skin areas but also skin areas of hands and feet.

[0037] Next, in step S207, subject state determination unit 117 determines whether the face is facing forward (frontal face) or side (profile) based on the state of the subject determined in step S205. If it is determined to be facing forward or side, the process proceeds to step S208, and if it is determined not to be facing forward or side, the process proceeds to step S213.

[0038] In this embodiment, the profile condition is defined as a state in which the face is tilted from side to side within a range of approximately ±90° from the front position. This is assumed to be a level that ensures a sufficient area for acquiring the brightness value of the skin area of ​​the head region and suppresses exposure variations. If the angle exceeds 90°, it is determined that the area of ​​the skin area is not sufficient to acquire the brightness value.

[0039] In step S208, the luminance calculation unit 106 calculates an average skin luminance SkinY for the overlapping area between the face area detected in step S204 and the skin area detected in step S206. Here, a method for calculating the average skin luminance SkinY will be described with reference to the example shown in FIG.

[0040] First, the brightness calculation unit 106 determines in which of the divided blocks shown in Fig. 3(b) the face area detected by the subject detection unit 107 exists in the live image shown in Fig. 3(a). In this case, the block area corresponds to block area 301 in Fig. 3(d).

[0041] Next, the brightness calculation unit 106 determines in which of the divided blocks shown in Fig. 3(b) the skin area detected by the skin detection unit 118 exists in the live image shown in Fig. 3(a). In this case, the block area corresponds to the block area 302 in Fig. 3(f).

[0042] Then, the average skin luminance SkinY is calculated by averaging the luminance values ​​of the blocks commonly included in block area 301 and block area 302. In the example shown in Fig. 3, the average value of 174, 168, 197, and 204, 185.75, is obtained as the average skin luminance SkinY.

[0043] Here, if the number of skin regions is small, there is a concern that the average skin luminance SkinY may vary. Variation in the average skin luminance SkinY will lead to variation in the photometric value calculated later in S210, which will lead to variation in the final exposure. Therefore, the average skin brightness SkinY obtained can be calculated by averaging it over time. Specifically, the average skin brightness SkinY of the past N frames is used. In other words, if the average skin brightness in the current frame T is SkinY(T), then the average skin brightness SkinY is calculated as follows: SkinY=(SkinY(T)+SkinY(T-1)+SkinY(T-2)+SkinY(TN-1)) / N.

[0044] Alternatively, the previously calculated value may be used if the difference between the previous calculated value and the current value does not exceed a predetermined value. In other words, SkinY(T) is used when SkinY(T) - SkinY(T-1) is equal to or greater than a predetermined threshold, and SkinY(T-1) is used when it is less than the threshold. The process of suppressing the variation in SkinY may be performed only when the area of ​​the skin region is small. That is, if the area of ​​the skin region is less than TH1, the average value of SkinY for N frames is used, and if the area is TH1 or greater, the value of SkinY for the current frame calculated in S208 is used. Note that the method for suppressing variations in the average skin brightness SkinY is not limited to the above method. By using these methods, variations in the photometric value can be suppressed and exposure can be stabilized.

[0045] Next, in step S209, the luminance calculation unit 106 calculates a difference ΔBvFace from a predetermined target luminance value of the face using the average skin luminance SkinY. The difference ΔBvFace can be calculated by the following formula (1) where the target luminance value is ReferenceY: 。 ΔBvFace=LOG2(SkinY / ReferenceY) …(1)

[0046] Next, in step S210, the brightness calculation unit 106 calculates an appropriate photometric value using the difference ΔBvFace. If the photometric value calculated here is Bv, it is calculated using the following equation (2). Bv=CtrlBv+ΔBvFace+BvCorr…(2) Here, CtrlBv is the exposure value at which the image was captured, and BvCorr is a variety of correction values, including correction based on the degree of backlighting of the subject, correction based on the proportion of the sky that is a high-brightness area, and night view correction.

[0047] Thereafter, in step S211, the photometric value obtained in step S210 is stored for when the face is facing other than forward or sideways.

[0048] The photometric value may be calculated by further using the image luminance value obtained in step S203. When the image luminance value is further used, the photometric value Bv is calculated by the following equation (3) instead of equation (2). Bv=(CtrlBv+ΔBvEa)×a+(CtrlBv+ΔBvFace)×b +BvCorr...(3) Here, ΔBvEa is the difference between the image luminance value calculated in step S203 and the target luminance value. Also, a and b are adjustment coefficients that determine the usage rate of the overall image luminance and the luminance of the skin region, and a+b=1.0.

[0049] By calculating the photometric value taking the image brightness value into consideration as described above, it is possible to control the exposure so that the brightness of the entire image, not just the face, is appropriate.

[0050] On the other hand, in step S207, if the face orientation is other than front or profile, for example, as shown in Figure 12(c), and skin is detected due to the influence of the skin area underlying the hair, the photometric value calculated will be for the hair area, which will result in overexposure in the case of black hair. To avoid this, in step S213, it is decided to use the photometric value held in the most recent step S211.

[0051] Then, in step S212, the exposure correction value calculation unit 119 calculates an exposure value based on the photometric value obtained in step S210 or S213, and feeds it back to the AE processing unit 109, which then performs convergence control to the appropriate exposure.

[0052] In this way, when the detected subject state is a frontal or profile state and the skin area can be detected at a sufficient size, exposure compensation value calculation unit 119 calculates an exposure compensation value for the area where the detected subject area and skin detection area overlap. On the other hand, when the detected subject state is not a frontal or profile state and the skin area cannot be detected at a sufficient size, the stored photometric value is used.

[0053] In step S214, the control unit 102 determines whether SW1 has been turned ON by operating the shutter switch included in the operation unit 101. If SW1 has not been turned ON, the process returns to step S202, where a live image is acquired using the exposure value controlled in S212, and the above process is repeated for the acquired live image.

[0054] When SW1 is turned ON in step S214, the final photometric value is acquired in step S215. At this time, the AF processing unit 108 also performs AF processing using the image information at this time point, and adjusts the focus to the subject. hot The lens 1081 is controlled so as to

[0055] Then, in step S216, the control unit 102 determines whether SW2 has been turned ON by operating the shutter switch. If SW2 has not been turned ON, the process returns to step S214. If SW2 has been turned ON, the process proceeds to step S217, where the main exposure begins. The charge corresponding to the light incident on the sensor unit 103 via the lens and exposure mechanism is read out, and the charge is output to the A / D conversion unit 104 as an analog image signal.

[0056] The A / D conversion unit 104 performs sampling, gain adjustment, A / D conversion, etc. on the analog image signal output from the sensor unit 103 and outputs it as a digital image signal. The image processing unit 105 performs various image processing on the digital image signal and outputs the processed digital image signal.

[0057] The digital signal output from the image processing unit 105 is converted into a format such as JPEG by the encoder unit 112 and output to the image recording unit 113. The image recording unit 113 performs processing to record the format-converted image data in a predetermined memory.

[0058] As described above, according to the first embodiment, the subject area, subject information, and skin area are acquired, and it is determined whether or not a sufficient amount of skin area can be secured. By controlling the exposure so that the brightness of the skin area of ​​the face is appropriate, it is possible to control the exposure without being affected by the back of the head, attached objects, etc.

[0059] In the above example, the face orientation is used to determine whether the subject's state is such that a sufficient skin area can be secured, but the present invention is not limited to the face orientation. Information on the face area, the whole body area, and the presence or absence of accessories such as a mask or sunglasses determined by the subject state determination unit 117 may be used comprehensively to determine the state based on, for example, the size of the face area obtained from the face area and the whole body area, or based on whether the subject is wearing a mask. Furthermore, these determination methods may be combined to perform the determination.

[0060] <Second embodiment> Next, a second embodiment of the present invention will be described, which deals with exposure control in a case where the skin region cannot be detected sufficiently, such as in the case of a frontal face.

[0061] Imaging device configuration Fig. 4 is a block diagram showing the functional configuration of the imaging device 100 according to the second embodiment. In the configuration shown in Fig. 4, the same components as those shown in Fig. 1 are given the same reference numerals, and descriptions thereof will be omitted.

[0062] The hair detection unit 401 detects hair and beard on the screen and acquires their areas. Hair detection can be performed using known methods, such as extracting a predetermined color gamut defined as hair color, using an algorithm that learns hair areas through machine learning, or estimating the location of hair and beard from a detected face area and extracting a pattern matching area. For example, in machine learning techniques, learning is performed by relating concepts of each granularity, from the overall image of the object to the details, in a hierarchical structure. When learning hair, images of people of various races, ages, and genders are used for learning.

[0063] Based on the determination result of the subject state determination unit 117, the exposure compensation calculation unit 402 calculates an exposure compensation value using image signals of areas where the subject area (face area or head area) detected by the subject detection unit 107, the skin area detected by the skin detection unit 118, and the hair or beard area detected by the hair detection unit 401 are superimposed.

[0064] ●Exposure operation Next, exposure control in the second embodiment by the imaging device 100 having the above configuration will be described along the flow of the shooting operation with reference to the flowchart in Fig. 5. However, the same steps as those in the first embodiment shown in Fig. 2 are given the same step numbers, and descriptions thereof will be omitted as appropriate.

[0065] Based on the live image captured in step S202, the brightness value and image After the calculation of brightness values ​​(step S203), the detection of the subject (step S204), and the determination of the subject state (step S205) are completed, the skin detection unit 118 detects the skin area present in the live image in the next step S501. Here, the detectable skin area is defined as the skin of a person regardless of differences in skin color or brightness, and is capable of detecting not only the skin area of ​​the face but also the skin areas of the hands and feet. At the same time, the hair detection unit 401 detects hair regions present in the live image. Here, a detectable hair region is one that can detect a person's hair and beard regardless of race, age, or sex, and that can be detected without relying on color information such as white hair, black hair, or red hair.

[0066] Then, in step S502, brightness calculation unit 106 calculates an average skin brightness, SkinY, for the overlapping region of the face region detected in step S204 and the skin region detected in step S501, as shown in Fig. 3. Similarly, brightness calculation unit 106 calculates an average hair brightness, HairY, for the overlapping region of the face region detected in step S204 and the hair region detected in step S501.

[0067] Then, in step S503, the skin Bv value SkinBv and hair Bv value HairBv are calculated and stored from the average skin luminance SkinY and average hair luminance HairY calculated in step S502 and the target luminance value. When the exposure value at which the image was acquired is CtrlBv and the target luminance value is ReferenceY, the skin Bv value SkinBv and hair Bv value HairBv can be calculated using the following equation (4): SkinBv=CtrlBv+LOG2(SkinY / ReferenceY) HairBv=CtrlBv+LOG2(HairY / ReferenceY) …(4)

[0068] Next, in step S504, the brightness difference ΔBvSH between the skin and hair is calculated from the thus obtained Bv value of the skin, SkinBv, and the Bv value of the hair, HairBv, using the following equation (5). ΔBvSH=|SkinBv-HairBv| …(5)

[0069] Next, in step S207, subject state determination unit 117 determines whether the face is facing forward or sideways based on the state of the subject detected in step S205. If it is determined to be a frontal or sideways face, the process proceeds to step S506, and if it is determined not to be a frontal or sideways face, the process proceeds to step S505.

[0070] In step S506, the luminance calculation unit 106 calculates the difference ΔBvFace from the target luminance value of the face using the average skin luminance SkinY calculated in step S502. ΔBvFace is calculated based on equation (1).

[0071] Then, in step S507, the brightness difference ΔBvSH between the skin and hair calculated in step S504 is stored as the previous value preΔBvSH for the next time when it is determined that the face is other than a frontal face or a profile face.

[0072] Then, in step S210, the brightness calculation unit 106 calculates an appropriate photometric value using the difference ΔBvFace. The photometric value Bv is calculated based on equation (2). After that, in step S211, the photometric value obtained in step S210 is stored in case a face angle other than the front or profile is detected.

[0073] On the other hand, if it is determined in step S207 that the face direction is not the front or the profile, the process proceeds to step S505. For example, as shown in FIG. 12(c), if the back of the head is determined and skin is detected due to the influence of the skin area underlying the hair, there is a possibility that an inappropriate photometric value for the hair area will be calculated. In this case, the exposure for the hair area will become unstable. To avoid this, the brightness difference ΔBvSH between the skin and hair calculated in step S504 and the previous value PreΔB v It is determined whether the difference (amount of change) from SH is less than a predetermined threshold. If the difference is less than the predetermined threshold, the process proceeds to step S506, where the difference ΔBvFace is calculated using the average skin luminance SkinY calculated in step S502. If the difference is equal to or greater than the predetermined threshold, the process proceeds to step S213, where it is determined that the exposure value held in the most recent step S211 is to be used. This makes it possible to suppress fluctuations in exposure.

[0074] 6(a) is a conceptual diagram showing an example of the subject area, skin area, and hair area detected when the face is turned sideways, and FIG. 6(b) is a conceptual diagram showing an example of the distribution of brightness in the skin area and the distribution of brightness in the hair area at this time. Also, FIG. 6(c) is a conceptual diagram showing an example of the subject area, skin area, and hair area detected when the face is turned backwards. 4 and hair region 60 3 6(d) is a conceptual diagram showing an example of the luminance distribution of the skin region and the luminance distribution of the hair region at this time. It can be seen that when the face is facing backwards, the luminance distribution of the skin region 604 and the luminance distribution of the hair region 603 are closer than when the face is facing sideways. In other words, by comparing the previous luminance step ΔBvSH with the luminance step calculated this time, it is possible to determine whether the direction of the face has changed, etc.

[0075] Then, in step S212, the exposure correction value calculation unit 119 calculates an exposure value based on the photometric value obtained in step S210 or S213, and feeds it back to the AE processing unit 109, which then performs convergence control to the appropriate exposure.

[0076] As described above, according to the second embodiment, the subject area, subject information, skin area, and hair area are acquired, and it is determined whether a sufficient amount of skin area can be secured and whether the external light environment is stable, and by controlling the exposure to be appropriate for the face, it is possible to set appropriate exposure without being affected by the back of the head, attached items, etc.

[0077] In the first and second embodiments described above, the luminance value is calculated in units of blocks, but the present invention is not limited to this, and the luminance value may be calculated in units of pixels.

[0078] <Third embodiment> Next, a third embodiment of the present invention will be described, which deals with a case where the subject is a front face or a profile face, but the skin area is small. The configuration of the imaging device in the third embodiment is the same as that described in the first embodiment with reference to FIG. 1, and therefore a description thereof will be omitted.

[0079] ●Exposure operation Next, exposure control in the third embodiment by the imaging device 100 will be described along the flow of the shooting operation with reference to the flowchart in Fig. 9. However, the same step numbers are used for processes similar to those in the first embodiment shown in Fig. 2, and descriptions thereof will be omitted as appropriate.

[0080] Based on the live image captured in step S202, the brightness value and imageAfter the calculation of brightness values ​​(step S203), subject detection (step S204), subject state determination (step S205), and skin region detection (step S206) are completed, in step S207, subject state determination unit 117 determines whether the face is facing forward or sideways based on the state of the subject detected in step S205. If it is determined to be a frontal or sideways face, the process proceeds to step S901, and if it is determined not to be a frontal or sideways face, the process proceeds to step S213.

[0081] In step S901, it is determined whether the area of ​​the skin region is large. In step S207, it is determined whether a sufficient skin area can be secured to obtain the brightness of the skin region based on the face orientation conditions. However, even if the face is a frontal or profile face, if the face is small relative to the image, or in a scene where detection using a skin detection method using machine learning is difficult, the skin region may not be detected sufficiently. Therefore, in step S901, the area of ​​the skin region is evaluated if the face is determined to be a frontal or profile face. The area evaluated here is specifically the number of blocks when the image is divided into blocks of a predetermined size, or the ratio of the number of blocks of the skin region to the face or head region of the subject detected in step S204. In step S901, it is determined whether the area is equal to or greater than a predetermined threshold value TH. If it is determined to be equal to or greater than the threshold value TH, proceed to step S208. If it is determined to be less than the threshold value TH, proceed to step S902.

[0082] Note that the determination in step S901 may be performed using multiple thresholds. Specifically, the determination is made using a flag Flg as shown in FIGS. 10 and 11. First, in step S1001, it is determined whether the area of ​​the skin region is equal to or greater than a first threshold TH1. If the area is less than the first threshold TH1, Flg is set to True in step S1002. If the area of ​​the skin region is equal to or greater than the first threshold TH1 in subsequent step S1001, it is determined in S1003 whether the area of ​​the skin region is equal to or greater than a second threshold TH2. The second threshold TH2 is a predetermined threshold set to a value greater than the first threshold TH1. If the area of ​​the skin region is less than the second threshold TH2, the previous determination result is retained. If the area of ​​the skin region is equal to or greater than the second threshold TH, Flg is set to False in step S1004. In this way, a flag is generated that becomes True when the area falls below the first threshold TH1 and does not become False until the area becomes equal to or greater than a second threshold TH2 that is greater than the first threshold TH1.

[0083] 11 is a diagram illustrating the area of ​​the skin region and the flag Flg in the time domain. The flag Flg rises when the area of ​​the skin region falls below a first threshold TH1, and then falls when the area becomes equal to or greater than a second threshold TH2.

[0084] If the flag Flg set in this way is set, the determination in step S901 is NO, and if the flag Flg is not set, the determination in step S901 is YES.

[0085] As a result, in the determination in step S901 in FIG. 9, it is possible to calculate the photometric value using the brightness of the skin region with one threshold value or the brightness of the face region with one threshold value. Use to measure the light By changing the brightness calculation method less frequently than by switching between calculation methods, variations in photometric values ​​can be reduced.

[0086] Furthermore, when determining the area of ​​the skin region in step S901, there is a concern that the determination results may vary due to variations in the skin detection method using machine learning, which may result in frequent changes in whether the photometric value is calculated using the luminance of the skin region or the luminance of the face region, leading to flickering of the photometric value.

[0087] Therefore, the area of ​​the skin region may be averaged in the time direction. Specifically, the area of ​​the skin region for N frames is used as the average value. In other words, if the area of ​​the skin region in the current frame T is SkinNum(T), the area of ​​the skin region is calculated as SkinNum=(SkinNum(T-1)+SkinNum(T-2)+SkinNum(T-1)) / N.

[0088] Alternatively, if the difference between the previously calculated value and the current value does not exceed a predetermined value, the previously calculated value may be used. In other words, if SkinNum(T) - SkinNum(T-1) is equal to or greater than a predetermined threshold, SkinNum(T) is used, and if it is less than the threshold, SkinNum(T-1) is used. The method for suppressing the variation in SkinY is not limited to the above method.

[0089] This prevents frequent changes in the determination in step S901 of FIG. 9 as to whether the photometric value is calculated using the luminance of the skin region or the luminance of the face region, thereby reducing variation in the photometric value.

[0090] If it is determined in step S901 that the skin area is small, the luminance FaceY of the face area is calculated in step S902. The face area is the face area detected in step S204. Next, in step S903, it is determined whether photometry of the skin area of ​​the subject to be detected has been performed in the past. Photometry of the skin area refers to the calculation of a photometric value using the average skin luminance SkinY calculated in step S208. In this embodiment, in step S208, photometric calculation using the luminance of the skin area is performed only for faces that are frontal or profile and have a skin area detected to be equal to or greater than the threshold TH. If photometry of the skin area has not been performed in the past, there is no photometric value stored to be calculated in S210. In this case, the process proceeds to step S904, where the luminance FaceY of the face area calculated in S902 is used to calculate a difference ΔBvFace from a predetermined target luminance value of the face. The difference ΔBvFace is calculated based on equation (1) in which SkinY is replaced with FaceY.

[0091] Next, in step S210, the brightness calculation unit 106 calculates an appropriate photometric value using the difference ΔBvFace. The photometric value Bv is calculated based on equation (2).

[0092] When calculating the photometric value, the brightness of the face area is used, so it is hand, The face area may contain accessories such as a mask or sunglasses, or a hair area. For example, if the face area includes part of a white mask, the photometric value will be brighter than the luminance of the skin area, resulting in a final exposure that is darker than if the luminance of the skin area were used. On the other hand, if sunglasses are included, the photometric value will be darker than the luminance of the skin area, resulting in a final exposure that is brighter than if the luminance of the skin area were used. Similarly, when photometry is performed using a face frame for a profile shot, the inclusion of the hair area causes the calculated photometric value to be darker, resulting in a final exposure that is brighter than if the luminance of the skin area were used. In consideration of these factors, the contribution of the difference ΔBvFace to the calculation of the photometric value in step S210 is reduced.

[0093] Specifically, when calculating the photometric value, the following equation (6) is used instead of equation (2). Bv=(CtrlBv+ΔBvFace)×k+BvCorr …(6) Here, k is a coefficient ranging from 0 to 1.0. When the luminance of the face region is used, the contribution of the difference ΔBvFace to the photometric value Bv is reduced. On the other hand, when the luminance of the skin region SkinY is used, the contribution is not reduced. Therefore, in step S209, k is set to k1, and in step S904, k is set to k2. Here, k1 and k2 are predetermined values ​​that satisfy k1>k2, for example, k1=1.0 and k2=0.5. Through the above processing, the influence of the luminance of the face region is reduced when metering the photometry of the face region compared to when metering the photometry of the skin region, and therefore the influence of accessories such as masks and sunglasses and hair on the photometric value can be reduced.

[0094] Note that a method for suppressing the influence of the difference ΔBvFace may be applied to a value included in the exposure compensation value BvCorr. For example, the backlight compensation value AlphaCorr, which is one type of exposure compensation value BvCorr, is calculated as follows: AlphaCorr=(ΔBvFace-ΔBvEa)×α …(7) The backlight correction value AlphaCorr is a correction amount that is applied to the photometric value by calculating the degree of backlighting of the subject from the difference between the difference ΔBvFace between the luminance of the face area and the target value and the difference ΔBvEa between the overall luminance and the target value. α is a coefficient that adjusts how much the degree of backlighting is reflected in the photometric value. Instead of k, which is applied directly to the difference ΔBvFace described above, the value of α may be changed depending on whether the photometric value is calculated using the luminance of the skin area or the luminance of the face area.

[0095] Thereafter, in step S211, the photometric value obtained in step S210 is stored for when the face is facing other than front or side and the area of ​​the skin region is small.

[0096] The photometric value obtained in step S203 image The calculation may further use the brightness value. imageBy calculating the photometric value taking the brightness value into account, it is possible to control the exposure so that the brightness of the entire image, not just the face, is appropriate. In this case, the photometric value Bv is calculated based on the following formula (8), which adds the coefficient k used in formula (6) to formula (3). Bv=(CtrlBv+ΔBvEa)×a +(CtrlBv+ΔBvFace)×b×k+BvCorr…(8)

[0097] In S207, if the face orientation is other than front or profile, or if in step S903 photometry of the skin area has been performed in the past, i.e., if a photometric value calculated in step S210 exists, the process proceeds to step S213, where it is determined that the photometric value held in the most recent step S211 will be used.

[0098] Then, in step S212, the exposure correction value calculation unit 119 calculates an exposure value based on the photometric value obtained in step S210 or S213, and feeds it back to the AE processing unit 109, which then performs convergence control to the appropriate exposure.

[0099] As described above, according to the third embodiment, in a scene where a skin region is difficult to detect and there is a concern about variations in the photometric value using the luminance of the skin region, stable photometric calculation can be performed by using the luminance of the face region. Furthermore, if a sufficient number of skin regions are detected and the luminance of the skin region can be used, the photometric value of the most recent skin region can be used, making it possible to calculate an appropriate exposure using the skin region.

[0100] <Other embodiments> The present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device.

[0101] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0102] <Summary> The disclosure of this embodiment includes the following configuration.

[0103] (Item 1) an input means for inputting an image; a first detection means for detecting a first region representing a subject included in the image; a second detection means for detecting a second region having a predetermined first characteristic included in the image; a determination means for determining the state of the subject; a calculation means for calculating a photometric value using a luminance value of a first overlapping area where the first area and the second area overlap when the determination means determines that the state of the subject satisfies a predetermined condition; and a determining means for determining an exposure value based on the photometric value; 1. An image processing device comprising: (Item 2) 2. The image processing device according to item 1, characterized in that, when the determining means determines that the state does not satisfy the condition, the determining means determines the exposure value using the most recently determined photometric value. (Item 3) Item 1. The image processing device according to item 1, characterized in that, when the determination means determines that the state does not satisfy the condition, the determination means determines the exposure value using a photometric value calculated using the luminance value of the first area. (Item 4) 4. The image processing device according to item 3, wherein the calculation means, when calculating a photometric value using the luminance value of the first region, reduces the contribution of the luminance value of the first region to the photometric value. (Item 5) Item c. The image processing device according to item 1, wherein the calculation means calculates the photometric value based on a luminance value obtained by averaging a predetermined number of luminance values ​​obtained in the past. (Item 6) 2. The image processing device according to item 1, wherein the calculation means calculates the photometric value based on the amount of change in the luminance value obtained most recently. (Item 7) a third detection means for detecting a third region having a predetermined second feature included in the image and different from the first feature; Item 1 is an image processing device according to item 1, characterized in that, when the judgment means judges that the state of the subject does not satisfy a predetermined condition, if the change in the difference between the luminance value of the first overlapping region and the luminance value of the second overlapping region in which the first region and the third region overlap is less than a predetermined threshold, the determination means determines the exposure value using the photometric value, and if the change in the difference is equal to or greater than the predetermined threshold, the determination means determines the exposure value using the photometric value most recently calculated by the calculation means. (Item 8) 8. The image processing device according to item 7, wherein the subject is a human face, and the region having the second feature is hair and beard. (Item 9) 9. The image processing device according to any one of items 1 to 8, wherein the condition is a condition relating to the size of the second region included in the subject. (Item 10) 10. The image processing device according to item 9, wherein the condition regarding the size of the second region is whether the size of the second region is larger or smaller than a predetermined first threshold value. (Item 11) a predetermined first threshold value for the size of the second region and a predetermined second threshold value that is greater than the first threshold value; Item 10. The image processing device according to item 9, wherein the determination means does not change the determination result of the condition after the size of the second region becomes less than the first threshold value until the size becomes equal to or greater than the second threshold value. (Item 12) 10. The image processing device according to item 9, wherein the determination means determines whether the condition is met based on an average value of the sizes of a predetermined number of the second regions obtained in the past. (Item 13) 10. The image processing device according to item 9, wherein the determining means determines whether the condition is met based on the amount of change in size of the second region most recently obtained. (Item 14) 10. The image processing device according to item 9, wherein the size of the second region is the number of blocks when the image is divided into blocks each having a predetermined area. (Item 15) 10. The image processing device according to item 9, wherein the size of the second region is a ratio of the second region to the first region. (Item 16) The image processing device described in any one of items 1 to 9, characterized in that the subject is a human face, and the state of the subject includes at least one of the orientation of the face, the presence and type of attachment, and the presence and color of beard and hair. (Item 17) Item 17. The image processing device according to item 16, wherein the condition is that the orientation of the face is within a predetermined angle range from the front. (Item 18) 18. The image processing device according to any one of items 1 to 17, wherein the calculation means calculates the photometric value based on the difference between the luminance value of the first overlapping area and a predetermined target luminance value. (Item 19) 19. The image processing device according to any one of items 1 to 18, wherein the subject is a human face, and the region having the first feature is skin. (Item 20) 20. The image processing device according to any one of items 1 to 19, wherein the calculation means further calculates the photometric value using a luminance value of the entire image of the image. (Item 21) An image processing device according to any one of items 1 to 20, an imaging means for capturing and outputting an image; a control means for controlling exposure based on the exposure value determined by the determination means; An electronic device comprising: (Item 22) an input step of inputting an image; a first detection step of detecting a first region representing a subject included in the image; a second detection step of detecting a second region having a predetermined first characteristic included in the image; a determination step of determining a state of the subject; a calculation step of calculating a photometric value using a luminance value of a first overlapping area where the first area and the second area overlap when it is determined in the determination step that the state of the subject satisfies a predetermined condition; a determining step of determining an exposure value based on the photometric value; An image processing method comprising: (Item 23) a third detection step of detecting a third region having a predetermined second feature different from the first feature included in the image; Item 23. The image processing method according to item 22, characterized in that, in the determination step, if it is determined in the judgment step that the state of the subject does not satisfy a predetermined condition, and if the amount of change in the difference between the luminance value of the first overlapping region and the luminance value of a second overlapping region in which the first region and the third region overlap is less than a predetermined threshold, the exposure value is determined using the photometric value, and if the amount of change in the difference is equal to or greater than the predetermined threshold, the exposure value is determined using the photometric value most recently calculated in the calculation step. (Item 24) A program for causing a computer to function as each means of the image processing device according to any one of items 1 to 20. (Item 25) Item 25. A computer-readable storage medium storing the program described in item 24.

[0104] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0105] 101: operation unit, 102: control unit, 103: sensor unit, 104: A / D conversion unit, 105: image processing unit, 106: brightness calculation unit, 107: subject detection unit, 108: AF processing unit, 1081: lens, 109: AE processing unit, 1091: mechanical mechanism, 110: EF processing unit, 111: flash unit, 112: encoder unit, 113: image recording unit, 114: external connection unit, 115: display unit, 116: memory unit, 117: subject state determination unit, 118: skin detection unit, 119: 402: exposure compensation calculation unit, 401: hair detection unit

Claims

1. an input means for inputting an image; a first detection means for detecting a first region representing a subject included in the image; a second detection means for detecting a second region having a predetermined first characteristic included in the image; a determination means for determining the state of the subject; a calculation means for calculating a photometric value using a luminance value of a first overlapping area where the first area and the second area overlap when the determination means determines that the state of the subject satisfies a predetermined condition; and a determining means for determining an exposure value based on the photometric value; and An image processing device characterized in that, when the judgment means determines that the state does not satisfy the condition, the determination means determines an exposure value using a photometric value calculated using the luminance value of the first area.

2. 2. The image processing device according to claim 1, wherein the calculation means applies a coefficient to the luminance value of the first region when calculating a photometric value using the luminance value of the first region to a value smaller than a coefficient to be applied to the luminance value of the first region when calculating a photometric value using the luminance value of the second region.

3. 2. The image processing device according to claim 1, wherein the condition is a condition relating to the size of the second region included in the subject.

4. 4. The image processing apparatus according to claim 3, wherein the condition regarding the size of the second region is whether the size of the second region is larger than a predetermined first threshold value.

5. a predetermined first threshold value for the size of the second region and a predetermined second threshold value that is greater than the first threshold value; 4. The image processing device according to claim 3, wherein the determining means does not change the determination result of the condition until the size of the second region becomes equal to or greater than the second threshold value after the size of the second region becomes less than the first threshold value.

6. 4. The image processing apparatus according to claim 3, wherein the determining means determines whether the condition is met based on an average value of the sizes of a predetermined number of the second regions obtained in the past.

7. The image processing device according to claim 3, characterized in that the determination means determines the condition based on an amount of change indicating the difference between the size of the second region most recently obtained and the size of the second region obtained one before.

8. 4. The image processing apparatus according to claim 3, wherein the size of the second region is the number of blocks obtained by dividing the image into blocks each having a predetermined area.

9. 4. The image processing apparatus according to claim 3, wherein the size of the second region is a ratio of the second region to the first region.

10. 2. The image processing device according to claim 1, wherein the subject is a human face, and the state of the subject includes at least one of the orientation of the face, the presence and type of attachment, and the presence and color of beard and hair.

11. 11. The image processing device according to claim 10, wherein the condition is that the face direction is within a predetermined angle range from the front.

12. 2. The image processing apparatus according to claim 1, wherein the calculation means calculates the photometric value based on the difference between the luminance value of the first overlapping area and a predetermined target luminance value.

13. 2. The image processing apparatus according to claim 1, wherein the subject is a human face, and the second region having the first feature is skin.

14. 2. The image processing apparatus according to claim 1, wherein said calculation means further calculates said photometric value using a luminance value of the entire image of said image.

15. An input means for inputting an image; a first detection means for detecting a first region representing a subject included in the image; a second detection means for detecting a second region having a predetermined first characteristic included in the image; a third detection means for detecting a third region having a predetermined second feature included in the image, the second feature being different from the first feature; a determination means for determining the state of the subject; a calculation means for calculating a photometric value using a luminance value of a first overlapping area where the first area and the second area overlap when the determination means determines that the state of the subject satisfies a predetermined condition; and a determining means for determining an exposure value based on the photometric value; and The image processing device is characterized in that, when the judgment means judges that the state of the subject does not satisfy a predetermined condition, the determination means compares the most recent difference between the luminance value of the first overlapping area and the luminance value of the second overlapping area where the first area and the third area overlap, and if the change in the difference between the most recent difference and the difference obtained just before the most recent difference is less than a predetermined threshold, the determination means determines the exposure value using the photometric value, and if the change in the difference is greater than or equal to the predetermined threshold, the determination means determines the exposure value using the photometric value calculated most recently by the calculation means.

16. The image processing device described in Claim 15, characterized in that the subject is a human face and the area having the second feature is hair and a beard.

17. The image processing device according to claim 15, wherein the condition is a condition relating to the size of the second area included in the subject.

18. An image processing device as described in Claim 17, characterized in that the condition regarding the size of the second region is whether the size of the second region is larger or smaller than a predetermined first threshold value.

19. A method for detecting a size of the second region, the method comprising: a first predetermined threshold value for the size of the second region; and a second predetermined threshold value for the size of the second region, the second threshold value being greater than the first threshold value; 18. The image processing device according to claim 17, wherein the determining means does not change the determination result of the condition until the size of the second region becomes equal to or greater than the second threshold value after the size of the second region becomes less than the first threshold value.

20. The image processing device described in Claim 17, characterized in that the judgment means judges the condition based on an average value of the sizes of a predetermined number of the second regions obtained in the past.

21. The image processing device described in Claim 17, characterized in that the judgment means judges the condition based on a change amount indicating the difference between the size of the second area most recently obtained and the size of the second area most recently obtained.

22. An image processing device as described in Claim 17, characterized in that the size of the second area is the number of blocks when the image is divided into blocks each having a predetermined area.

23. The image processing device according to claim 17, wherein the size of the second region is a ratio of the second region to the first region.

24. The image processing device described in Claim 15, characterized in that the subject is a human face, and the state of the subject includes at least one of the direction of the face, the presence and type of attachment, and the presence and color of beard and hair.

25. The image processing device described in Claim 24, characterized in that the condition is that the orientation of the face is within a predetermined angle range from the front.

26. The image processing device described in Claim 15, characterized in that the calculation means calculates the photometric value based on the difference between the luminance value of the first overlapping area and a predetermined target luminance value.

27. The image processing device of claim 15, wherein the subject is a human face and the second area having the first characteristic is skin.

28. The image processing device according to claim 15, wherein the calculation means further calculates the photometric value using the brightness value of the entire image of the image.

29. an image processing device according to any one of claims 1 to 28; an imaging means for capturing and outputting an image; a control means for controlling exposure based on the exposure value determined by the determination means; An electronic device comprising:

30. an input step of inputting an image; a first detection step of detecting a first region representing a subject included in the image; a second detection step of detecting a second region having a predetermined first characteristic included in the image; a determination step of determining a state of the subject; a calculation step of calculating a photometric value using a luminance value of a first overlapping area where the first area and the second area overlap when it is determined in the determination step that the state of the subject satisfies a predetermined condition; a determining step of determining an exposure value based on the photometric value; and An image processing method characterized in that, if it is determined in the judgment step that the state does not satisfy the condition, in the determination step, an exposure value is determined using a photometric value calculated using the luminance value of the first area.

31. An input step for inputting an image; a first detection step of detecting a first region representing a subject included in the image; a second detection step of detecting a second region having a predetermined first characteristic included in the image; a third detection step of detecting a third region having a predetermined second feature included in the image, the second feature being different from the first feature; a determination step of determining a state of the subject; a calculation step of calculating a photometric value using a luminance value of a first overlapping area where the first area and the second area overlap when it is determined in the determination step that the state of the subject satisfies a predetermined condition; a determining step of determining an exposure value based on the photometric value; and In the determination step, when it is determined in the judgment step that the state of the subject does not satisfy a predetermined condition, if the change in the difference between the luminance value of the first overlapping area and the luminance value of the second overlapping area where the first area and the third area are overlapped, between the most recent difference and the difference obtained immediately before, is less than a predetermined threshold, the exposure value is determined using the photometric value, and if the change in the difference is equal to or greater than the predetermined threshold, the exposure value is determined using the photometric value calculated most recently in the calculation step.

32. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 28.

33. A computer-readable storage medium storing the program according to claim 32.

Images