Image processing device, image processing method, camera device and machine-readable medium

Abstract

The present invention relates to an image processing device, an image processing method, a camera device and a machine-readable medium. The image processing device comprises: a first detection unit, used to detect a predetermined partial area from image data using a first method and a second method respectively; a second detection unit, used to detect a predetermined feature area from the data of the detected partial area. The device selects one of the detection result of the feature area of ​​the partial area detected using the first method and the detection result of the feature area of ​​the partial area detected using the second method as the final detection result of the feature area, wherein the partial area is related to the same subject.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method, an imaging device and a machine-readable medium, and in particular to a technology for detecting a feature area from an image. Background Art

[0002] A technique for detecting a partial region (feature region) including a predetermined feature from an image is used in various fields. For example, in an image pickup device such as a digital camera, focus adjustment is performed to focus on the feature region, and exposure is controlled so that the brightness of the feature region is appropriate. In Japanese Patent Laid-Open No. 2008-129664, an image processing device for detecting an eye region as a feature region is described.

[0003] In Japanese Patent Laid-Open No. 2008-129664, a face region is first detected from an image, and then an eye region is detected from the detected face region. By first detecting a partial region expected to contain a feature region to be detected eventually and limiting the region used to detect the feature region in this manner, for example, the processing speed and the detection accuracy can be improved. However, unless the partial region can be correctly detected, the detection of the feature region also results in failure. Summary of the invention

[0004] In view of such problems in the related art, embodiments of the present invention provide an image processing apparatus, an image processing method, and an imaging apparatus capable of detecting a feature area with high accuracy using less processing.

[0005] According to one aspect of the present invention, there is provided an image processing device, comprising: a first detection component for detecting a predetermined partial area from image data using a first method and a second method respectively; a second detection component for detecting a predetermined feature area from the data of the detected partial area; and a selection component for selecting one of a detection result of the feature area for the partial area detected using the first method and a detection result of the feature area for the partial area detected using the second method as a final detection result of the feature area, wherein the partial areas are related to the same subject.

[0006] According to another aspect of the present invention, there is provided an imaging device, comprising: an imaging component for generating image data; an image processing device according to the present invention, which uses the image data generated by the imaging component; and a control component for performing focus detection and / or exposure control on the detection result of the feature area selected by the selection component of the image processing device.

[0007] According to another aspect of the present invention, there is provided an image processing method performed by an image processing device, comprising: detecting a predetermined partial area from image data using a first method and a second method respectively; detecting a predetermined feature area from the data of the detected partial area; and selecting one of the detection result of the feature area for the partial area detected using the first method and the detection result of the feature area for the partial area detected using the second method as the final detection result of the feature area, wherein the partial areas are related to the same subject.

[0008] According to another aspect of the present invention, there is provided a machine-readable medium storing a program for causing a computer to perform the image processing method according to the present invention.

[0009] Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings). BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a functional block diagram of a camera as an example of an image processing apparatus according to an embodiment.

[0011] Figure 2 is a functional block diagram of a characteristic region detection unit of the first embodiment.

[0012] Figure 3 It is a flowchart related to the operation of the characteristic region detection unit of the first embodiment.

[0013] Figure 4 It is a flowchart related to the operation of the characteristic region detection unit of the first embodiment.

[0014] Figure 5 is a flowchart related to the operation of the camera control unit of the first embodiment.

[0015] Figure 6 is a functional block diagram of a characteristic region detection unit of the second embodiment.

[0016] Figure 7 It is a flowchart related to the operation of the characteristic region detection unit of the second embodiment. DETAILED DESCRIPTION

[0017] Embodiments will be described in detail below with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but there is no limitation that the present invention requires all such features, and a plurality of such features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are given to the same or similar structures, and their repeated descriptions are omitted.

[0018] Note that in the embodiments described below, the present invention is embodied as a digital camera system with interchangeable lenses. However, the present invention can be applied to any electronic device capable of processing image data. Examples of such electronic devices include cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smart phones, game consoles, robots, drones, and driving recorders. These are examples, and the present invention can be applied to other electronic devices.

[0019] First embodiment

[0020] Figure 1 1 is a block diagram showing the functional structure of a digital camera system with interchangeable lenses (hereinafter referred to as a camera) as an example of an image processing apparatus according to a first embodiment of the present invention. The camera includes a main body 120 and an interchangeable lens 100. Note that the lens may not be interchangeable.

[0021] The interchangeable lens 100 that forms an optical image of a subject includes a lens unit 101. The lens unit 101 includes a main imaging optical system 102, an aperture 103, and a focus lens 104. Note that the main imaging optical system 102 and the focus lens 104 are actually composed of a plurality of lenses. The lens unit 101 also includes a motor and an actuator, etc., to drive movable components such as the aperture 103 and the focus lens 104. The lens unit 101 may also include an image stabilization lens, a zoom lens, and other movable components.

[0022] The lens control unit 111 includes a microprocessor, a ROM, and a RAM, and realizes various functions by having a program stored in the ROM be read out to the RAM and executed by the microprocessor. Figure 1 Functional blocks 112 and 113 shown within the lens control unit 111 in FIG. 1 schematically illustrate representative functions implemented by the execution of a program by the microprocessor of the lens control unit 111 .

[0023] The lens control unit 111 communicates with the camera control unit 131 and controls driving of the focus lens 104 and the aperture 103 based on instructions from the camera control unit 131. In addition, the lens control unit 111 provides the camera control unit 131 with information related to the lens unit 101, such as the position and f-number of the focus lens 104.

[0024] In the main body 120, an image sensor 122 such as a complementary metal oxide semiconductor (CMOS) image sensor includes a plurality of pixels arranged in a two-dimensional array. The image sensor 122 photoelectrically converts an optical image of a subject formed by the interchangeable lens 100 via the plurality of pixels and outputs a set of electrical signals (analog image signals). A shutter 121 is provided between the interchangeable lens 100 and the image sensor 122. Note that in the case where the aperture 103 also serves as a shutter, the shutter 121 is unnecessary.

[0025] The analog image signal output by the image sensor 122 is input to an analog front end (AFE) 123. After applying noise reduction processing and / or AD conversion processing to the analog image signal, the AFE 123 outputs the analog image signal as a digital image signal (image data).

[0026] The signal processing circuit 124 applies preset image processing to the image data output by the AFE 123 and generates a signal or image data, obtains and / or generates various information, etc. The signal processing circuit 124 may be, for example, dedicated hardware such as an application specific integrated circuit (ASIC) designed to implement a specific function, or may be configured to implement a specific function via a programmable processor such as a digital signal processor (DSP) executing software.

[0027] Image processing that the signal processing circuit 124 can apply to image data includes color interpolation processing, correction processing, detection processing, data modification processing, evaluation value calculation processing, and the like.

[0028] The color interpolation process is a process for interpolating the value of a color component that is insufficient found in pixel data forming image data, and is also called a demosaic process.

[0029] The correction processing includes white balance adjustment, processing for correcting the influence of optical aberration of the lens unit 101, processing for correcting color tone, and the like.

[0030] The detection processing includes processing for detecting a characteristic region, processing for tracking a characteristic region, processing for recognizing a person, and the like.

[0031] The data modification process includes scaling processing, display image data generation processing, encoding and decoding processing, header information generation processing, and the like.

[0032] The evaluation value calculation process is a process for generating a signal or an evaluation value used in automatic focus detection (AF), a process for generating an evaluation value used in automatic exposure control (AE), and the like.

[0033] Note that these are merely examples of image processing that can be applied to image data by the signal processing circuit 124, and these examples are not intended to limit the image processing applied to image data by the signal processing circuit 124. In addition, one or more of the image processing given as the above examples may be performed by the camera control unit 131.

[0034] The image data after the image processing in the signal processing circuit 124 is output to one or more of the camera control unit 131 , the display unit 171 , and the memory card 125 according to the application.

[0035] The characteristic region detection unit 141 Figure 1 , a functional block that performs processing for detecting feature areas is schematically shown as having the function of the signal processing circuit 124. The feature area detection unit 141 in the present embodiment is configured to ultimately detect an eye area or a pupil area as a feature area. However, the feature area detection unit 141 may detect other parts of the face (such as a nose or a mouth, etc.). The feature area detection unit 141 first detects a partial area (a face or a head area in the present example) in the image data that is expected to contain a feature area. Then, the feature area detection unit 141 detects feature areas from each partial area. The feature area detection unit 141 outputs detection results (for example, information indicating the position and size of the detected area) related to the partial area and the feature area to the camera control unit 131.

[0036] The camera control unit 131 is, for example, a microprocessor, and realizes various functions by programs stored in the ROM 133 being read out to the RAM 132 and executed by the microprocessor. Figure 1 Functional blocks 150 to 154 shown within the camera control unit 131 in FIG. 1 are schematically shown representative functions realized by execution of a program by a microprocessor of the camera control unit 131 .

[0037] The ROM 133 is, for example, a rewritable nonvolatile memory and stores programs, setting values, GUI data, etc. executable by the microprocessor of the camera control unit 131. The RAM 132 is used to temporarily hold values ​​required for programs executed by the microprocessor of the camera control unit 131 and values ​​required during execution of the programs.

[0038] Both the main body 120 and the interchangeable lens 100 are provided with mounts that engage with each other. The mount of the main body 120 and the mount of the interchangeable lens 100 are provided with mount contact units 114, 161 that are configured to contact with each other when the mounts engage. By bringing the mount contacts 114, 161 into contact with each other, power can be supplied from the main body 120 to the interchangeable lens 100, and communication between the camera control unit 131 and the lens control unit 111 can be achieved.

[0039] The lens control unit 111 and the camera control unit 131 communicate by serial communication at a predetermined timing via the mount contact portions 114, 161. This communication allows focus lens driving information, iris driving information, and the like to be sent from the camera control unit 131 to the lens control unit 111, and allows focus lens position, f-number, focal length, and the like to be sent from the lens control unit 111 to the camera control unit 131.

[0040] The subject determination unit 150 of the camera control unit 131 determines the same subject based on the detection result of the partial area. In addition, the position determination unit 151 determines the characteristic area based on the detection result of the characteristic area and the determination result of the subject determination unit 150. The display frame setting unit 152 sets the detection frame displayed on the display unit 171 described below. In addition, the subject setting unit 153 notifies the focus detection unit 154 of information for identifying the image area of ​​the subject (target subject) on which the lens unit 101 is to focus. These functional blocks 150 to 153 operate based on the detection result of the characteristic area detection unit 141.

[0041] The focus detection unit 154 adjusts the position of the focus lens 104 of the lens unit 101 to focus on the image area based on the information provided by the subject setting unit 153. The focus detection unit 154 can adjust the position of the focus lens 104 via a known method. For example, the focus detection unit 154 is capable of performing focus detection by contrast. The focus detection unit 154 moves the position of the focus lens 104 in stages, captures an image, and obtains a contrast evaluation value related to the image area to be focused on in the image data obtained from the captured image from the signal processing circuit 124. In addition, the focus detection unit 154 obtains the position of the focus lens 104 with the maximum contrast evaluation value, and instructs the lens control unit 111 to drive the focus lens 104 to the position.

[0042] Note that when the main body 120 generates an image signal for phase detection AF, the focus detection unit 154 can perform focus detection by phase detection. For example, in other examples, the main body 120 includes a sensor for phase detection AF, or the image sensor 122 has a structure suitable for image plane phase detection AF.

[0043] Note that in the case where an instruction to drive the focus lens 104 is sent from the focus detection unit 154 to the lens control unit 111, the instruction may be an instruction of a target position of the focus lens 104, or may be an instruction of a difference (such as a defocus amount) between the current position and the target position. The lens control unit 111 (focus lens control unit 113) controls the driving of the focus lens 104 based on the information received via the instruction from the camera control unit 131. In this way, the camera control unit 131 controls the driving of the focus lens 104 via the focus lens control unit 113.

[0044] The display unit 171 is a touch display including a video memory, a display controller, a display device, and a touch panel 172. In the present embodiment, the display unit 171 is used as an electronic viewfinder (EVF) by continuing to shoot a video when the device is in a shooting standby state and displaying the shot video in real time on the display unit 171. In addition, a GUI such as a menu screen and information such as setting values ​​are displayed on the display unit 171. In addition, image data read out from the memory card 125 and decoded by the signal processing circuit 124 is also displayed on the display unit 171.

[0045] The display unit 171 of the present embodiment as a touch display can receive a touch operation performed on the touch panel 172. The camera control unit 131 detects the operation performed on the touch panel 172, recognizes the touch operation based on the temporal change of the touch position, etc., and executes processing corresponding to the recognized touch operation.

[0046] The release switch (SW) 181 is an input device to which instructions related to still image shooting are input. For example, the release SW 181 includes a switch (SW1) that is turned on in a half-pressed state, and a switch (SW2) that is turned on in a fully pressed state. The camera control unit 131 recognizes that SW1 is turned on as a preparation instruction for still image shooting, and performs AF and AE processing. In addition, the camera control unit 131 recognizes that SW2 is turned on as a start instruction for still image shooting, and controls the still image shooting process and the generation process of image data for recording, the recording process of the image data, etc. The image data for recording is recorded on the memory card 125.

[0047] Next, the structure and operation of the characteristic region detection unit 141 of the present embodiment will be described. Figure 2141 is a diagram showing the function of the characteristic region detection unit 141 which is further subdivided and as a functional block. As described above, the characteristic region detection unit 141 is a function of the signal processing circuit 124. Therefore, the functional block of the characteristic region detection unit 141 is also a function of the signal processing circuit 124.

[0048] The characteristic region detection unit 141 includes a face detection unit 201 and a head detection unit 202 serving as first detection means for detecting partial regions, and a first pupil detection unit 203 and a second pupil detection unit 204 serving as second detection means for detecting characteristic regions from the partial regions.

[0049] The face detection unit 201 applies a known face detection process (first method) to the image data, and detects a partial area (face area) containing facial features. The specific details of the method for detecting a face are not particularly limited. For example, a method based on detecting a feature-like edge or pattern in a face area may be used, or a method using a neural network trained on a face area via machine learning may be used. Specifically, the detection accuracy of the face area depends on the detection accuracy of the characteristic parts or organs of the face (such as eyes, nose, and mouth, etc.). Therefore, in the case where parts of these parts or organs cannot be detected, the face area detection tends to fail, and the possibility of false detection tends to increase.

[0050] The head detection unit 202 applies a known head detection process (second method) to the image data, and detects a partial area containing head features (head area). The method for detecting the head area is not particularly limited. For example, a method for detecting an area of ​​a predetermined shape such as an omega (Ω) shape may be used, or a method using a neural network trained on the head area via machine learning may be used. In addition, the head area may be detected in conjunction with detecting other parts of the human body, such as the limbs or the trunk. Since the head area is mainly detected based on the contour shape, it does not rely on the detection of characteristic parts or organs of the face. Therefore, even in the case where the face detection unit 201 fails to detect the face area, the head detection unit 202 can successfully and correctly detect the head.

[0051] The first pupil detection unit 203 detects an eye region or a pupil region from the face region detected by the face detection unit 201. The second pupil detection unit 204 detects an eye region or a pupil region from the head region detected by the head detection unit 202. The first pupil detection unit 203 and the second pupil detection unit 204 may use different detection methods, or may use the same detection method. In the case of using the same detection method, one pupil detection unit may be used to detect the eye region or the pupil region for the face region and the head region. The detection method is not particularly limited. For example, a method using pattern matching may be used, or a method using a neural network trained on the eye region or the pupil region via machine learning may be used.

[0052] Will use Figure 3 The operation of the face detection unit 201 and the first pupil detection unit 203 will be explained using the flowchart of Figure 4 The operation of the head detection unit 202 and the second pupil detection unit 204 is described in detail with reference to the flowchart of FIG. Figure 3 and Figure 4 The operations may be performed in the order shown, or at least a part of these operations may be performed in parallel. In addition, in this example, the feature area detection unit 141 captures an image of the display device of the display unit 171 to be used as an EVF, and detects the feature area from the image data constituting one frame of the video supplied from the AFE 123. However, for example, the feature area detection unit 141 is capable of detecting the feature area from any image data (such as video or still image data read out from, for example, the memory card 125). In addition, the signal processing circuit 124 may apply preprocessing to the image data to reduce the processing load or improve the detection accuracy before processing by the feature area detection unit 141. The preprocessing may include, but is not limited to, resolution change (scaling) and color conversion, etc.

[0053] like Figure 3 As shown, in step S301, the face detection unit 201 applies face detection processing to the supplied image data. Note that for the face detection unit 201, an area with a certain detection reliability or greater detection reliability as a result of the face detection processing being applied is included as a detection result. The face detection unit 201 outputs the detection result to the first pupil detection unit 203, and holds the detection result.

[0054] In step S302, the first pupil detection unit 203 determines whether the face detection unit 201 has detected one or more face regions. If the first pupil detection unit 203 has detected one or more face regions, step S303 is performed. If the first pupil detection unit 203 has not detected a face region, step S306 is performed.

[0055] In step S303, the first pupil detection unit 203 sets one of the face regions for which pupil detection has not yet been performed as a pupil detection region, and then performs step S304.

[0056] In step S304, the first pupil detection unit 203 applies pupil detection processing to the pupil detection area set in step S303. Note that, instead of the pupil detection processing, the eye detection processing may be applied. Note that, for the first pupil detection unit 203, an area having a certain detection reliability or greater detection reliability as a result of the pupil detection processing being applied is included as a detection result. The first pupil detection unit 203 holds the detection result, and then performs step S305.

[0057] In step S305, the first pupil detection unit 203 determines whether pupil detection processing has been applied to all face regions detected in step S301. In the case where it is determined that there is a face region to which pupil detection processing has not been applied, the first pupil detection unit 203 repeatedly executes the processing from step S303. In the case where the first pupil detection unit 203 determines that pupil detection processing has been applied to all face regions, the characteristic region detection unit 141 executes step S306.

[0058] In step S306, the characteristic region detection unit 141 outputs the face detection result held by the face detection unit 201 and the pupil detection result held by the first pupil detection unit 203 to the camera control unit 131. In each detection result, information indicating at least the center coordinates (image coordinates), size, and detection reliability of the detected region is included. Note that in the case where no face region is detected in step S302, the characteristic region detection unit 141 outputs a null result.

[0059] like Figure 4 As shown, in step S401, the head detection unit 202 applies head detection processing to the supplied image data. Note that for the head detection unit 202, an area with a certain detection reliability or greater detection reliability as a result of the head detection processing being applied is included as a detection result. The head detection unit 202 outputs the detection result to the second pupil detection unit 204 and holds the detection result.

[0060] In step S402, the second pupil detection unit 204 determines whether the head detection unit 202 has detected one or more head regions. If the second pupil detection unit 204 has detected one or more head regions, step S403 is performed. If the second pupil detection unit 204 has not detected a head region, step S406 is performed.

[0061] In step S403, the second pupil detection unit 204 sets one of the head regions for which pupil detection has not been performed as a pupil detection region, and then performs step S404.

[0062] In step S404, the second pupil detection unit 204 applies pupil detection processing to the pupil detection area set in step S403. Note that, instead of the pupil detection processing, the eye detection processing may be applied. Note that, for the second pupil detection unit 204, an area having a certain detection reliability or greater detection reliability as a result of the pupil detection processing being applied is included as a detection result. The second pupil detection unit 204 holds the detection result, and then performs step S405.

[0063] In step S405, the second pupil detection unit 204 determines whether the pupil detection process has been applied to all the head regions detected in step S401. When it is determined that there is a head region to which the pupil detection process has not been applied, the second pupil detection unit 204 repeatedly performs the process from step S403. When the second pupil detection unit 204 determines that the pupil detection process has been applied to all the head regions, the characteristic region detection unit 141 performs step S406.

[0064] In step S406, the characteristic region detection unit 141 outputs the head detection result held by the head detection unit 202 and the pupil detection result held by the second pupil detection unit 204 to the camera control unit 131. In each detection result, information indicating at least the center coordinates (image coordinates), size, and detection reliability of the detected region is included. Note that in the case where the head region is not detected in step S402, the characteristic region detection unit 141 outputs a null result.

[0065] Note that the characteristic region detecting unit 141 may output the detection results obtained from step S306 and step S406 together, or may output these detection results individually.

[0066] Next, use Figure 5 The operation of the camera control unit 131 using the detection result from the characteristic region detection unit 141 is explained with reference to the flowchart of FIG.

[0067] In step S501, the subject determination unit 150 determines whether there is a face area whose pupil position has not been set (hereinafter referred to as unchecked). In the case where the subject determination unit 150 determines that there is an unchecked face area, step S502 is performed. In the case where the subject determination unit 150 determines that there is no unchecked face area, step S511 is performed. In the case where no face area is detected, the subject determination unit 150 determines that there is no unchecked face area.

[0068] In step S502, the subject determination unit 150 selects an unchecked face region and the pupil detection result of the face region. The unchecked face region may be selected in any order.

[0069] In step S503, the subject determination unit 150 determines whether there is an unchecked head region. If the subject determination unit 150 determines that there is an unchecked head region, step S504 is performed. If the subject determination unit 150 determines that there is no unchecked head region, step S509 is performed. If no head region is detected, the subject determination unit 150 determines that there is no unchecked head region.

[0070] In step S504, the subject judgment unit 150 performs a subject judgment process that detects a head region of a subject that is considered to be the same as the subject in one of the face regions selected in step S502 from the unchecked face regions. The subject judgment process can be implemented by any method, and, for example, can be implemented by a method using the distance and / or overlap rate between two regions intended for judgment processing. The distance can be the distance between representative coordinates of the regions (such as center or centroid coordinates, etc.). In addition, for the overlap rate, for example, the intersection over union (IoU, also called the Jaccard coefficient) can be used. The IoU is the percentage of the common (overlapping) portion in the union of two regions.

[0071] IoU is equal to the size of the common part of two regions divided by the size of the union of the two regions.

[0072] In the present embodiment, in the case where the overlap ratio is equal to or greater than the threshold value and the center or centroid coordinates of each of the two regions are within the other region, the subject determination unit 150 determines that the two regions are regions of the same subject. Note that in the case where a plurality of head regions are determined to be regions related to the same subject in one face region, the subject determination unit 150 uses the head region with the highest overlap ratio with the face region as the final determination result. Furthermore, in the case where a plurality of head regions have the highest overlap ratio, the head region with the smallest distance from the center coordinates of the face region is used as the final determination result. In the case where the distances between the center coordinates of a plurality of head regions and the center coordinates of the face region are the smallest, one head region is selected as the final determination result by an arbitrary method. The subject determination unit 150 outputs information related to the combination of the face region and the head region obtained as the final determination result to the position determination unit 151. In the case where there is no head region determined to be related to the same subject as the subject in the same face region, the subject determination unit 150 outputs null information or does not output any information.

[0073] In step S505, the position determination unit 151 determines whether a combination of a face region and a head region determined to be related to the same subject is output from the subject determination unit 150. If the position determination unit 151 determines that a combination of a face region and a head region related to the same subject has been output, step S506 is performed. If the position determination unit 151 determines that a combination of a face region and a head region related to the same subject has not been output, step S509 is performed.

[0074] In step S506, the position determination unit 151 selects an area from the face area and the head area determined by the subject determination unit 150 to be related to the same subject. The position determination unit 151 may, for example, select an area with the highest detection reliability, or may select an area determined to have a better pupil detection result. In this example, the "better pupil detection result" has as a first condition that one or two pupil areas with a specific detection reliability or greater have been detected. In addition, in the case where three or more pupil areas have been detected, in addition to the first condition, the second condition is that the difference in detection reliability of the third value relative to the previous two detection reliability values ​​is above a specific value. Note that the selection method described here is an example, and a selection method based on other benchmarks may be used.

[0075] In step S507 , the position determination unit 151 adds the pupil detection result of the face region or the head region selected in step S506 to the detection result of the feature region for the image data.

[0076] In step S508, the position determination unit 151 puts the face region and the head region used in step S506 and determined by the subject determination unit 150 to be related to the same subject into a pupil position setting completed (checked) state. Then, the processing from step S501 is repeated.

[0077] In a case where there is no unchecked head region in step S503 , or in a case where there is no head region determined to be associated with the same subject as the subject in the face region in step S504 , the position determination unit 151 executes step S509 .

[0078] In step S509 , the position determination unit 151 adds the pupil detection result of the face area selected in step S502 to the detection result of the feature area for the image data.

[0079] In step S510, the position determination unit 151 puts the face area selected in step S502 into a check-completed state. Then, the processing from step S501 is repeatedly performed.

[0080] In the case where the above-described processing is repeated and there are no more unchecked face regions, or in the case where the face detection unit 201 has not detected any face region, the processing transitions from step S501 to step S511.

[0081] In step S511, the position determination unit 151 adds the pupil detection result for the unchecked head region (in other words, there is no head region determined to be a facial region related to the same subject) to the detection result for the characteristic region of the image data. In the case where the head detection unit 202 has not detected the head region, since there is no pupil detection result for the head region, nothing is added to the detection result for the characteristic region of the image data.

[0082] In step S512 , the position determination unit 151 outputs the detection result of the feature area for the image data added by the above-described processing to the camera control unit 131 .

[0083] In step S513, the camera control unit 131 performs processing using the detection result of the feature area output by the position determination unit 151 in step S512. For example, an indicator indicating the detected feature area is superimposed on the EVF, and focus detection is performed so that the feature area becomes a focus detection area (AF frame).

[0084] Specifically, the display frame setting unit 152 generates an image of a frame-shaped indicator corresponding to the center coordinates and the size of the pupil area included in the detection result of the feature area, writes the image into the video memory of the display unit 171, and superimposes the image on the video being displayed.

[0085] In addition, the subject setting unit 153 notifies the focus detection unit 154 of information of the pupil area selected by the touch operation on the touch panel 172, for example, as setting information of the focus detection area. The focus detection unit 154 performs a focus detection operation that makes the pupil area received via the notification from the subject setting unit 153 a focus detection area. In this way, the position of the focus lens 104 is set so that the lens unit 101 focuses on the pupil area. Note that the information of the selected pupil area can be used for other processing such as automatic exposure control.

[0086] According to the present embodiment, when a predetermined feature region is detected from image data, a partial region containing the feature region is detected based on different methods, and the feature region is detected for each partial region. Therefore, if conditions prevent one method from detecting a partial region, the possibility that the other method detects the partial region increases, and the possibility that the final feature region can be detected increases. In addition, in the case where two methods can be used to detect a partial region of the same subject, one region that is judged to have a better detection result of the feature region is selected from the partial regions. Therefore, the reliability of the detection result can be improved.

[0087] Second embodiment

[0088] Next, a second embodiment of the present invention will be described. This embodiment is the same as the first embodiment except for the structure and operation of the characteristic region detection unit. Therefore, in the following description, the characteristic region detection unit will be focused on.

[0089] Figure 6 Schematically shows the characteristic region detection unit 141' of this embodiment. Figure 2 As can be seen from the comparison, the characteristic region detection unit 141' of this embodiment includes a partial region detection unit 601 serving as both a face detection unit and a head detection unit and a pupil detection unit 602 serving as both a first pupil detection section and a second pupil detection section.

[0090] The partial region detection unit 601 may be, for example, a detector using a neural network trained to detect a face region and a head region. Since one detector has been trained for the face region and the head region, different algorithms can be used to detect partial regions while also keeping the circuit scale small.

[0091] The pupil detection unit 602 detects a pupil area or an eye area for each partial area detected by the partial area detection unit 601. The detection method may be the same as that used in the first embodiment.

[0092] Next, use Figure 7 The operation of the characteristic region detecting unit 141' is described with reference to the flowchart of FIG. Figure 7 The flowchart of the first embodiment Figure 3 and Figure 4 The flowchart corresponds to .

[0093] In step S701, the partial region detection unit 601 detects partial regions (face region and head region) from the supplied image data. Face detection processing is applied to the partial regions. The partial region detection unit 601 outputs the detection result to the pupil detection unit 602 and holds the detection result.

[0094] In step S702, pupil detection unit 602 determines whether one or more partial regions have been detected by partial region detection unit 601. If pupil detection unit 602 has detected one or more partial regions, step S703 is performed. If pupil detection unit 602 has not detected a partial region, step S706 is performed.

[0095] In step S703, pupil detection unit 602 sets one of the partial areas where pupil detection has not been performed as a pupil detection area, and then executes step S704.

[0096] In step S704, the pupil detection unit 602 applies pupil detection processing to the pupil detection area set in step S703. Note that, instead of the pupil detection processing, eye detection processing may be applied. The pupil detection unit 602 holds the detection result, and then executes step S705.

[0097] In step S705, the pupil detection unit 602 determines whether the pupil detection process has been applied to all partial regions detected in step S701. In the case where it is determined that there are partial regions to which the pupil detection process has not been applied, the pupil detection unit 602 repeatedly performs the process from step S703. In the case where the pupil detection unit 602 determines that the pupil detection process has been applied to all partial regions, the characteristic region detection unit 141' performs step S706.

[0098] In step S706, the characteristic region detection unit 141' outputs the detection result held by the partial region detection unit 601 and the pupil detection result held by the pupil detection unit 602 to the camera control unit 131. In each detection result, information indicating at least the center coordinates (image coordinates), size, and detection reliability of the detected region is included. Note that in the case where no partial region is detected in step S702, the characteristic region detection unit 141' outputs a null result.

[0099] The processing performed by the camera control unit 131 using the detection results of the feature areas may be the same as the processing described in the first embodiment.

[0100] In the first embodiment, the face region and the head region are detected by different detection units, and for pupil detection, the face region and the head region are also detected by different detection units. In contrast, in the present embodiment, the face region and the head region are detected by a single detection unit, and the pupil region is also detected by a single detection unit. In the present embodiment, the circuit scale of the feature region detection unit 141' can be made smaller than that in the first embodiment. With the structure of the first embodiment, the processing speed can be improved by performing a series of processing of face region detection and pupil detection for the face region and a series of processing of head region detection and pupil detection for the head region in parallel.

[0101] Note that in the first embodiment, a plurality of detection units are provided for the detection unit of the partial area and the detection unit of the characteristic area. In the second embodiment, only one detection unit is provided for each of the detection unit of the partial area and the detection unit of the characteristic area. However, the number of detection units of the partial area and the number of detection units of the characteristic area do not have to be the same.

[0102] Two embodiments of the present invention have been described. However, the present invention is not limited to these specific embodiments. The present invention includes various modifications included in the scope of the claims. For example, more than three detection units may be set for a partial area. In addition, in a case where multiple head areas are judged to be related to the same subject as the subject in the same facial area, one head area may be selected, and information of multiple head areas may be used at the same time. For example, information of pupil areas detected in multiple head areas may be combined (for example, the positions of pupil areas having a distance equal to or less than a threshold value may be averaged), and the detection position and size of the new pupil area may be calculated.

[0103] Other embodiments

[0104] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0105] Although the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing device, comprising: A first detection component, configured to detect a first partial area from the image data using a first method, and to detect a second partial area from the image data using a second method, wherein the first partial area and the second partial area are related to the same object; a second detecting component, configured to detect a first characteristic region from the data of the first partial region, and to detect a second characteristic region from the data of the second partial region; and A selection component is used to determine, for each first partial area detected by the first detection component, a second partial area related to the same subject as the first partial area from among the multiple second partial areas detected by the first detection component based on the overlap rate with the first partial area, and for the first partial area and the second partial area determined to be related to the same subject, select any one of the detection results of the first characteristic area for the first partial area and the detection results of the second characteristic area for the second partial area.

2. The image processing apparatus according to claim 1, wherein: The first detecting section detects a partial region expected to include the first feature region as the first partial region, and detects a partial region expected to include the second feature region as the second partial region.

3. The image processing apparatus according to claim 1, wherein: The first partial area is a facial area; and The second partial area is a head area.

4. The image processing apparatus according to claim 1, wherein: The first feature area and the second feature area are each an eye area or a pupil area.

5. The image processing apparatus according to any one of claims 1 to 4, wherein: The selection component selects one of the detection results that is judged to be more excellent.

6. The image processing apparatus according to any one of claims 1 to 4, wherein: The selection component selects one of the detection results that is determined to have a higher reliability in feature region detection.

7. The image processing apparatus according to any one of claims 1 to 4, wherein: The first detection means includes a first detection section that detects the first partial area using the first method and a second detection section that detects the second partial area using the second method, and the first detection section and the second detection section operate in parallel.

8. A camera device, comprising: A camera component for generating image data; The image processing apparatus according to any one of claims 1 to 7, which uses the image data generated by the image pickup means; A control component is used to perform focus detection and / or exposure control on the detection result of the feature area selected by the selection component of the image processing device.

9. An image processing method performed by an image processing device, comprising: Using a first method to detect a first partial area from the image data, and using a second method to detect a second partial area from the image data, wherein the first partial area and the second partial area are related to the same object; Detecting a first characteristic region from the data of the first partial region, and detecting a second characteristic region from the data of the second partial region; For each detected first partial area, determining, from the plurality of detected second partial areas, a second partial area associated with the same subject as that of the first partial area based on an overlap ratio with the first partial area; and For the first partial region and the second partial region determined to be related to the same object, any one of a detection result of the first characteristic region for the first partial region and a detection result of the second characteristic region for the second partial region is selected. 10 . A machine-readable medium storing a program for causing a computer to perform the image processing method according to claim 9 .

Images