Electronic apparatus, method for controlling electronic apparatus, and program

Abstract

To prevent the occurrence of a reduction in focusing accuracy when a shield area is present in a subject area.SOLUTION: When there is a shield that blocks a subject, an electronic apparatus performs focusing on the basis of a result of detection of a first focus state based on the phase difference between a pair of images in a first direction when a shield area of the shield spreads in the first direction, and performs focusing on the basis of a result of detection of a second focus state based on the phase difference between a pair of images in a second direction when the shield area of the shield spreads in the second direction.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an electronic device, a control method for an electronic device, and a program. [Background technology]

[0002] A technology is used to detect a subject pattern (for example, a facial area of ​​a person as a subject) from an image captured by an imaging device such as a camera. As a related technology, the technology of Patent Document 1 is proposed. The technology of Patent Document 1 realizes highly accurate focus adjustment and exposure control of a person's face by performing face area detection, which detects a person's face from an image, and detection of AF / AE / WB evaluation values ​​for the same frame.

[0003] In recent years, deep learning neural networks have been used to detect subjects from images. Convolutional neural networks (CNNs) are used as neural networks suitable for image recognition and the like. For example, Non-Patent Document 1 proposes a technology (Single Shot Multibox Detector) that applies CNN to detect objects in an image. Non-Patent Document 2 proposes a technology (Semantic Image Segmentation) that semantically divides regions in an image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2005-318554 A [Non-patent literature]

[0005] [Non-Patent Document 1] “Liu,SSD:Single Shot Multibox Detector. In: ECCV2016” [Non-Patent Document 2] “Chen et.al, DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, arXiv, 2016” Summary of the Invention [Problem to be solved by the invention]

[0006] In recent years, a method has been used in which a pupil division function is used to detect phase differences and focus adjustment is performed using the detected phase difference information. When a subject is occluded by some object during shooting, an occluded area may occur in the subject area in the image. Perspective conflict may occur depending on the distribution direction of the occluded area in the subject area. When perspective conflict occurs, there is a problem that the focus adjustment accuracy decreases. The technology of Patent Document 1 does not solve such a problem.

[0007] Therefore, an object of the present invention is to suppress the occurrence of a decrease in focus adjustment accuracy when an occluded area exists in a subject area. [Means for solving the problem]

[0008] In order to achieve the above-mentioned object, the electronic device of the present invention comprises an image sensor having a plurality of pixels, each having a plurality of photoelectric conversion units that receive light beams passing through different pupil regions of an imaging optical system, and outputting a signal capable of detecting a first focus state of the imaging optical system from a phase difference between a pair of images in at least a first direction, and a second focus state of the imaging optical system from the phase difference between a pair of images in a second direction different from the first direction, and a control means that performs focus adjustment in accordance with a focus detection result based on the signal output from the image sensor, wherein when there is an object blocking the subject, the control means performs focus adjustment based on the detection result of the first focus state when the blocking area of ​​the object extends in the first direction, and performs focus adjustment based on the detection result of the second focus state when the blocking area of ​​the object extends in the second direction. Effect of the Invention

[0009] According to the present invention, it is possible to suppress the occurrence of a decrease in focus adjustment accuracy when an occluded area exists in a subject area. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of a digital camera. [Diagram 2] 3A and 3B are diagrams illustrating an example of a pixel arrangement of an image sensor and a direction of pupil division of the pixels. [Diagram 3] FIG. 1 is a diagram illustrating an example of a CNN that infers the likelihood of an occluded region. [Figure 4] 5 is a flowchart showing an example of the flow of focus adjustment processing in the first embodiment. [Diagram 5] 1A and 1B are diagrams illustrating an example of image data, a distribution of a subject region, and a blocking region. [Figure 6] 13 is a diagram showing an example in which the direction of pupil division of pixels is the Y direction. FIG. [Figure 7] 10 is a flowchart showing an example of the flow of focus adjustment processing in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in the embodiments.

[0012] First Embodiment 1 is a diagram showing an example of a digital camera C as an electronic device. The digital camera C is a single-lens reflex camera with interchangeable lenses. The digital camera C may be a camera that is not an interchangeable lens type. Furthermore, the electronic device is not limited to a digital camera and may be any device such as a smartphone or a tablet terminal.

[0013] Digital camera C has a lens unit 100, which is an imaging optical system, and a camera body 120. Lens unit 100 is detachably attached to camera body 120 via mount M (lens mount) shown by a dotted line in the center of FIG. 1. Lens unit 100 has an optical system and a drive control system. The optical system includes a first lens group 101, an aperture 102, a second lens group 103, and a focus lens 104 (focus lens group). Lens unit 100 is a photographing lens that forms an optical image of a subject.

[0014] The first lens group 101 is disposed at the tip of the lens unit 100 and is held so as to be movable in the optical axis direction OA. The aperture 102 functions to adjust the amount of light during shooting, and also functions as a mechanical shutter that controls exposure time during still image shooting. The aperture 102 and the second lens group 103 are movable together in the optical axis direction OA, and move in conjunction with the first lens group 101 to achieve a zoom function. The focus lens 104 is also movable in the optical axis direction OA, and the subject distance (focusing distance) at which the lens unit 100 focuses changes depending on the position. By controlling the position of the focus lens 104 in the optical axis direction OA, focus adjustment is performed to adjust the focusing distance of the lens unit 100.

[0015] The drive control system has a zoom actuator 111, an aperture actuator 112, and a focus actuator 113. The drive control system also has a zoom drive circuit 114, an aperture shutter drive circuit 115, a focus drive circuit 116, a lens MPU (MPU: microprocessor) 117, and a lens memory 118.

[0016] The zoom driving circuit 114 drives the first lens group 101 and the second lens group 103 in the optical axis direction OA using the zoom actuator 111, and controls the angle of view of the optical system of the lens unit 100. The aperture shutter driving circuit 115 drives the aperture 102 using the aperture actuator 112, and controls the aperture diameter and opening / closing operation of the aperture 102. The focus driving circuit 116 drives the focus lens 104 in the optical axis direction OA using the focus actuator 113, and changes the focal distance of the optical system of the lens unit 100. The focus driving circuit 116 also detects the current position of the focus lens 104 using the focus actuator 113.

[0017] The lens MPU 117 performs various calculations and controls related to the lens unit 100, thereby controlling the zoom driving circuit 114, the aperture shutter driving circuit 115, and the focus driving circuit 116. The lens MPU 117 is also connected to the camera MPU 125 through the mount M, and communicates commands and data with the camera MPU 125. For example, the lens MPU 117 detects the position of the focus lens 104, and notifies the camera MPU 125 of lens position information in response to a request from the camera MPU 125. The lens position information includes information such as the position of the focus lens 104 in the optical axis direction OA, the position and diameter of the exit pupil in the optical axis direction OA when the optical system is not moving, and the position and diameter of the lens frame that limits the light flux of the exit pupil in the optical axis direction OA. The lens MPU 117 also controls the zoom driving circuit 114, the aperture shutter driving circuit 115, and the focus driving circuit 116 in response to a request from the camera MPU 125. The lens memory 118 stores optical information required for automatic focus detection in advance. The camera MPU 125 controls the operation of the lens unit 100 by executing programs stored in, for example, an internal non-volatile memory or a lens memory 118 .

[0018] Like the lens unit 100, the camera body 120 has an optical system and a drive control system. The optical system includes an optical low pass filter 121 and an image sensor 122. The first lens group 101, the aperture 102, the second lens group 103, and the focus lens 104 of the lens unit 100, and the optical low pass filter 121 of the camera body 120 form an imaging optical system. The optical low pass filter 121 is a filter that reduces false colors and moiré in a captured image.

[0019] The imaging element 122 includes a CMOS image sensor and peripheral circuits. The imaging element 122 receives incident light from an imaging optical system. The imaging element 122 has m pixels arranged horizontally and n pixels arranged vertically (n and m are integers of 2 or more). The imaging element 122 has a pupil division function and is capable of performing phase difference AF (autofocus) using image data. The image processing circuit 124 generates data for phase difference AF and image data for display and recording from the image data output by the imaging element 122.

[0020] The drive control system includes an image sensor drive circuit 123, an image processing circuit 124, a camera MPU 125, a display 126, a group of operation switches 127, a memory 128, an image sensor surface phase difference detection unit 129, a recognition unit 130, and a communication unit 131. The image sensor drive circuit 123 controls the operation of the image sensor 122, and also A / D converts the acquired image signal and transmits it to the camera MPU 125. The image processing circuit 124 performs general image processing performed in digital cameras, such as gamma conversion, white balance adjustment processing, color interpolation processing, and compression encoding processing, on the image data acquired by the image sensor 122. The image processing circuit 124 also generates a signal for phase difference AF.

[0021] The camera MPU 125 performs various calculations and controls related to the camera body 120. The camera MPU 125 controls the image sensor drive circuit 123, the image processing circuit 124, the display 126, the operation switch group 127, the memory 128, the image plane phase difference detection unit 129, the recognition unit 130, and the communication unit 131. The camera MPU 125 is connected to the lens MPU 117 via a signal line of the mount M, and performs communication regarding commands and data with the lens MPU 117. The camera MPU 125 makes various requests to the lens MPU 117. For example, the camera MPU 125 requests information on the lens position and optical information specific to the lens unit 100 from the lens MPU 117. The camera MPU 125 also requests the lens MPU 117 to drive the aperture, focus lens, zoom, etc. at a predetermined drive amount.

[0022] The camera MPU 125 includes a ROM 125a, a RAM 125b, and an EEPROM 125c. The ROM (Read Only Memory) 125a stores a program that controls the imaging operation. The RAM (Random Access Memory) 125b temporarily stores variables. The EEPROM (Electrically Erasable Programmable Read-Only Memory) 125c stores various parameters.

[0023] The display 126 is composed of an LCD (liquid crystal display) etc., and displays information about the camera's shooting mode, a preview image before shooting, a confirmation image after shooting, an in-focus state display image at the time of focus detection, etc. The operation switch group 127 includes a power switch, a release (shooting trigger) switch, a zoom operation switch, a shooting mode selection switch, etc. The memory 128 is a removable flash memory, and records shot images.

[0024] The imaging surface phase difference detection unit 129 performs focus detection processing by a phase difference detection method using focus detection data acquired from the image processing circuit 124. The image processing circuit 124 generates pairs of image data formed by light beams passing through two pairs of pupil regions as focus detection data. Then, the imaging surface phase difference detection unit 129 detects a focus deviation amount based on the deviation amount of each generated pair of image data. The imaging surface phase difference detection unit 129 performs phase difference AF (imaging surface phase difference AF) based on the output of the image sensor 122 without using a dedicated AF sensor. The imaging surface phase difference detection unit 129 may be realized by a part of the camera MPU 125, or may be realized by a dedicated circuit, CPU, etc.

[0025] The recognition unit 130 performs object recognition based on image data acquired from the image processing circuit 124. As object recognition, the recognition unit 130 performs object detection to detect the position of the target object in the image data, and area division to divide the object area into an object area and an obstructed area in which the object is obstructed. In this embodiment, the recognition unit 130 determines an area of ​​phase difference information to be used for focus adjustment based on the distribution direction of the obstruction information and the detection direction of the amount of focus shift (amount of image shift). Hereinafter, the detection direction of the amount of image shift may be referred to as the pupil division direction of the phase difference information.

[0026] The recognition unit 130 of this embodiment uses a CNN (convolutional neural network) to perform object detection and region division. In this embodiment, a CNN that has undergone deep learning for object detection is used for object detection, and a CNN that has undergone deep learning for region division is used for region division. However, the recognition unit 130 may use a CNN that has undergone deep learning for both object detection and region division.

[0027] The recognition unit 130 acquires an image from the camera MPU 125 and inputs it to a CNN that has undergone deep learning of object detection. The object is detected as an output result of the inference process of the CNN. The recognition unit 130 also inputs the image of the recognized object to a CNN that has undergone deep learning of region division. The inference process of the CNN detects an occluded area in the image of the object as an output result. The recognition unit 130 may be realized by the camera MPU 125, or may be realized by a dedicated circuit, CPU, or the like. In addition, since the recognition unit 130 performs inference processing by the CNN, it is preferable that the recognition unit 130 has a built-in GPU that is used for the arithmetic processing of the inference processing.

[0028] CNN deep learning will be explained. CNN deep learning can be performed by any method. CNN deep learning for object detection will be explained. For example, CNN deep learning is realized by supervised learning using an image containing a subject as the correct answer as training data and many learning images as input data. In this case, a method such as backpropagation is applied to CNN deep learning.

[0029] The deep learning of CNN may be performed by a predetermined computer such as a server. In this case, the communication unit 131 of the camera body 120 may communicate with the predetermined computer to acquire the deep-learned CNN from the predetermined computer. Then, the camera MPU 125 sets the CNN acquired by the communication unit 131 to the recognition unit 130. This allows the recognition unit 130 to perform subject detection using the deep-learned CNN. If the digital camera C has a built-in high-performance CPU or GPU suitable for deep learning, or a dedicated processor specialized for deep learning, the digital camera C may perform the deep learning of CNN. However, since the deep learning of CNN requires abundant hardware resources, it is preferable that an external device (a predetermined computer) performs the deep learning of CNN and the digital camera C acquires and uses the deep-learned CNN.

[0030] The object detection may be realized by using any method other than CNN. For example, the object detection may be realized by a rule-based method. In addition, a trained model trained by machine learning by any method other than deep learning CNN may be used for the object detection. For example, the object detection may be realized by using a trained model trained by machine learning by any machine learning algorithm such as a support vector machine or a logistic regression.

[0031] Next, the operation of the imaging surface phase difference detection unit 129 will be described. FIG. 2 is a diagram showing an example of the arrangement of pixels of the imaging element 122 and the direction of pupil division of the pixels. FIG. 2(A) shows a state in which a range of 6 rows in the vertical direction (Y direction) and 8 columns in the horizontal direction (X direction) of a two-dimensional C-MOS area sensor is observed from the lens unit 100 side. The imaging element 122 is provided with color filters in a Bayer array, and green (G) and red (R) color filters are alternately arranged in order from the left in the pixels of odd rows, and blue (B) and green (G) color filters are alternately arranged in order from the left in the pixels of even rows. In the pixel 211, a plurality of photoelectric conversion units are arranged inside the on-chip microlens (microlens 211i) indicated by a circle. In the example of FIG. 2(B), four photoelectric conversion units 211a, 211b, 211c, and 211d are arranged inside the on-chip microlens of the pixel 211.

[0032] In the pixel 211, the photoelectric conversion units 211a, 211b, 211c, and 211d are divided into two in the X direction and the Y direction, and the photoelectric conversion signals of the individual photoelectric conversion units can be read out, and the sum of the photoelectric conversion signals of each photoelectric conversion unit can also be read out independently. The photoelectric conversion signals of the individual photoelectric conversion units are used as data for phase difference AF. The photoelectric conversion signals of the individual photoelectric conversion units may also be used to generate parallax images that form a 3D (3-Dimensional) image. The sum of the photoelectric conversion signals is used when generating normal captured image data.

[0033] A pixel signal when performing phase difference AF will be described. In this embodiment, the light beam emitted from the photographing optical system is pupil-divided by the photoelectric conversion units 211a, 211b, 211c, and 211d via the microlens 211i in FIG. 2(A). The areas indicated by two dotted lines in FIG. 2(B) respectively indicate the photoelectric conversion unit 212A and the photoelectric conversion unit 212B. The photoelectric conversion unit 211A is composed of the photoelectric conversion units 211a and 211c, and the photoelectric conversion unit 211B is composed of the photoelectric conversion units 211a and 211d. In order to perform focus detection based on the image shift amount (phase difference), a sum signal obtained by adding together the signals output from the photoelectric conversion units 211a and 211c and a sum signal obtained by adding together the signals output from the photoelectric conversion units 211a and 211d are used as a pair. This allows focus detection based on the image shift amount (phase difference) in the X direction.

[0034] Here, the description focuses on phase difference AF by focus detection based on the image shift amount in the X direction. For a plurality of pixels 211 within a predetermined range arranged in the same pixel row, an image formed by combining the sum signals of the photoelectric conversion units 211a and 211c belonging to the photoelectric conversion unit 212A is defined as an AF image A. An image formed by combining the sum signals of the photoelectric conversion units 212b and 211d belonging to the photoelectric conversion unit 212B is defined as an AF image B. For the output of the photoelectric conversion units 212A and 212B, a pseudo luminance (Y) signal calculated by adding the outputs of green, red, blue, and green included in the unit array of the color filter is used. However, an AF image A and an AF image B may be formed for each color of red, blue, and green. By detecting the relative image shift amount of a pair of image signals of the AF image A and the AF image B generated as described above by correlation calculation, a prediction [bit] that is the correlation degree of the pair of image signals can be detected. The camera MPU 125 can detect the defocus amount [mm] of a predetermined area by multiplying the prediction by a conversion coefficient. The sum of the output signals of the photoelectric conversion units 212A and 212B generally forms one pixel (output pixel) of the output image.

[0035] The subject detection and occlusion information area division by the recognition unit 130 will be described. In this embodiment, the recognition unit 130 detects (recognizes) a person's face from an image. Any method (for example, the method in Non-Patent Document 1) can be applied as a method for detecting a person's face from an image. However, the recognition unit 130 may detect a subject that can be a target for focus adjustment, such as a person's whole body, an animal, or a vehicle, instead of a person's face.

[0036] The recognition unit 130 performs area division for the occluded area for the detected subject area. Any method (for example, the method of Non-Patent Document 2) can be applied as the area division method. The recognition unit 130 uses a deep-learned CNN to infer the likelihood of an occluded area for each pixel area. However, as described above, the recognition unit 130 may infer the likelihood of an occluded area using a trained model machine-learned by any machine learning algorithm, or may determine the likelihood of an occluded area based on a rule base. When a CNN is used to infer the likelihood of an occluded area, the CNN performs deep learning with the occluded area as a positive example and areas other than the occluded area as a negative example. As a result, the CNN outputs the likelihood of an occluded area in each pixel area as an inference result.

[0037] FIG. 3 is a diagram showing an example of a CNN that infers the likelihood of an occluded region. FIG. 3(A) shows an example of a subject region of an input image input to a CNN. Subject region 301 is detected from an image by the above-mentioned subject detection. Subject region 301 includes face region 302, which is a detection target of subject detection. Face region 302 in FIG. 3(A) includes two occluded regions (occluded regions 303 and 304). Occluded region 303 is a region with no depth difference, and occluded region 304 is a region with a depth difference. Occluded regions are also called occlusions, and the distribution of occluded regions is also called occlusion distribution.

[0038] FIG. 3(B) shows an example of the definition of occlusion information. Each of images 1 to 3 in FIG. 3(B) is divided into a white region and a black region, with the white region indicating a positive example and the black region indicating a negative example. The occlusion information obtained by dividing the image of the subject region in FIG. 3(B) is an image that is assumed to be a candidate for training data used when performing deep learning of CNN. Hereinafter, it will be described which of the occlusion information in FIG. 3(B) is used as training data in this embodiment.

[0039] Image 1 in Fig. 3(B) shows an example of occlusion information when the area is divided into a subject area (face area) and an area other than the subject, the subject area is a positive example, and the area other than the subject area is a negative example. Image 2 in Fig. 3(B) shows an example of occlusion information when the area is divided into a foreground occluded area relative to the subject and the other area, the foreground occluded area is a positive example, and the area other than the foreground occluded area relative to the subject is a negative example. Image 3 in Fig. 3(C) shows an example of occlusion information when the area is divided into an occluded area that causes perspective conflict and the other area, the occluded area that causes perspective conflict is a positive example, and the area other than the occluded area that causes perspective conflict is a negative example.

[0040] In the example of Fig. 2(B), the photoelectric conversion units 211a, 211b, 211c, and 211d are pupil-divided into two in the X direction. In this case, if the occluded areas in the image are distributed in the Y direction, perspective conflicts are likely to occur. Therefore, teacher data (teacher image) for the occluded areas where perspective conflicts are likely to occur can be generated based on the pupil division direction of the multiple photoelectric conversion units in a pixel.

[0041] As shown in FIG. 3B, the face of a person in an image has a distinctive visibility pattern and the variance of the pattern is small, so that the region can be divided with high accuracy. For example, the occlusion information shown in FIG. 3B, No. 1 is suitable as training data in a learning process when generating a CNN that detects a person as a subject. From the viewpoint of detection accuracy, the occlusion information shown in FIG. 3B, No. 1 is more suitable than the occlusion information shown in FIG. 3B, No. 3. However, an image shown in FIG. 3B, No. 3 is suitable as training data in a learning process when generating a CNN that detects an occlusion region that causes perspective conflict. Parallax images of the images A and B used in focus detection may be used as training data in a learning process when generating a CNN that detects an occlusion region that causes perspective conflict. The occlusion information is not limited to the above example, and may be generated based on any method of dividing an area into an occlusion region and an area outside the occlusion region.

[0042] Fig. 3(C) shows the flow of deep learning of CNN. In this embodiment, an RGB image is used as an input image 310 for learning. In addition, as a teacher image, a teacher image 314 (teacher image of occlusion information) as shown in Fig. 3(C) is used. The teacher image 314 is an image of occlusion information that causes perspective conflict in Fig. 3(B).

[0043] An input image 310 for learning is input to a neural network system 311 (CNN). The neural network system 311 can employ, for example, a layer structure in which convolutional layers and pooling layers are alternately stacked between an input layer and an output layer, and a multi-layer structure in which a fully connected layer is connected to the rear of the layer structure. A score map indicating the likelihood of an occluded region in the input image is output from the output layer 312 in FIG. 3(C). The score map is output in the form of an output result 313.

[0044] In the deep learning of CNN, the error between the output result 313 and the teacher image 314 is calculated as a loss value 315. The loss value 315 is calculated using a method such as cross entropy or squared error. Then, coefficient parameters such as weights and biases of each node of the neural network system 311 are adjusted so that the value of the loss value 315 gradually decreases. By performing deep learning of CNN sufficiently using many learning input images 310, the neural network system 311 will output a more accurate output result 313 when an unknown input image is input. In other words, when an unknown input image is input, the neural network system 311 (CNN) will output occlusion information obtained by dividing an occluded area and an area other than the occluded area as the output result 313 with high accuracy. Note that it takes a lot of work to create teacher data that specifies the occluded area (area of ​​overlapping objects). For this reason, it is also possible to create teacher data using CG or by using image synthesis in which an object image is cut out and superimposed.

[0045] As described above, an example of applying the image 3 in FIG. 3B in which an area with a depth difference (an area in the foreground of the subject where the depth difference is equal to or greater than a predetermined value) is the occluded area has been described as the teacher image 314. Here, an image like the image 2 in FIG. 3B in which an area without a depth difference (an area in the foreground of the subject where the depth difference is less than a predetermined value) is the occluded area may be applied as the teacher image 314. Even if an image like the image 2 in FIG. 3B is used as the teacher image 314, when an unknown input image is input to the CNN, the CNN can infer an area that causes perspective conflict. However, the accuracy of inferring an occluded area that causes perspective conflict by the CNN is improved when an image like the image 3 in FIG. 3B in which an area with a depth difference is the occluded area is used as the teacher image 314.

[0046] Any method can be applied to detect the occluded area, not just CNN. For example, the detection of the occluded area may be realized by a rule-based method. In addition, a trained model trained by machine learning using any method other than deep learning CNN may be used to detect the occluded area. For example, the occluded area may be detected using a trained model trained by machine learning using any machine learning algorithm such as a support vector machine or logistic regression. This is similar to the subject detection.

[0047] Next, the focus adjustment process will be described. Fig. 4 is a flowchart showing an example of the flow of the focus adjustment process in the first embodiment. When performing focus adjustment, the camera MPU 125 determines a reference area for focus adjustment based on the pupil division direction of the phase difference in the image pickup surface phase difference detection unit 129 and the direction of the occluded area by the recognition unit 130.

[0048] In S401, the recognition unit 130 detects a subject from an image acquired from the image processing circuit 124 via the camera MPU 125, for example, using a CNN that performs subject detection. The recognition unit 130 may detect a subject from an image using a method other than CNN. In S402, the recognition unit 130 detects an occluded area within the area of ​​the subject detected from the image. At this time, the recognition unit 130 inputs the image acquired from the image processing circuit 124 as an input image to the CNN described in FIG. 3. If the CNN is sufficiently trained, the image output as the output result from the CNN is an image in which the occluded area and an area other than the occluded area can be distinguished. The recognition unit 130 detects the occluded area from the image output from the CNN.

[0049] In S403, the recognition unit 130 determines the distribution direction of the detected occluded area. For example, the recognition unit 130 may compare the edge integral value in the X direction with the edge integral value in the Y direction and determine the direction in which the integral value is smaller as the distribution direction of the occluded area. Fig. 5 is a diagram showing an example of the distribution of image data, subject areas, and occluded areas.

[0050] 5(A) is a diagram showing an example of a case where an obstruction (e.g., a rod) extending in the X direction is present between digital camera C and a subject. Obstruction 502 is reflected in image 500 (image data) acquired from image processing circuit 124 so as to obstruct subject 501. Recognition unit 130 detects an image of subject area 510 from image 500. Subject area 510 includes subject 511 and obstruction 512. Obstruction 512 obstructs subject 511 in the X direction.

[0051] When an image 500 including a subject region 510 is input to the CNN, the recognition unit 130 outputs an output result 520 as an inference result of the subject region 510. The output result 520 includes an occluded region 522 and a region 521 other than the occluded region. The occluded region 522 has a distribution corresponding to the occluding object 512. In the example of FIG. 5(A), the occluded region 522 is distributed in the X direction. The recognition unit 130 compares the edge integral value in the X direction and the edge integral value in the Y direction in the output result 520, and determines that the X direction in which the integral value is smaller is the distribution direction of the occluded region.

[0052] 5B is a diagram showing an example of a case where an obstruction extending in the Y direction is present between digital camera C and a subject. Obstruction 552 is reflected in image 550 acquired from image processing circuit 124 so as to obstruct subject 551. Recognition unit 130 detects subject area 560 from image 550. Subject area 560 includes subject 561 and obstruction 562. Obstruction 562 obstructs subject 511 in the Y direction.

[0053] When the recognition unit 130 inputs an image 550 including a subject region 560 to the CNN, it outputs an output result 570 as an inference result for the subject region 560. The output result 570 includes an occluded region 572 and a region 571 other than the occluded region. The recognition unit 130 determines the distribution direction of the occluded region using a method similar to that of FIG. 3(A). In the example of FIG. 3(B), the recognition unit 130 determines that the distribution direction of the occluded region is the Y direction. From the above, the recognition unit 130 can determine the distribution direction of the occluded region.

[0054] Returning to FIG. 4, the process from S404 onward will be described. In S404, the camera MPU 125 determines whether the distribution direction of the masked regions is the X direction or the Y direction based on the determination result in S403. If the camera MPU 125 determines in S404 that the distribution direction of the masked regions is the X direction, the flow proceeds to S405. On the other hand, if the camera MPU 125 determines in S404 that the distribution direction of the masked regions is the Y direction, the flow proceeds to S406. In S404, the camera MPU 125 may determine that the distribution direction of the masked regions is the X direction even if the distribution direction of the masked regions does not completely match the X direction, as long as it is within a predetermined angle range with the X direction as a reference. Similarly, in S404, the camera MPU 125 may determine that the distribution direction of the masked regions is the Y direction even if the distribution direction of the masked regions does not completely match the Y direction, as long as it is within a predetermined angle range with the Y direction as a reference.

[0055] In S405, the camera MPU 125 controls focus adjustment based on the image shift amount for the area where the object is detected (object area). When the distribution direction of the occluded area is the X direction, the possibility of perspective conflict occurring in the image shift amount (phase difference) in the X direction is low. In this case, the image plane phase difference detection unit 129 performs focus detection based on the image shift amount (phase difference) in the X direction with reference to the entire object area. Then, the camera MPU 125 controls focus adjustment based on the image shift amount detected by the image plane phase difference detection unit 129. Note that the image plane phase difference detection unit 129 may perform focus detection based on the image shift amount (phase difference) in the X direction, excluding the occluded area of ​​the object area. After the process of S405 is performed, the flowchart in FIG. 4 ends.

[0056] In S406, the camera MPU 125 controls focus adjustment based on the image shift amount for the area excluding the occluded area from the subject area. If the distribution direction of the occluded area is the Y direction, there is a high possibility that perspective conflict will occur in the image shift amount (phase difference) in the X direction. In this case, the imaging surface phase difference detection unit 129 performs focus detection based on the image shift amount (phase difference) in the X direction by referring to the area excluding the occluded area from the subject area.

[0057] When there are a plurality of correlation calculation blocks in the subject area and there is a block that does not include a shading area, the imaging surface phase difference detection unit 129 performs focus detection based on the image shift amount of the correlation calculation. On the other hand, when there are a plurality of correlation calculation blocks in the subject area and there is no block that does not include a shading area, the imaging surface phase difference detection unit 129 shifts the calculation position of the correlation calculation so as not to include the shading area and calculates the image shift amount. Then, the imaging surface phase difference detection unit 129 performs focus detection based on the calculated image shift amount. After the process of S406 is performed, the flowchart of FIG. 5 ends.

[0058] As described above, according to the first embodiment, the distribution direction of the occluded area in the subject area of ​​the captured image is detected, and the area used for focus detection is controlled according to the distribution direction of the detected occluded area and the pupil division direction of each pixel of the image sensor. This prevents a decrease in focus adjustment accuracy caused by perspective conflict even when an occluded area is present in the subject area. In the first embodiment, as shown in FIG. 2B, an example is shown in which the pupil division direction of each pixel of the image sensor 122 is the X direction, but the first embodiment can also be applied to the case where it is the Y direction.

[0059] <Second embodiment> Next, a second embodiment will be described. In the second embodiment, the digital camera C can calculate both the amount of image shift in the horizontal direction (X direction) and the amount of image shift in the vertical direction (Y direction). Therefore, the digital camera C can switch the direction of pupil division between the X direction and the Y direction. The configuration of the second embodiment is similar to the configuration of FIG. 1 described in the first embodiment, and therefore a description thereof will be omitted.

[0060] The calculation of the image shift amount in the X direction is the same as in the first embodiment. The calculation of the image shift amount in the Y direction will be described. FIG. 6 is a diagram showing an example in which the pupil division direction of the pixel 211 is the Y direction. As in FIG. 2(B), in the pixel 211, the photoelectric conversion units 211a, 211b, 211c, and 211d are divided into two in the X direction and the Y direction. Of the photoelectric conversion units 211a, 211b, 211c, and 211d, the photoelectric conversion unit 212C is composed of the photoelectric conversion units 211a and 211b. Of the photoelectric conversion units 211a, 211b, 211c, and 211d, the photoelectric conversion unit 212D is composed of the photoelectric conversion units 211c and 211d.

[0061] In the second embodiment, a sum signal obtained by adding together output signals of photoelectric conversion units 211a and 211b belonging to photoelectric conversion unit 212C and a sum signal obtained by adding together output signals of photoelectric conversion units 211c and 211d belonging to photoelectric conversion unit 212D are used as a pair. This makes it possible to perform focus detection based on the image shift amount (phase difference) in the Y direction. The correlation calculation is similar to that of the first embodiment, except that the paired direction is the Y direction instead of the X direction.

[0062] Furthermore, in the second embodiment, the direction of pupil division of pixel 211 can be switched between the X direction and the Y direction. For example, under the control of camera MPU 125, image sensor drive circuit 123 may switch between reading output signals from photoelectric conversion units 212A and 212B, or reading output signals from photoelectric conversion units 212C and 212D of the second embodiment. This allows the direction of pupil division of pixel 211 to be switched between the X direction and the Y direction.

[0063] FIG. 7 is a flowchart showing an example of the flow of focus adjustment processing in the second embodiment. Steps S701 to S703 are the same as steps S401 to S403 in FIG. 4, and therefore will not be described. However, in deep learning for CNN, both a teacher image in which occluded regions are distributed in the X direction and a teacher image in which occluded regions are distributed in the Y direction are used. This makes it possible to infer the likelihood of the existence of occluded regions distributed in the X and Y directions as an output result using CNN with an unknown input image as input. Deep learning for CNN is the same as in the first embodiment.

[0064] In S704, the camera MPU 125 determines whether the distribution direction of the masked areas is the X direction or the Y direction based on the determination result of S803. If the camera MPU 125 determines in S704 that the distribution direction of the masked areas is the X direction, the flow proceeds to S705. On the other hand, if the camera MPU 125 determines in S704 that the distribution direction of the masked areas is the Y direction, the flow proceeds to S706.

[0065] In S705, the camera MPU 125 switches the direction of pupil division and controls focus adjustment based on the image shift amount in the X direction. When the distribution direction of the occluded area is the X direction, the possibility of perspective conflict occurring in the image shift amount (phase difference) in the X direction is low. In this case, the image plane phase difference detection unit 129 performs focus detection based on the image shift amount (phase difference) in the X direction. In S705, the image plane phase difference detection unit 129 does not perform focus detection based on the image shift amount (phase difference) in the Y direction. Then, the camera MPU 125 controls focus adjustment based on the image shift amount detected by the image plane phase difference detection unit 129. After the processing of S705 is performed, the flowchart in FIG. 5 ends.

[0066] In S706, the camera MPU 125 switches the direction of pupil division and controls focus adjustment based on the image shift amount in the Y direction. When the distribution direction of the occluded area is the Y direction, the possibility of perspective conflict occurring in the image shift amount (phase difference) in the Y direction is low. In this case, the image plane phase difference detection unit 129 performs focus detection based on the image shift amount (phase difference) in the Y direction. In S706, the image plane phase difference detection unit 129 does not perform focus detection based on the image shift amount (phase difference) in the X direction. Then, the camera MPU 125 controls focus adjustment based on the image shift amount detected by the image plane phase difference detection unit 129. After the processing of S406 is performed, the flowchart in FIG. 5 ends.

[0067] As described above, according to the second embodiment, in a configuration in which the direction of pupil division of pixel 211 can be switched between the X direction and the Y direction, the direction of pupil division is set according to the distribution direction of occluded areas, and focus adjustment is controlled based on phase difference information. This makes it possible to suppress a decrease in focus adjustment accuracy caused by perspective conflict when an occluded area exists in the subject area, regardless of whether the distribution direction of the occluded areas is the X direction or the Y direction.

[0068] The camera MPU 125 may preferentially select a direction based on a direction with contrast (a direction with high contrast) in the subject area excluding the occluded area as the direction of phase difference information when controlling focus adjustment. In this case, the camera MPU 125 performs focus adjustment based on the image shift amount (phase difference) in the direction with contrast in the subject area excluding the occluded area, not the direction of pupil division and the distribution direction of the occluded area. If there is no contrast in the subject area, even if perspective conflict can be avoided, the target subject cannot be detected and focus adjustment cannot be performed. For this reason, the camera MPU 125 performs focus adjustment based on the image shift amount in the direction with contrast in the subject area excluding the occluded area. This is the same as in the first embodiment.

[0069] Furthermore, when the contrast of the occluded area is low (when the value indicating the contrast is lower than a predetermined threshold), the camera MPU 125 may not execute the processes of S705 and S706 according to the determination result of S704. This is because even if an occluded area occurs in the subject area due to an obstruction when digital camera C captures an image, the influence of perspective conflict is low at the obstruction and at the boundary between the obstruction and the subject. This makes it possible to omit the processes of S705 and S706 according to the determination result of S704.

[0070] <Modification> In each of the above-described embodiments, an example has been described in which machine learning is performed by supervised learning using a teacher image with the occluded area as a positive example and the area other than the occluded area as a negative example, and the distribution direction of the occluded area is detected from the image. In this regard, in each of the embodiments, a trained model machine-learned by unsupervised learning may be used. In this case, for example, a training image in which the occluded areas occluding the subject are distributed in the X direction and a training image in which the occluded areas occluding the subject are distributed in the Y direction are used for machine learning. The trained image may be used to generate the trained model used in each of the embodiments by performing machine learning by unsupervised learning. Then, the generated trained model is used for detecting the occluded area.

[0071] For example, when an image in which occluding regions that occlude subjects in a subject region are distributed in the X direction is input to the trained model, the occluded regions distributed in the X direction are extracted as features from the image, and the input image is classified as an image in which occluded regions are distributed in the X direction. Similarly, when an image in which occluding regions that occlude subjects in a subject region are distributed in the Y direction is input to the trained model, the occluded regions distributed in the Y direction are extracted as features from the image, and the input image is classified as an image in which occluded regions are distributed in the Y direction. In this way, the distribution direction of the occluded regions can be determined.

[0072] By using a trained model that has been machine-learned through unsupervised learning as a trained model for detecting occluded regions, it becomes unnecessary to prepare training data (trained images). For example, clustering and principal component analysis can be applied as a machine learning algorithm for unsupervised learning.

[0073] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0074] 120 Camera body 122 Image sensor 125 Camera MPU 129 Imaging surface phase difference detection unit 130 Recognition part 211 pixels 211a-d Photoelectric conversion unit 311 Neural Network System C. Digital camera

Claims

1. an image sensor having a plurality of pixels each having a plurality of photoelectric conversion units that receive light beams passing through different pupil regions of an image capturing optical system, and outputting a signal capable of detecting a first focus state of the image capturing optical system from a phase difference between a pair of images in at least a first direction, and a second focus state of the image capturing optical system from a phase difference between a pair of images in a second direction different from the first direction; a control means for performing focus adjustment in response to a focus detection result based on a signal output from the image sensor; The electronic device is characterized in that, when there is an object blocking the subject, the control means performs focus adjustment based on the detection result of the first focus state when the blocking area of ​​the object extends in a first direction, and performs focus adjustment based on the detection result of the second focus state when the blocking area of ​​the object extends in a second direction.

2. The electronic device of claim 1 , wherein the first direction is a horizontal direction and the second direction is a vertical direction.

3. Rather than controlling the focus adjustment based on the direction in which the occlusion area extends, The electronic device according to claim 1 , wherein focus adjustment control is performed preferentially based on phase difference information in a direction in which contrast exists in the subject area excluding the blocked area.

4. The electronic device according to claim 1 , wherein when a value indicating a contrast of the blocking area is lower than a predetermined threshold, the focus adjustment is not controlled based on a direction in which the blocking area extends.

5. The electronic device according to any one of claims 1 to 4, wherein an occluded area is detected from an image based on the likelihood of the occluded area obtained by inputting an image based on the output from the imaging element into a trained model that has been machine-learned using a plurality of teacher images in which an occluded area in a subject area is treated as a positive example and an area other than the occluded area is treated as a negative example.

6. The electronic device according to claim 5 , wherein the occluded region of the teacher image is a foreground region present in a subject region of the teacher image.

7. The electronic device according to claim 5 , wherein the occluded region of the teacher image is a region in which a depth difference is equal to or greater than a predetermined value.

8. 5. The electronic device according to claim 1, wherein an image based on the output from the imaging element is input to a trained model that has been machine-learned by unsupervised learning using a plurality of training images in which occluded areas of a subject area are distributed horizontally and a plurality of training images in which occluded areas of the subject area are distributed vertically, and an occluded area is detected from the image.

9. an image sensor including a plurality of pixels capable of photoelectrically converting light beams that have passed through different pupil regions of an image capturing optical system and outputting a pair of signals; a focus detection means capable of detecting a first focus state of the photographing optical system from a phase difference between a pair of images in at least a first direction based on an output of the image sensor, and a second focus state of the photographing optical system from a phase difference between a pair of images in a second direction different from the first direction; a control means for performing focus adjustment in response to a focus detection result based on a signal output from the image sensor; The electronic device is characterized in that, when there is an object blocking the subject, the control means performs focus adjustment based on the detection result of the first focus state when the blocking area of ​​the object extends in a first direction, and performs focus adjustment based on the detection result of the second focus state when the blocking area of ​​the object extends in a second direction.

10. The electronic device of claim 9 , wherein the first direction is a horizontal direction and the second direction is a vertical direction.

11. Rather than controlling the focus adjustment based on the direction in which the occlusion area extends, The electronic device according to claim 9 , wherein focus adjustment control is performed preferentially based on phase difference information in a direction in which contrast exists in the subject area excluding the blocked area.

12. The electronic device according to claim 9 , wherein when a value indicating a contrast of the blocking area is lower than a predetermined threshold, the focus adjustment is not controlled based on a direction in which the blocking area extends.

13. The electronic device according to any one of claims 9 to 12, wherein an occluded area is detected from an image based on the likelihood of the occluded area obtained by inputting an image based on the output from the imaging element into a trained model that has been machine-learned using a plurality of teacher images in which an occluded area in a subject area is treated as a positive example and an area other than the occluded area is treated as a negative example.

14. The electronic device according to claim 13 , wherein the occluded region of the teacher image is a foreground region present in a subject region of the teacher image.

15. The electronic device according to claim 13 , wherein the occluded region of the teacher image is a region in which a depth difference is equal to or greater than a predetermined value.

16. The electronic device according to any one of claims 9 to 12, wherein an image based on the output from the imaging element is input to a trained model that has been machine-learned by unsupervised learning using a plurality of training images in which occluded areas of a subject area are distributed horizontally and a plurality of training images in which occluded areas of the subject area are distributed vertically, and an occluded area is detected from the image.

17. A method for controlling an electronic device capable of detecting a first focus state of an imaging optical system from a phase difference between a pair of images in at least a first direction, and a second focus state of the imaging optical system from a phase difference between a pair of images in a second direction different from the first direction, based on an output from an image sensor having a plurality of pixels, each pixel having a plurality of photoelectric conversion units that receive light beams passing through different pupil regions of the imaging optical system, the method comprising: A control step of performing focus adjustment in response to a focus detection result based on a signal output from the image sensor, A control method for an electronic device, characterized in that in the control process, when there is an object blocking the subject, focus adjustment is performed based on the detection result of the first focus state when the blocking area of ​​the object extends in a first direction, and focus adjustment is performed based on the detection result of the second focus state when the blocking area of ​​the object extends in a second direction.

18. A method for controlling an electronic device having an image sensor including a plurality of pixels capable of photoelectrically converting light beams passing through different pupil regions of an image capturing optical system and outputting a pair of signals, comprising the steps of: a focus detection step capable of detecting a first focus state of the photographing optical system from a phase difference between a pair of images in at least a first direction based on an output of the image sensor, and a second focus state of the photographing optical system from a phase difference between a pair of images in a second direction different from the first direction; A control step of performing focus adjustment in response to a focus detection result based on a signal output from the image sensor, A control method for an electronic device, characterized in that in the control process, when there is an object blocking the subject, focus adjustment is performed based on the detection result of the first focus state when the blocking area of ​​the object extends in a first direction, and focus adjustment is performed based on the detection result of the second focus state when the blocking area of ​​the object extends in a second direction.

19. A program for causing a computer to execute each of the means of the electronic device according to any one of claims 1 to 16.

Images