Image capturing device and control method therefor

Abstract

To provide a technique that enables acquisition of a good defocus amount even in the presence of other objects at different distances from a target object.SOLUTION: Pixel values of a region not classified as a region of a target object or classified as a specific object according to class classification of at least one image in a parallax image are updated based on pixel values of other regions. A defocus amount is determined based on an updated parallax image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus and a control method thereof. [Background technology]

[0002] Cameras equipped with a focus adjustment function that automatically adjusts the focal position of the lens are becoming widespread. Various AF methods, such as phase difference AF and contrast AF, are in practical use to adjust the focus.

[0003] The phase-difference AF method has the advantage that it can directly calculate the amount of deviation in the focal plane from two images with parallax, making it possible to focus more quickly than the contrast AF method. In addition, in the field of autonomous driving, technologies that obtain depth information of a subject using dedicated distance sensors such as LiDAR (Light Detection and Ranging, Laser Imaging Detection) and ToF (Time of Flight) in addition to cameras have also been put to practical use. In general, depth information has a lower resolution than RGB images captured by cameras, and is prone to errors due to sampling.

[0004] In contrast, the invention disclosed in Patent Document 1 labels an image captured by a camera using semantic region segmentation, and obtains the depth of a target pixel by interpolating the depth of a pixel without depth information among pixels of a subject that has been assigned a specific label from pixels with depth information that have been assigned the same label.

[0005] In addition, the invention disclosed in Patent Document 2 proposes a method of outputting a defocus map that has been passed through filters of different frequency bands depending on the subject, and suppressing the effect of obstacles in front of the subject on the defocus amount at a stage prior to sampling. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2019-28861 A [Patent Document 2] JP 2021-176009 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the invention disclosed in Patent Document 1, the camera is fixed to a moving object and interpolation and extrapolation are performed under the assumption that the depth within the same label is the same. Therefore, there was a problem that it was difficult to apply to cases where the optical axis is not necessarily parallel to the ground, such as when taking pictures with a general camera.

[0008] In the invention disclosed in Patent Document 2, the frequency band of an obstacle relative to the subject is assumed in advance, so when the subject and the obstacle are not the intended combination, it is difficult to effectively avoid the obstacle and calculate the defocus amount. The present invention provides a technology that makes it possible to obtain a good defocus amount even when there are objects with different distances to the subject. [Means for solving the problem]

[0009] One aspect of the present invention is characterized in that it comprises an updating means for updating pixel values ​​of an area in a disparity image that is not classified as a subject area or that is classified as a specific subject by class classification for at least one image, based on pixel values ​​of the other area, and a calculating means for calculating a defocus amount based on the disparity image after the update. Effect of the Invention

[0010] According to the present invention, it is possible to obtain a good defocus amount even when other objects at different distances from the subject are mixed. [Brief description of the drawings]

[0011] [Figure 1] 1A and 1B are diagrams for explaining the principle of achieving focus detection using a phase difference detection method. [Diagram 2] FIG. 1 is a block diagram showing an example of the hardware configuration of an imaging device 100. [Diagram 3] 5 is a flowchart of a process performed by the imaging device 100 to calculate a defocus amount based on parallax images captured by the imaging device 100. [Figure 4] 11 is a flowchart showing details of subject loss correction processing in step S304. [Diagram 5] FIG. 11 is a diagram for explaining the process in step S405. [Figure 6] 5 is a flowchart of a process performed by the imaging device 100 to calculate a defocus amount based on parallax images captured by the imaging device 100. [Figure 7] 5 is a flowchart of a process performed by the imaging device 100 to calculate a defocus amount based on parallax images captured by the imaging device 100. [Figure 8] A diagram showing images A and B when two people are photographed through a net. [Figure 9] 1 is a schematic diagram showing the positional relationship between representative cell A and representative cell B with respect to the shift amount of correlation calculation, and a diagram showing the corresponding correlation amount and occlusion flag overlap rate, respectively. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0013] [First embodiment] First, the principle of achieving focus detection using the phase difference detection method in the imaging device 100 according to this embodiment will be described with reference to Fig. 1. Fig. 1 shows a schematic diagram of a light beam emitted from the exit pupil of a photographing lens entering one of a plurality of unit pixels 200 of an imaging element. Fig. 2 shows an example of the hardware configuration of the imaging device 100 according to this embodiment.

[0014] The unit pixel 200 has a photodiode 201A which is a first photoelectric conversion unit, and a photodiode 201B which is a second photoelectric conversion unit. In addition, a color filter 202 and a microlens 203 are provided corresponding to the unit pixel 200.

[0015] The center of a light beam emitted from an exit pupil 204 of the photographing lens with respect to a unit pixel 200 in which a microlens 203 is provided is defined as an optical axis 205. The light that has passed through the exit pupil 204 is incident on the unit pixel 200 via the microlens 203 and the color filter 202, with the optical axis 205 as the center. The pupil region 206 and the pupil region 207 in the exit pupil 204 are partial regions of the exit pupil of the photographing lens 111 shown in FIG. 2. As shown in FIG. 1, the light beam that passes through the pupil region 206 is received by the photodiode 201A through the microlens 203 and the color filter 202. On the other hand, the light beam that passes through the pupil region 207 is received by the photodiode 201B through the microlens 203 and the color filter 202. In this way, the photodiode 201A and the photodiode 201B each receive light from a different region divided by the exit pupil 204 of the photographing lens. That is, the image sensor 101 of the image capturing device 100 includes a pixel unit having a first photoelectric conversion unit and a second photoelectric conversion unit for each microlens 203, which generate an image signal by photoelectrically converting a light beam that has passed through different divided regions of an exit pupil of the image capturing optical system for the microlens 203. The image capturing device 100 can obtain a defocus amount based on a phase difference between an image based on an image signal output by the photodiode 201A and an image based on an image signal output by the photodiode 201B, and perform a focus adjustment process based on the defocus amount.

[0016] If the image signal obtained from the photodiode 201A is an image signal A and the image signal obtained from the photodiode 201B is an image signal B, the imaging device 100 can use the (A+B) image signal obtained by adding the image signals A and B as the imaging signal of the unit pixel 200.

[0017] Next, an example of the hardware configuration of the imaging device 100 according to this embodiment will be described with reference to the block diagram of Fig. 2. The photographing lens 111 forms an image of external light on the image sensor 101. The lens driving circuit 110 drives and controls the photographing lens 111 under the control of the control circuit 104. More specifically, the lens driving circuit 110 performs zoom control, focus control, aperture control, and the like for the photographing lens 111.

[0018] The image sensor 101 is a CMOS image sensor in which pixels each having two pairs of pupil-divided photoelectric conversion units (for example, the above-mentioned photodiode 201A and photodiode 201B) are arranged in a matrix. Each photoelectric conversion unit in the image sensor 101 photoelectrically converts light incident via the photographing lens 111 and outputs an image signal. The timing generation circuit 102 outputs a timing signal for driving the image sensor 101.

[0019] The signal processing circuit 103 acquires parallax images (a pair of an image based on an image signal from one photoelectric conversion unit and an image based on an image signal from the other photoelectric conversion unit) by performing various corrections and compressions on the image signals output from the image sensor 101. Then, the signal processing circuit 103 stores the acquired parallax images in the memory 105.

[0020] The control circuit 104 controls the overall operation of the imaging device 100, and executes or controls various processes described as processes performed by the imaging device 100. For example, the control circuit 104 performs focus detection using a phase difference detection method and calculates a defocus amount based on the parallax images.

[0021] The control circuit 104 generally includes a CPU (Central Processing Unit). However, if the imaging device 100 includes an FPGA (Field Programming Gate Array), a configuration may be adopted in which part of the processing of the CPU is executed by the FPGA. The control circuit 104 may also be an ASIC (Application Specific Integrated Circuit) that combines a CPU and an FPGA into one package.

[0022] The memory 105 has an area for storing the parallax images output from the signal processing circuit 103, and an area for storing computer programs and data loaded from the non-volatile memory 106. The memory 105 further has a work area used when the control circuit 104 executes various processes. In this way, the memory 105 can provide various areas as appropriate.

[0023] The non-volatile memory 106 stores setting data for the imaging device 100, computer programs and data related to the startup of the imaging device 100, computer programs and data related to the basic operation of the imaging device 100, etc. The non-volatile memory 106 also stores computer programs and data for causing the control circuit 104 to execute or control various processes described as processes performed by the imaging device 100. The data stored in the non-volatile memory 106 includes various thresholds, adjustment values ​​that differ for each imaging device 100, etc.

[0024] The display circuit 107 has a touch panel screen or a liquid crystal screen, and displays images (for example, one or both images in a parallax image) and characters (for example, the current date and time, character strings on a setting screen) under the control of the control circuit 104.

[0025] The recording circuit 108 is a circuit for reading and writing information from and to a memory (a removable memory such as a semiconductor memory) that is detachable from the imaging device 100 under the control of the control circuit 104. An external storage device may be used to fulfill a similar role. Here, the external storage device can be realized, for example, by a medium (recording medium) and an external storage drive for realizing access to the medium. Known examples of such media include a flexible disk (FD), a CD-ROM, a DVD, a USB memory, an MO, and a flash memory. The external storage device may also be a server device or the like connected to the imaging device 100 via a network.

[0026] The operation circuit 109 is a user interface such as a switch, a button, a touch panel, etc., and can input various information and instructions to the imaging device 100 by a user operating it.

[0027] Next, a process for calculating a defocus amount based on a parallax image captured by the imaging device 100 will be described with reference to the flowchart of FIG. 3. In step S301, each photoelectric conversion unit in the imaging element 101 photoelectrically converts light incident through the photographing lens 111 to output an image signal. The signal processing circuit 103 acquires an image (hereinafter referred to as an A image) by performing various corrections and compressions on an image signal output from a photoelectric conversion unit corresponding to a first photoelectric conversion unit in a pixel group in the imaging element 101. The signal processing circuit 103 also acquires an image (hereinafter referred to as a B image) by performing various corrections and compressions on an image signal output from a photoelectric conversion unit corresponding to a second photoelectric conversion unit in a pixel group in the imaging element 101. The signal processing circuit 103 then stores the acquired set of A and B images (parallax image) in the memory 105.

[0028] In step S302, the control circuit 104 performs a subject region extraction process for each of the images A and B stored in the memory 105. For example, the control circuit 104 performs semantic region classification (class classification) for each pixel of each of the images A and B to divide the images into regions for each attribute (class), and extracts regions of subjects such as people from the divided regions as subject regions.

[0029] In step S303, the control circuit 104 determines, for each of the A and B images stored in the memory 105, the areas that have not been extracted as subject areas in the images as non-subject areas (missing areas).

[0030] In step S304, the control circuit 104 performs a subject defect correction process for correcting pixel values ​​of pixels that are originally pixels in the subject region but are determined to be pixels in the non-subject region (defective portion) for each of the images A and B. Details of the subject defect correction process in step S304 will be described later with reference to FIG.

[0031] In step S305, the control circuit 104 sets a focus detection area common to the images A and B that have been subjected to the subject loss correction process in step S304. The focus detection area is further divided into a plurality of cells, and each cell is set to a size sufficient to sample the defocus amount of the main subject (for example, the size of one cell is half the size of the face of the main subject).

[0032] In step S306, the control circuit 104 performs vertical row averaging, which is a process of calculating an average value of pixel values ​​in each column in a cell for each cell in each of the images A and B, to calculate a waveform represented by the average value of each cell. Hereinafter, the waveform calculated for the image A will be referred to as the image A waveform, and the waveform calculated for the image B will be referred to as the image B waveform.

[0033] In step S307, the control circuit 104 performs a shift process to relatively shift the A-image waveforms and the B-image waveforms in the pupil division direction (parallax direction) between positionally corresponding cells in the A-image and the B-image, and performs a correlation calculation to calculate a correlation amount that indicates the degree of signal coincidence.

[0034] The image A waveform in the kth column of a cell in the image A is denoted by A(k), the image B waveform in the kth column of a corresponding cell in the image B is denoted by B(k), the range of the number k corresponding to the focus detection area is denoted by W, the shift amount by the shift process is denoted by s, and the shift range of the shift amount s is denoted by Γ. At this time, the control circuit 104 calculates the correlation amount COR by using equation (1).

[0035]

number

[0036] That is, the control circuit 104 generates a difference between the kth A image waveform A(k) and the (ks)th B image waveform B(ks) by shifting the image by the shift amount s as a shift subtraction signal. The control circuit 104 then sums up the absolute values ​​of the generated shift subtraction signals for the number k within a range W corresponding to the focus detection area, and calculates the sum as the correlation amount COR(s).

[0037] In step S308, the control circuit 104 determines, from the correlation amount determined in step S307, the shift amount of real values ​​at which the correlation amount becomes the minimum value, as the "image shift amount", by sub-pixel calculation.

[0038] In step S309, the control circuit 104 obtains a defocus amount by multiplying the image shift amount obtained in step S308 by a conversion coefficient. The control circuit 104 may store the defocus amount in the memory 105, or may control the lens driving circuit 110 to perform focus adjustment processing based on the defocus amount.

[0039] Next, the subject loss correction process in step S304 described above will be explained according to the flowchart of FIG. 4. The process according to the flowchart of FIG. 4 is performed for each of the A image and the B image. Hereinafter, the process for the A image will be described, but the same description also applies to the B image.

[0040] In step S401, the control circuit 104 registers all the person regions Ak (k∈{0,1,2,…,n−1}) extracted by semantic region division for the A image in the memory 105.

[0041] In step S402, the control circuit 104 assigns a region (contour Lk) including the person region Ak to all the person regions Ak. In step S403, the control circuit 104 initializes the value of a variable i used as a counter in the following process to 0.

[0042] In step S404, the control circuit 104 determines whether there is an overlapping region (common area) between the contour Li and the contour Lj for all j that satisfy i<j. As a result of this determination, if there is one or more common areas, the process proceeds to step S405, and if there is no common area, the process proceeds to step S407.

[0043] In step S405, the control circuit 104 updates the pixel value of each pixel included in the common area to a new pixel value. The new pixel value is obtained by performing interpolation using the pixel values of the pixels within each overlapping contour in the common area. The process in step S405 will be described later with reference to FIG. 5.

[0044] In step S406, the control circuit 104 stores in the memory 105 the position information defining the region whose pixel value has been updated by interpolation in step S405, that is, the common area (information indicating the coordinates of two opposite corners in the common area when the common area is rectangular).

[0045] In step S407, the control circuit 104 stores the position information defining the person area Ai (when the person area Ai is rectangular, the information indicating the coordinates of two opposite corners in the person area Ai) in the memory 105.

[0046] In step S408, the control circuit 104 increments the value of the variable i by 1. Then in step S409, the control circuit 104 determines whether i < n - 1. As a result of this determination, if i < n - 1, the process proceeds to step S404, and if i ≥ n - 1, the process proceeds to step S410.

[0047] In step S410, the control circuit 104 sets the image within the area (all person areas Ak, common area) defined by the position information stored in the memory 105 in image A as the corrected image A (defect correction A image) corrected by the subject defect correction process.

[0048] Since the defect correction B image can be obtained by performing the process according to the flowchart of FIG. 4 on image B as well, in step S305, the defect correction A image and the defect correction B image are used as the A image and the B image on which the subject defect correction process has been performed, respectively.

[0049] Next, the process in step S405 described above will be described taking FIG. 5 as an example. In FIG. 5, a case is shown where there are a common area C123 where three contours, namely the contour L1 of the person area A1, the contour L2 of the person area A2, and the contour L3 of the person area A3, overlap each other, a common area C13 where two contours, namely the contour L1 and the contour L3, overlap each other, and a common area C23 where two contours, namely the contour L2 and the contour L3, overlap each other.

[0050] Generally, in the contour assignment process, a range equidistant from the outer periphery of the area is often assigned as the contour. However, in FIG. 5, for the sake of explanation, a rectangular area 3 pixels wider than the outer periphery is given as the contour. It is shown with a slight shift instead of at a precise interval for the sake of explanation.

[0051] First, a method for determining a new pixel value of a pixel (interpolated pixel) Px included in the common area C123 by interpolation will be described. The interpolated pixel Px is a pixel included in the three regions of the contours L1 to L3. The control circuit 104 then identifies pixel P1, which is the corner (vertex) closest to the interpolated pixel Px in the person region A1, pixel P2, which is the corner (vertex) closest to the interpolated pixel Px in the person region A2, and pixel P3, which is the corner (vertex) closest to the interpolated pixel Px in the person region A3. The control circuit 104 then determines whether or not the interpolated pixel Px is located inside a triangle T whose vertices are pixels P1, P2, and P3. In the case of FIG. 5, the interpolated pixel Px is located inside the triangle T. The control circuit 104 then determines a new pixel value of the interpolated pixel Px by bilinear interpolation using the pixel values ​​of pixels P1, P2, and P3, and updates the pixel value of the interpolated pixel Px to the determined new pixel value.

[0052] Next, a method for determining, by interpolation, a new pixel value of a pixel (interpolated pixel) Py included in the common area C23 will be described. The interpolated pixel Py is a pixel included in two areas of contours L2 and L3, and is located inside triangle T. The control circuit 104 determines a new pixel value of the interpolated pixel Py by bilinear interpolation using the pixel values ​​of pixels P1, P2, and P3, and updates the pixel value of the interpolated pixel Py to the determined new pixel value.

[0053] Next, a method for determining, by interpolation, a new pixel value of pixel (interpolated pixel) Pz included in common area C23 will be described. Interpolated pixel Pz is a pixel included in two areas of contours L2 and L3, and is located outside triangle T. Therefore, control circuit 104 determines a new pixel value of interpolated pixel Pz by linear interpolation using the pixel values ​​of pixels P2 and P3, and updates the pixel value of interpolated pixel Pz to the determined new pixel value.

[0054] Even if the common area of ​​the contour is shared by four or more regions, the calculation can be performed in the same way by interpolating the pixels with the most common number. However, in order to reduce the amount of calculation, it is also possible to always consider only three regions, namely, the two regions Lj and Lk whose indexes are close to Li.

[0055] In this way, by extracting the subject area and performing phase difference calculation, it is possible to calculate a defocus amount that suppresses the influence of obstacles in front. The process of adjusting the focus on the subject to be focused on can be properly performed using the calculated defocus amount. In this embodiment, in order to completely remove the influence of obstacles during correlation calculation, area extraction and defect correction are performed on each of the A image and the B image, but even if defect correction processing is performed only on the A image, the correlation amount of the obstacle can be suppressed to some extent. When there are restrictions on processing time and power consumption, such a configuration can be selected.

[0056] [Second embodiment] In the following embodiments including this embodiment, the differences from the first embodiment will be described, and unless otherwise specified below, it is assumed that the embodiments are the same as the first embodiment. In the first embodiment, a person area is extracted and missing parts in the person area are interpolated to calculate a defocus amount that suppresses the influence of obstacles in front, but when there are a large number of small person areas, the amount of calculation becomes significantly large.

[0057] Therefore, in this embodiment, the missing portion is an obstacle region (obstacle region) rather than a non-subject region. Here, an obstacle refers to an object that blocks the main subject, such as a net, a wire mesh, or a fence. In this embodiment, an obstacle region detector is used that uses a learning model that has been trained in advance so that the obstacle region can be detected by semantic region segmentation.

[0058] The process of calculating the defocus amount based on the parallax images captured by the imaging device 100 will be described with reference to the flowchart of Fig. 6. In the flowchart of Fig. 6, the same process steps as those shown in Fig. 3 are denoted by the same step numbers, and the description of these process steps will be omitted.

[0059] In step S602, the control circuit 104 performs obstacle region extraction using the obstacle region detector for each of the images A and B stored in the memory 105. For example, the control circuit 104 inputs the image A to the obstacle region detector and performs calculations of the obstacle region detector to obtain the likelihood of each pixel in the image A being an obstacle region, and extracts a region consisting of pixels with a likelihood equal to or greater than a threshold as an obstacle region in the image A. The control circuit 104 performs the same process for the image B.

[0060] In step S603, the control circuit 104 determines the obstacle area in each of the images A and B stored in the memory 105 as a missing part. In step S604, the control circuit 104 obtains new pixel values ​​of each pixel in the obstacle area, which is the missing part, by interpolation, and updates the pixel value of the pixel to the new pixel value. In this embodiment, the missing part is limited to the obstacle area, so that pixels not included in the obstacle area (non-missing pixels) can be easily found from the periphery of the pixels included in the obstacle area (missing pixels). Therefore, in this embodiment, a complex algorithm such as that shown in FIG. 4 is not necessary, and the new pixel value of the missing pixel can be uniformly calculated from the non-missing pixels existing in the horizontal and vertical directions of the missing pixel by the nearest neighbor method or bilinear interpolation. In this embodiment, the new pixel value of the missing pixel is obtained by using bilinear interpolation. In step S605, the control circuit 104 sets a focus detection area common to the images A and B processed in step S604 in the same manner as in step S305 described above.

[0061] In this way, by extracting the obstacle area as a missing part, it is possible to calculate a defocus amount that suppresses the influence of the obstacle in front while greatly reducing the amount of calculation. In addition, by using a macro object position detector in combination as in object detection, it is possible to limit the area that requires interpolation, and further reduce the amount of calculation required for interpolation. Note that an obstacle is merely one example of a specific type of subject.

[0062] [Third embodiment] In this embodiment, the defocus amount is calculated without performing an interpolation process, so that the amount of calculation can be reduced even more than in the first and second embodiments. The process performed by the imaging device 100 to calculate the defocus amount based on the captured parallax images will be described with reference to the flowchart in Fig. 7. In the flowchart in Fig. 7, the same process steps as those shown in Figs. 3 and 6 are denoted by the same step numbers, and the description of these process steps will be omitted.

[0063] In step S703, the control circuit 104 sets a common focus detection area for the images A and B acquired in step S301 in the same manner as in step S305 above. In step S704, the control circuit 104 sets a flag (occlusion flag) in the occlusion column of the focus detection area set for each of the images A and B. Here, an occlusion column refers to a column in which the proportion of obstacle areas in the column direction is high among the cells in the focus detection area, for example, a column in which obstacle areas account for 50% or more in the column direction. In other words, an occlusion column is a column in which obstacle areas exist at a rate equal to or greater than a threshold value in columns perpendicular to the parallax direction (obstacle-present column).

[0064] Figure 8 shows images A and B when two people are photographed through a net, and the occlusion column when focusing on cells (representative cell A, representative cell B) in one of the focus detection areas is indicated by an upward arrow in the lower row. The occlusion column looks like a single line, but in reality it has a certain thickness and is also curved due to bending, so even a thin net will have about 5 to 10 columns.

[0065] In step S705, the control circuit 104 performs vertical row averaging for each focus detection region to obtain an A image waveform and a B image waveform for each focus detection region. In step S706, the control circuit 104 performs shift processing to relatively shift the A image waveform and the B image waveform in the pupil division direction, and calculates a correlation amount that indicates the degree of signal matching using equation (1). At this time, the overlap rate OR of the occlusion flag is also calculated as shown in FIG.

[0066] The occlusion flag in the kth column in the A image is a(k)∈{0,1}, the occlusion flag in the kth column in the B image is b(k)∈{0,1}, and the range of the number k corresponding to the representative cell is W. The shift amount by the shift process is s, and the shift range of the shift amount s is Γ. At this time, the control circuit 104 obtains the overlap rate OR of the occlusion flags by calculating the following formula (2).

[0067]

number

[0068] By shifting by the shift amount s, the occlusion flag a(k) in the kth column in image A and the occlusion flag b(ks) in the (ks)th column in image B are matched and a logical product is taken, and the overlap of the occlusion flags is counted for each column. The denominator is a normalization term, and the smaller of the number of columns of the occlusion flags in image A and image B is used. The occlusion flag overlap rate OR(s) is calculated by dividing the numerator by the normalization term.

[0069] In step S707, the control circuit 104 obtains, from the correlation amount, a real-valued shift amount at which the correlation amount is the minimum value by sub-pixel calculation as an "image shift amount candidate." In step S708, the control circuit 104 obtains the overlap rate of the occlusion strings of the images A and B when the image shift amount candidate is obtained, and judges whether the overlap rate is less than 80% (less than a threshold value). If the overlap rate is 80% or more as a result of this judgment, the process proceeds to step S707, and an image shift amount candidate with the next smallest correlation amount after the previously obtained image shift amount candidate is searched for. On the other hand, if the overlap rate of the occlusion strings of the images A and B is less than 80%, the process proceeds to step S709. In step S709, the control circuit 104 determines the image shift amount candidate in which the overlap rate of the occlusion strings of the images A and B is less than 80% as the "image shift amount."

[0070] 9 is a schematic diagram showing the positional relationship between representative cell A and representative cell B with respect to the shift amount of the correlation calculation, and shows the corresponding correlation amount and occlusion flag overlap rate. There are two points in front of the focal plane where the correlation amount is minimal. Of these, image shift amount s1 (the one closer to the imaging device 100) has an occlusion flag overlap rate of more than 80%. Therefore, it is not used as the image shift amount, and the other image shift amount s2 has an occlusion flag overlap rate of 0%, so the image shift amount s2 can be used to calculate the defocus amount.

[0071] In step S710, the control circuit 104 obtains the defocus amount by multiplying the image shift amount determined in step S709 (image shift amount s2 in the example of FIG. 9) by a conversion coefficient.

[0072] In this way, by extracting the obstacle area and performing correlation calculations with the obstacle area masked, it is possible to calculate the desired defocus amount without performing interpolation processing, which requires a large amount of calculation.

[0073] The numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to these examples.

[0074] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0075] (Other embodiments) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0076] The invention of this specification includes the following imaging device and method for controlling the imaging device. (Item 1) an updating means for updating pixel values ​​of a region in the parallax image that is not classified as a subject region or that is classified as a specific subject by class classification for at least one of the images, based on pixel values ​​of the other region; A calculation means for calculating a defocus amount based on the updated parallax image; An imaging device comprising: (Item 2) The imaging device according to item 1, characterized in that the parallax image is an image acquired by an imaging element having a pixel unit that generates an image signal by photoelectrically converting light beams that have passed through different divided areas of an exit pupil of an imaging optical system relative to a microlens. (Item 3) The imaging device described in item 1 or 2, characterized in that the update means interpolates pixel values ​​of areas in the image that are not classified as subject areas by semantic area classification for each pixel in the image based on pixel values ​​of areas classified as subject areas by semantic area classification for each pixel in the image. (Item 4) The imaging device described in item 1 or 2, characterized in that the update means interpolates pixel values ​​of areas in the image that are classified as obstacle areas by semantic area classification for each pixel, based on pixel values ​​of areas in the image that are not classified as obstacle areas by semantic area classification for each pixel. (Item 5) The imaging device described in any one of items 1 to 4, characterized in that the calculation means calculates an image shift amount in a focus detection area set in the updated parallax image, and calculates a defocus amount based on the image shift amount. (Item 6) An acquisition means for acquiring a parallax image; a calculation means for determining an obstacle presence row as a row in which an obstacle region exists at a ratio equal to or greater than a threshold value with respect to rows perpendicular to a parallax direction in focus detection areas set for each image in the parallax image, and calculating a defocus amount based on a result in which an overlap rate of the obstacle presence row is less than a threshold value among results obtained by shifting each focus detection area relatively to each other in the parallax direction and performing a correlation calculation; An imaging device comprising: (Item 7) 7. The imaging device according to item 6, wherein the overlap rate of the obstacle presence sequences is a result of dividing the number of overlapping obstacle presence sequences of each image in the disparity image at the time of the shift related to the correlation calculation by the lesser number of the obstacle presence sequences of each image. (Item 8) A control method for an imaging device, comprising: an updating step in which an updating unit of the imaging device updates pixel values ​​of a region in the parallax image that is not classified as a subject region or a region that is classified as a specific subject by class classification for at least one of the images, based on pixel values ​​of another region; a calculation step in which a calculation means of the imaging device calculates a defocus amount based on the updated parallax image; 13. A method for controlling an imaging apparatus comprising: (Item 9) A control method for an imaging device, comprising: An acquisition step in which an acquisition means of the imaging device acquires a parallax image; a calculation step in which a calculation means of the imaging device determines, as an obstacle presence column, a column in which an obstacle region exists at a ratio equal to or greater than a threshold value with respect to columns perpendicular to the parallax direction in focus detection regions set for each image in the parallax image, and determines a defocus amount based on a result in which an overlap rate of the obstacle presence column is less than a threshold value among results obtained by shifting each focus detection region relatively to each other in the parallax direction and performing a correlation calculation; 13. A method for controlling an imaging apparatus comprising:

[0077] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0078] 100: Imaging device 101: Imaging element 102: Timing generation circuit 103: Signal processing circuit 104: Control circuit 105: Memory 106: Non-volatile memory 107: Display circuit 108: Recording circuit 109: Operation circuit 110: Lens driving circuit 111: Imaging lens

Claims

1. an updating means for updating pixel values ​​of a region in the parallax image that is not classified as a subject region or that is classified as a specific subject by class classification for at least one of the images, based on pixel values ​​of the other region; A calculation means for calculating a defocus amount based on the updated parallax image; An imaging device comprising:

2. 2. The imaging device according to claim 1, wherein the parallax image is an image acquired by an imaging element having a pixel unit that generates an image signal by photoelectrically converting light beams that have passed through different divided areas of an exit pupil of an imaging optical system relative to a microlens.

3. The imaging device according to claim 1, characterized in that the update means interpolates pixel values ​​of areas in the image that have not been classified as subject areas by semantic area classification for each pixel in the image, based on pixel values ​​of areas classified as subject areas by semantic area classification for each pixel in the image.

4. The imaging device according to claim 1, characterized in that the update means interpolates pixel values ​​of areas in the image that are classified as obstacle areas by semantic area classification for each pixel, based on pixel values ​​of areas in the image that are not classified as obstacle areas by semantic area classification for each pixel.

5. 2. The imaging apparatus according to claim 1, wherein the calculation means calculates an image shift amount in a focus detection area set in the updated parallax image, and calculates a defocus amount based on the image shift amount.

6. An acquisition means for acquiring a parallax image; a calculation means for determining an obstacle-present row as a row in which an obstacle region exists at a ratio equal to or greater than a threshold value with respect to rows perpendicular to a parallax direction in focus detection areas set for each image in the parallax image, and calculating a defocus amount based on a result in which an overlap rate of the obstacle-present row is less than a threshold value among results obtained by shifting each focus detection area relatively to each other in the parallax direction and performing a correlation calculation; An imaging device comprising:

7. 7. The imaging device according to claim 6, wherein the overlap rate of the obstacle presence sequences is a result of dividing the number of overlapping obstacle presence sequences of each image in the disparity image at the time of the shift related to the correlation calculation by the lesser number of the obstacle presence sequences of each image.

8. A control method for an imaging device, comprising: an updating step in which an updating unit of the imaging device updates pixel values ​​of a region in the parallax image that is not classified as a subject region or a region that is classified as a specific subject by class classification for at least one of the images, based on pixel values ​​of another region; a calculation step in which a calculation means of the imaging device calculates a defocus amount based on the updated parallax image; 13. A method for controlling an imaging apparatus comprising:

9. A control method for an imaging device, comprising: An acquisition step in which an acquisition means of the imaging device acquires a parallax image; a calculation step in which a calculation means of the imaging device determines, as an obstacle presence column, a column in which an obstacle region exists at a ratio equal to or greater than a threshold value with respect to columns perpendicular to the parallax direction in focus detection regions set for each image in the parallax image, and determines a defocus amount based on a result in which an overlap rate of the obstacle presence column is less than a threshold value among results obtained by shifting each focus detection region relatively to each other in the parallax direction and performing a correlation calculation; 13. A method for controlling an imaging apparatus comprising:

Images