Imaging device, method for driving the imaging device, and program

Abstract

Provided are an image capturing device, a method for driving an image capturing device, and a program that make it possible to accurately track a subject. An image capturing device according to the present invention is provided with: an image sensor that has a plurality of phase-difference pixels and that outputs phase-difference information and a captured image; and at least one processor. The processor is configured to acquire, on the basis of the phase-difference information, subject distance information indicating the distance from a subject present in a focusing target region and surrounding distance information indicating the distance from an object present in a region surrounding the focusing target region.

Description

Technical Field

[0001] The technology of the present disclosure relates to an imaging device, a driving method of the imaging device, and a program.

Background Art

[0002] Patent Document 1 discloses an imaging device having focus detection means for detecting a defocus amount for each of a plurality of predetermined focus detection regions from an image signal output from an imaging element, creation means for creating distance distribution information based on the defocus amount, focus adjustment means for performing focus adjustment based on the distance distribution information and the defocus amount, and control means for controlling such that when creating distance distribution information by the creation means, shooting is performed with an aperture included in the imaging optical system set to a first aperture value that gives a first depth of field, and when performing focus adjustment by the focus adjustment means, shooting is performed with the aperture set to a second aperture value that gives a second depth of field shallower than the first depth of field.

[0003] Patent Document 2 discloses an imaging device having imaging means for generating an imaging image, distance map acquisition means, distance map management means, focus range indication means, focusability determination means, lens setting determination means, and display means, wherein the focusability determination means determines whether a range indicated by the focus range indication means is a refocusable range, the lens setting determination means determines whether to change the lens setting according to the determination result of the focusability determination means, and the display means displays information regarding lens setting change according to the determination result of the lens setting determination means.

[0004] Patent Document 3 discloses an imaging device comprising: an image sensor having pupil-divided pixels; a reading means for reading signals from each pixel of the image sensor; a setting means for setting regions from which signals with different parallaxes are read by the pupil-divided pixels by the reading means; a first information acquisition means for acquiring first depth information for detecting a subject using signals read from the first region set by the setting means; a second information acquisition means for acquiring second depth information for detecting the focus state of a subject using signals read from the second region set by the setting means; and a control means for variably controlling the proportion of the screen in which the first region is set and the proportion of the screen in which the second region is set by the setting means. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2018-017876 [Patent Document 2] Japanese Patent Publication No. 2019-023679 [Patent Document 3] Japanese Patent Publication No. 2017-194654 [Overview of the project] [Problems that the invention aims to solve]

[0006] One embodiment of the technology of this disclosure provides an imaging device, a method for driving the imaging device, and a program that enable accurate tracking of a subject. [Means for solving the problem]

[0007] To achieve the above objective, the imaging device of the present disclosure comprises an image sensor having a plurality of phase difference pixels and outputting phase difference information and an captured image, and at least one processor, wherein the processor is configured to acquire subject distance information representing the distance of a subject in the focus area and peripheral distance information representing the distance of an object in the peripheral area of ​​the focus area, based on the phase difference information.

[0008] Preferably, the system includes a focusing lens, and the processor is configured to perform focusing control, which controls the position of the focusing lens based on subject distance information.

[0009] Preferably, the processor is configured to detect objects present between the subject and the imaging device based on subject distance information and surrounding distance information, and to change the focus control when the distance of an object within the field of view relative to the subject decreases.

[0010] The processor is preferably configured to estimate the position of a subject based on the subject's past position when an object occludes the subject.

[0011] Preferably, the processor is configured to move the focus area to the estimated position of the subject, and if the subject is not detected within the moved focus area, to move the focus area to the position of the object.

[0012] Preferably, the processor is configured to record the captured image and distance distribution information corresponding to the captured image, and to acquire subject distance information and surrounding distance information based on the distance distribution information.

[0013] The processor is preferably configured to generate and record an image file containing the captured image and distance distribution information.

[0014] Preferably, the peripheral distance information included in the distance distribution information includes the relative distance of objects within the peripheral region to the area of ​​focus.

[0015] Preferably, the processor is configured to perform correction processing on at least one of the in-focus area and the surrounding area of ​​the captured image based on distance distribution information.

[0016] The processor is preferably configured to modify the correction process applied to the object according to its relative distance.

[0017] The correction process for the object is preferably chromatic aberration correction.

[0018] The distance distribution information includes distance information corresponding to a plurality of pixels constituting the captured image, and it is preferable that the processor is configured to generate a composite image by synthesizing a stereoscopic image with the captured image using the distance information.

[0019] The driving method of the imaging device according to the present disclosure is a driving method of an imaging device including an image sensor having a plurality of phase difference pixels and outputting phase difference information and a captured image. Based on the phase difference information, subject distance information representing the distance of a subject existing in the focusing target area and peripheral distance information representing the distance of an object existing in the peripheral area of the focusing target area are acquired.

[0020] The program according to the present disclosure is a program for operating an imaging device including an image sensor having a plurality of phase difference pixels and outputting phase difference information and a captured image. Based on the phase difference information, the imaging device is caused to execute a process of acquiring subject distance information representing the distance of a subject existing in the focusing target area and peripheral distance information representing the distance of an object existing in the peripheral area of the focusing target area.

Brief Description of the Drawings

[0021] [Figure 1] It is a diagram showing an example of the internal configuration of the imaging device. [Figure 2] It is a diagram showing an example of the configuration of the imaging pixel. [Figure 3] It is a diagram showing an example of the configuration of the phase difference pixel. [Figure 4] It is a diagram showing an example of the pixel array of the imaging sensor. [Figure 5] It is a block diagram showing an example of the functional configuration of the processor. [Figure 6] It is a diagram conceptually showing an example of the distance distribution information acquisition process. [Figure 7] It is a diagram conceptually showing an example of the encoding process by the LBE method. [Figure 8] This diagram conceptually illustrates an example of a shift operation, showing the shift operation when ΔX = 0. [Figure 9] This diagram conceptually illustrates an example of a shift operation, showing the shift operation when ΔX = -1. [Figure 10] This diagram conceptually illustrates an example of subpixel interpolation. [Figure 11] This diagram conceptually illustrates an example of focus control. [Figure 12] This diagram illustrates the occlusion of a subject by an object present between the subject and the imaging device. [Figure 13] This is a flowchart showing an example of AF control. [Figure 14] This diagram conceptually illustrates an example of moving the AF area when an object causes occlusion of the subject. [Figure 15] This figure shows an example of moving the AF area again. [Figure 16] This flowchart shows an example of AF control related to the first modified example. [Figure 17] This diagram conceptually illustrates an example of the correction process related to the second modified example. [Figure 18] This diagram conceptually illustrates chromatic aberration correction. [Figure 19] This diagram conceptually illustrates an example of the process for generating composite images. [Modes for carrying out the invention]

[0022] An example of an embodiment relating to the technology of this disclosure will be described with reference to the attached drawings.

[0023] First, let's explain the terminology used in the following explanation.

[0024] In the following explanation, "IC" is an abbreviation for "Integrated Circuit". "CPU" is an abbreviation for "Central Processing Unit". "ROM" is an abbreviation for "Read Only Memory". "RAM" is an abbreviation for "Random Access Memory". "CMOS" is an abbreviation for "Complementary Metal Oxide Semiconductor".

[0025] FPGA is an abbreviation for Field Programmable Gate Array. PLD is an abbreviation for Programmable Logic Device. ASIC is an abbreviation for Application Specific Integrated Circuit. OVF is an abbreviation for Optical View Finder. EVF is an abbreviation for Electronic View Finder. JPEG is an abbreviation for Joint Photographic Experts Group. AF is an abbreviation for Auto Focus. LBE is an abbreviation for Local Binary Encoding. LBP is an abbreviation for Local Binary Pattern. AR is an abbreviation for Augmented Reality.

[0026] As one embodiment of the imaging device, the technology of this disclosure will be explained using a lens-interchangeable digital camera as an example. However, the technology of this disclosure is not limited to lens-interchangeable cameras, but can also be applied to lens-integrated digital cameras.

[0027] Figure 1 shows an example of the configuration of the imaging device 10. The imaging device 10 is a lens-interchangeable digital camera. The imaging device 10 consists of a main body 11 and an imaging lens 12 that is interchangeably attached to the main body 11. The imaging lens 12 is attached to the front side of the main body 11 via a camera-side mount 11A and a lens-side mount 12A.

[0028] The main unit 11 is provided with an operating section 13, which includes a dial, a shutter release button, and the like. The operating modes of the imaging device 10 include, for example, a still image capture mode, a video capture mode, and an image display mode. The operating section 13 is operated by the user when setting the operating mode. The operating section 13 is also operated by the user when starting still image capture or video capture.

[0029] Furthermore, the main body 11 is equipped with a viewfinder 14. Here, the viewfinder 14 is a hybrid viewfinder (registered trademark). A hybrid viewfinder refers to a viewfinder in which, for example, an optical viewfinder (hereinafter referred to as "OVF") and an electronic viewfinder (hereinafter referred to as "EVF") are selectively used. The user can observe the optical image or live view image of the subject projected by the viewfinder 14 through the viewfinder eyepiece (not shown).

[0030] Furthermore, a display 15 is provided on the back of the main unit 11. The display 15 shows images based on image signals obtained through imaging, as well as various menu screens, etc.

[0031] The main unit 11 and the imaging lens 12 are electrically connected by contact between an electrical contact 11B provided on the camera-side mount 11A and an electrical contact 12B provided on the lens-side mount 12A.

[0032] The imaging lens 12 includes an objective lens 30, a focusing lens 31, a rear-end lens 32, and an aperture 33. Each component is arranged along the optical axis A of the imaging lens 12, from the objective side, in the order of objective lens 30, aperture 33, focusing lens 31, and rear-end lens 32. The objective lens 30, focusing lens 31, and rear-end lens 32 constitute the imaging optical system. The type, number, and arrangement order of the lenses constituting the imaging optical system are not limited to the example shown in Figure 1.

[0033] Furthermore, the imaging lens 12 has a lens drive control unit 34. The lens drive control unit 34 is composed of, for example, a CPU, RAM, and ROM. The lens drive control unit 34 is electrically connected to the processor 40 in the main unit 11 via electrical contacts 12B and 11B.

[0034] The lens drive control unit 34 drives the focus lens 31 and aperture 33 based on control signals transmitted from the processor 40. The lens drive control unit 34 controls the drive of the focus lens 31 based on focus control control signals transmitted from the processor 40 in order to adjust the focus position of the imaging lens 12. The processor 40 performs phase-difference focus adjustment.

[0035] The aperture 33 has an aperture whose diameter is variable around the optical axis A. The lens drive control unit 34 controls the drive of the aperture 33 based on an aperture adjustment control signal transmitted from the processor 40 in order to adjust the amount of light incident on the light-receiving surface 20A of the image sensor 20.

[0036] Furthermore, the main unit 11 houses an image sensor 20, a processor 40, and a memory 42. The image sensor 20, memory 42, operation unit 13, viewfinder 14, and display 15 are all controlled by the processor 40.

[0037] The processor 40 is composed of, for example, a CPU, RAM, and ROM. In this case, the processor 40 performs various processes based on a program 43 stored in memory 42. The processor 40 may also be composed of an assembly of multiple IC chips.

[0038] The imaging sensor 20 is, for example, a CMOS type image sensor. The imaging sensor 20 is positioned such that the optical axis A is perpendicular to the light-receiving surface 20A and the optical axis A is located at the center of the light-receiving surface 20A. Light (subject image) that has passed through the imaging lens 12 is incident on the light-receiving surface 20A. Multiple pixels are formed on the light-receiving surface 20A, which generate an image signal by performing photoelectric conversion. The imaging sensor 20 generates and outputs an image signal by performing photoelectric conversion on the light incident on each pixel. Note that the imaging sensor 20 is an example of an "image sensor" related to the technology of this disclosure.

[0039] Furthermore, a Bayer-arranged color filter array is positioned on the light-receiving surface of the image sensor 20, with one of the R (red), G (green), or B (blue) color filters positioned opposite each pixel. In addition, some of the multiple pixels arranged on the light-receiving surface of the image sensor 20 are designated as phase-difference pixels for acquiring parallax information. These phase-difference pixels do not have color filters. Hereinafter, pixels with color filters will be referred to as normal pixels.

[0040] Figure 2 shows an example of the configuration of imaging pixel N. Figure 3 shows an example of the configuration of phase-difference pixels P1 and P2. Phase-difference pixels P1 and P2 each receive one of the light beams that are divided in the X direction with the principal ray as the center.

[0041] As shown in Figure 2, the imaging pixel N is composed of a photodiode PD as a photoelectric conversion element, a color filter CF, and a microlens ML. The color filter CF is positioned between the photodiode PD and the microlens ML.

[0042] The color filter CF is a filter that transmits light of one of the colors R, G, or B. The microlens ML focuses the light beam LF incident from the exit pupil EP of the imaging lens 12 onto the approximate center of the photodiode PD via the color filter CF.

[0043] As shown in Figure 3, the phase-difference pixels P1 and P2 are each composed of a photodiode PD, a light-shielding layer SF, and a microlens ML. Similar to the imaging pixel N, the microlens ML focuses the light beam LF incident from the exit pupil EP of the imaging lens 12 to approximately the center of the photodiode PD.

[0044] The light-shielding layer SF is formed from a metal film or the like and is placed between the photodiode PD and the microlens ML. The light-shielding layer SF blocks a portion of the light beam LF that is incident on the photodiode PD via the microlens ML.

[0045] In the phase-difference pixel P1, the light-shielding layer SF shields the negative side with respect to the X direction, with reference to the center of the photodiode PD. That is, in the phase-difference pixel P1, the light-shielding layer SF allows the light beam LF from the negative side exit pupil EP1 to be incident on the photodiode PD, while shielding the light beam LF from the positive side exit pupil EP2 with respect to the X direction.

[0046] In the phase-difference pixel P2, the light-shielding layer SF shields the positive side with respect to the X direction, with reference to the center of the photodiode PD. That is, in the phase-difference pixel P2, the light-shielding layer SF allows the light beam LF from the positive exit pupil EP2 to be incident on the photodiode PD, while shielding the light beam LF from the negative exit pupil EP1 with respect to the X direction.

[0047] Figure 4 shows an example of the pixel arrangement of the image sensor 20. In Figure 4, "R" represents an image pixel N equipped with a color filter CF of color R. "G" represents an image pixel N equipped with a color filter CF of color G. "B" represents an image pixel N equipped with a color filter CF of color B. Note that the color arrangement of the color filter CF is not limited to a Bayer arrangement and may be other color arrangements.

[0048] Rows RL containing phase-difference pixels P1 and P2 are arranged every 10 pixels in the Y direction. Each row RL contains a pair of phase-difference pixels P1 and P2 and one imaging pixel N, which are repeatedly arranged in the Y direction. Note that the arrangement pattern of phase-difference pixels P1 and P2 is not limited to the example shown in Figure 4, and may also be a pattern in which multiple phase-difference pixels are arranged on a single microlens ML, for example, as shown in Figure 5 attached to Japanese Patent Application Publication No. 2018-56703.

[0049] Figure 5 shows an example of the functional configuration of the processor 40. The processor 40 implements various functional units by executing processing according to the program 43 stored in the memory 42. As shown in Figure 5, for example, the processor 40 implements a main control unit 50, an imaging control unit 51, an image processing unit 52, a distance distribution information acquisition unit 53, and an image file generation unit 54.

[0050] The main control unit 50 comprehensively controls the operation of the imaging device 10 based on instruction signals input from the operation unit 13. The imaging control unit 51 controls the imaging sensor 20 to perform imaging operations. The imaging control unit 51 drives the imaging sensor 20 in still image imaging mode or video imaging mode.

[0051] The image processing unit 52 generates an image 56 in a predetermined file format (e.g., JPEG format) by performing various image processing operations on the RAW image RD output from the imaging sensor 20. The image 56 output from the image processing unit 52 is input to the image file generation unit 54. The image 56 is an image generated based on the signal output from the imaging pixel N.

[0052] The distance distribution information acquisition unit 53 acquires distance distribution information 58 by performing a shift operation based on the signals output from phase difference pixels P1 and P2 (see Figure 4) within the imaging area 60 of the RAW image RD output from the imaging sensor 20. The distance distribution information 58 acquired by the distance distribution information acquisition unit 53 is input to the image file generation unit 54.

[0053] The image file generation unit 54 generates an image file 59 containing the captured image 56 and distance distribution information 58, and records the generated image file 59 in the memory 42.

[0054] Figure 6 conceptually illustrates an example of distance distribution information acquisition processing by the distance distribution information acquisition unit 53. As shown in Figure 6, the distance distribution information acquisition unit 53 acquires a first signal S1 from a plurality of phase difference pixels P1 included in the imaging area 60 and a second signal S2 from a plurality of phase difference pixels P2 included in the imaging area 60, based on the RAW image RD. The first signal S1 is composed of pixel signals output from the phase difference pixels P1. The second signal S2 is composed of pixel signals output from the phase difference pixels P2. The imaging area 60 contains approximately 2000 phase difference pixels P1 and P2 in the X direction.

[0055] The distance distribution information acquisition unit 53 acquires first phase difference information D1 and second phase difference information D2 by encoding the first signal S1 and the second signal S2. The distance distribution information acquisition unit 53 performs encoding using the local binary coding (LBE) method. The LBE method is a method of converting the phase difference information for each pixel or group of pixels into binary information according to a predetermined standard. Specifically, the distance distribution information acquisition unit 53 converts the first signal S1 into first phase difference information D1 using the LBE method, and converts the second signal S2 into second phase difference information D2 using the LBE method. In the shift operation, each pixel of the first phase difference information D1 and the second phase difference information D2 is represented by a binary local binary pattern (hereinafter referred to as LBP) encoded by the LBE method.

[0056] The distance distribution information acquisition unit 53 performs a shift operation using the first phase difference information D1 and the second phase difference information D2. In the shift operation, the distance distribution information acquisition unit 53 fixes the first phase difference information D1 and calculates the sum of squared differences by performing a correlation operation between the first phase difference information D1 and the second phase difference information D2 while shifting the second phase difference information D2 one pixel at a time in the X direction.

[0057] In the shift calculation, the shift range in which the distance distribution information acquisition unit 53 shifts the second phase difference information D2 is, for example, in the range of -2 ≤ ΔX ≤ 2. ΔX represents the amount of shift in the X direction. The shift calculation aims to speed up processing by narrowing the shift range.

[0058] As will be explained in more detail later, the distance distribution information acquisition unit 53 calculates the sum of squared differences by performing binary operations. The distance distribution information acquisition unit 53 performs binary operations on the LBP contained in the corresponding pixels of the first phase difference information D1 and the second phase difference information D2. The distance distribution information acquisition unit 53 generates a difference map 62 by performing binary operations each time the second phase difference information D2 is shifted by one pixel. As a result, a difference map 62 is generated for each of ΔX = 2, 1, 0, -1, and -2. Each pixel of the difference map 62 is represented by the result of the binary operation.

[0059] As will be explained in more detail later, the distance distribution information acquisition unit 53 generates distance distribution information 58 by performing processing such as subpixel interpolation based on multiple difference maps 62.

[0060] Figure 7 conceptually illustrates an example of encoding processing using the LBE method. As shown in Figure 7, an extraction region 64 is set for the first signal S1, and multiple pixel values ​​are obtained from the set extraction region 64. The pixel values ​​are the values ​​of the pixel signals output from the phase difference pixels P1. For example, the extraction region 64 is a region containing nine pixels arranged in the X direction. The size and shape of the extraction region 64 can be changed as appropriate.

[0061] The distance distribution information acquisition unit 53 designates the central pixel of the extracted region 64 as the target pixel PI and sets the pixel value of the target pixel PI as a threshold. Next, the distance distribution information acquisition unit 53 compares the values ​​of the surrounding pixels with the threshold and binarizes them, setting them to "1" if they are greater than or equal to the threshold, and to "0" if they are less than the threshold. Next, the distance distribution information acquisition unit 53 converts the binarized values ​​of the eight surrounding pixels into 8-bit data as LBP. Finally, the distance distribution information acquisition unit 53 replaces the value of the target pixel PI with LBP.

[0062] The distance distribution information acquisition unit 53 calculates LBP while changing the extraction region 64 pixel by pixel, and generates first phase difference information D1 by replacing the value of the pixel PI of interest with the calculated LBP.

[0063] The encoding process for generating the second phase difference information D2 is the same as the encoding process for generating the first phase difference information D1, so its explanation is omitted.

[0064] Figures 8 and 9 conceptually illustrate an example of a shift operation. Figure 8 shows the shift operation when ΔX = 0. Figure 9 shows the shift operation when ΔX = -1.

[0065] The distance distribution information acquisition unit 53 reads LBPs from corresponding pixels of the first phase difference information D1 and the second phase difference information D2, and calculates the exclusive OR (XOR) of the two read LBPs. The distance distribution information acquisition unit 53 also performs a bit count on the calculated exclusive OR. Bit counting is the process of counting the number of "1"s in the exclusive OR expressed in binary and determining the number of "1"s. Hereinafter, the value obtained by bit counting will be called the bit count value. In this embodiment, the bit count value is a value within the range of 0 to 8.

[0066] The distance distribution information acquisition unit 53 calculates the bit count value of the exclusive OR for each of the ΔX values ​​2, 1, 0, -1, and -2 for all corresponding pixels of the first phase difference information D1 and the second phase difference information D2. As a result, a difference map 62 is generated for each of the ΔX values ​​2, 1, 0, -1, and -2. Each pixel of the difference map 62 is represented by a bit count value.

[0067] Figure 10 conceptually illustrates an example of subpixel interpolation processing. As shown in Figure 10, the distance distribution information acquisition unit 53 reads bit count values ​​from corresponding pixels of multiple difference maps 62 generated by the shift calculation process, and plots the read bit count values ​​against the shift amount ΔX. The distance distribution information acquisition unit 53 then obtains an interpolation curve by interpolating the bit count values ​​and determines the shift amount δ for the minimum value of the interpolation curve. The shift amount δ represents the defocus amount, i.e., the distance from the in-focus position. The relationship between the shift amount δ and the actual distance depends on the depth of field.

[0068] The distance distribution information 58 described above is generated by performing subpixel interpolation on all pixels of the difference map 62. Each pixel of the distance distribution information 58 is represented by a shift amount δ (defocus amount). The distance distribution information 58 corresponds to the captured image 56 described above and represents the distance information of objects contained within the imaging area from which the captured image 56 is acquired.

[0069] Figure 11 conceptually illustrates an example of focus control by the main control unit 50. As shown in Figure 11, the main control unit 50 acquires subject distance information 74, which represents the distance of a subject located within the AF area 70, and peripheral distance information 76, which represents the distance of objects located within the peripheral area 72, based on the distance distribution information 58 acquired by the distance distribution information acquisition unit 53. In the example shown in Figure 11, subject H is located within the AF area 70, and objects O1 and O2 are located within the peripheral area 72. The AF area 70 is an example of a "focus target area" related to the technology of this disclosure.

[0070] The AF area 70 is, for example, an area that includes a subject specified using the operation unit 13. Alternatively, the AF area 70 may be an area that includes a subject recognized by the main control unit 50 through subject recognition based on the captured image 56. When the subject H moves, the main control unit 50 moves the AF area 70 to track the subject H.

[0071] The main control unit 50 performs focus control, controlling the position of the focus lens 31 so that the subject H is in focus, based on the subject distance information 74. Hereinafter, focus control based on the subject distance information 74 will be referred to as AF control.

[0072] Furthermore, the main control unit 50 interrupts or resumes AF control during AF control based on the subject distance information 74 and the peripheral distance information 76. Specifically, the main control unit 50 detects objects in the peripheral region 72 that are located between the subject H and the imaging device 10, including the image sensor 20, based on the subject distance information 74 and the peripheral distance information 76. The main control unit 50 also determines whether the detected object is approaching the subject H. Detecting an object located between the subject H and the imaging device 10 means detecting an object between the subject H and the imaging device 10 in the direction perpendicular to the image sensor 20. Therefore, the main control unit 50 can detect an object even if the positions of the imaging device 10 and the subject H are misaligned in a direction perpendicular to the direction perpendicular to the image sensor 20 within the plane of the image sensor 20.

[0073] Figure 12 illustrates the occlusion of subject H by object O3 located between subject H and the image sensor 20. In the example shown in Figure 12, subject H is moving towards object O3. Since object O3 is located between subject H and the imaging device 10, as subject H continues to move, object O3 will obstruct subject H (i.e., occlusion will occur). In this case, if the main control unit 50 continues AF control, when object O3 obstructs subject H, the focus position will move from the position corresponding to subject H to the position corresponding to object O3, which is in front of subject H. In other words, if subject H is moving and object O3 temporarily obstructs subject H, the focus position will fluctuate. Similarly, even if subject H does not move, but object O3 moves and object O3 obstructs subject H, a fluctuation in the focus position will occur.

[0074] In this embodiment, the main control unit 50 determines whether an object O3 located between the subject H and the imaging device 10 is relatively approaching the subject H. If the object O3 approaches the subject H within a certain range, the AF control is changed. Examples of changing the AF control include interrupting the AF control to maintain the focus position before the interruption, or forcibly continuing the AF control on the subject to maintain the focus position. Alternatively, the position of the subject H may be estimated based on the past position of the subject H (i.e., the movement history of the subject H), and focus control may be performed on the estimated position.

[0075] Figure 13 is a flowchart illustrating an example of AF control by the main control unit 50. As shown in Figure 13, first, the main control unit 50 detects the subject H from the AF area 70 (step S10). Based on the subject distance information 74, the main control unit 50 starts AF control to bring the detected subject H into focus (step S11).

[0076] When the main control unit 50 starts AF control, it performs a detection process to detect an object O3 that exists between the subject H and the image sensor 20 based on subject distance information 74 and peripheral distance information 76 (step S12). If the main control unit 50 does not detect object O3 (step S12: NO), it performs the detection process again. If the main control unit 50 detects object O3 (step S12: YES), it determines whether or not object O3 has approached within a certain range relative to the subject H (step S13). If object O3 has not approached within a certain range relative to the subject (step S13: NO), the main control unit 50 makes the determination again.

[0077] If the main control unit 50 determines that object O3 has approached within a certain range relative to subject H (step S13: YES), it interrupts AF control (step S14). While AF control is interrupted, the focus position before the interruption is maintained.

[0078] The main control unit 50 determines whether or not the subject H has been detected again (step S15). If the subject H is not detected (step S15: NO), the process returns to step S14. That is, the main control unit 50 suspends AF control until the subject H is detected again. If the subject H is detected again (step S15: YES), the main control unit 50 resumes AF control (step S16).

[0079] Next, the main control unit 50 determines whether or not the termination condition is met (step S17). The termination condition is, for example, a termination operation performed by the user using the operation unit 13. If the termination condition is not met (step S17: NO), the main control unit 50 returns to step S12. If the termination condition is met (step S17: YES), the main control unit 50 terminates the AF control.

[0080] As described above, the imaging device 10 of this disclosure interrupts AF control when occlusion occurs in the subject and maintains the focus position before the interruption, thereby enabling accurate tracking of the subject. Furthermore, the AF control of this embodiment is preferably applied during live view display. Since the focus position does not change even if occlusion occurs in the subject being focused on, the visibility of the live view display is improved.

[0081] Various modifications of the above embodiment are shown below.

[0082] [First variation] In the above embodiment, AF control is interrupted when an object O3 located in front of the subject H approaches the subject H. In contrast, in this modified example, the position of the subject H is estimated based on the past position of the subject H (i.e., the movement history of the subject H) without interrupting AF control, and the AF area 70 is moved to the estimated position.

[0083] Figure 14 conceptually illustrates an example of moving the AF area 70 when occlusion occurs in the subject H due to object O3. As shown in Figure 14, when the subject H moves in a direction toward object O3 and it is estimated that object O3 will be obscured by subject H, the main control unit 50 estimates the position in which subject H will reappear after being obscured by object O3, based on the movement history of subject H. The main control unit 50 then moves the AF area 70 to the estimated position.

[0084] Furthermore, if, after moving the AF area 70, the main control unit 50 does not detect the subject H within the moved AF area 70, it moves the AF area 70 again.

[0085] Figure 15 shows an example of moving the AF area 70 again. After moving the AF area 70 as shown in Figure 14, if the subject H is not detected within the moved AF area 70, the main control unit 50 infers that the subject H is still obscured by object O3. Then, as shown in Figure 15, the main control unit 50 moves the AF area 70 to the position of object O3. As a result, object O3 becomes the target of focus.

[0086] Figure 16 is a flowchart showing an example of AF control according to the first modified example. Steps S20 to S23 shown in Figure 16 are the same processes as steps S10 to S13 shown in Figure 13. In this modified example, if the main control unit 50 determines that object O3 has approached within a certain range of the subject H (step S23: YES), it estimates the position of the subject H based on the past position of the subject H (step S24). The main control unit 50 moves the AF area 70 to the estimated position (step S25).

[0087] The main control unit 50 determines whether the subject H is detected again from the AF area 70 after it has been moved (step S26). If the subject H is not detected (step S26: NO), it moves the AF area 70 to the position of object O3 (step S27). On the other hand, if the main control unit 50 detects the subject H from the AF area 70 after it has been moved (step S26: YES), it proceeds to step S28.

[0088] In step S28, the main control unit 50 determines whether or not the termination condition is met (step S28). The termination condition is, for example, a termination operation performed by the user using the operation unit 13. If the termination condition is not met (step S28: NO), the main control unit 50 returns to step S22. If the termination condition is met (step S28: YES), the main control unit 50 terminates AF control.

[0089] [Second variation] In the above embodiment, AF control based on subject distance information 74 and peripheral distance information 76 has been described. In this modified example, the image processing unit 52 performs correction processing on at least one of the AF area 70 and the peripheral area 72 of the captured image 56.

[0090] Figure 17 conceptually illustrates an example of the correction process related to the second modified example. As shown in Figure 17, the image processing unit 52 performs a correction process that blurs only the peripheral area 72. As a result, objects O1 and O2 located within the peripheral area 72 become blurred, making the subject H, which is in focus within the AF area 70, stand out impressively.

[0091] Furthermore, the peripheral distance information 76 includes the relative distances of objects O1 and O2 within the peripheral region 72 to the AF area 70. Therefore, the image processing unit 52 may change the correction content (e.g., blur amount) according to the respective distances of objects O1 and O2 within the peripheral region 72. For example, the image processing unit 52 may make the blur amount for objects located in front of the focus position greater than the blur amount for objects located behind the focus position.

[0092] By performing correction using subject distance information 74 and peripheral distance information 76, subjects within the AF area 70 and objects within the peripheral area 72 can be distinguished quickly and accurately, thus speeding up the correction. The correction process in this modified example is not limited to blur correction, but may also be brightness correction. For example, the image processing unit 52 distinguishes between subjects within the AF area 70 and objects within the peripheral area 72 and corrects the brightness of the subjects. The image processing unit 52 may also distinguish between subjects within the AF area 70 and objects within the peripheral area 72 and perform correction to reduce the brightness of peripheral objects. Furthermore, the image processing unit 52 may perform chromatic aberration correction for objects within the peripheral area 72 using subject distance information 74 and peripheral distance information 76.

[0093] Figure 18 conceptually illustrates chromatic aberration correction. As shown in Figure 18, the image processing unit 52 detects the contours of objects O1 and O2 present in the peripheral region 72 and performs chromatic aberration correction on the detected contours. Chromatic aberration correction is a process that corrects the color of edges such as contours on a pixel-by-pixel basis. For example, chromatic aberration correction can involve correcting the color of the contour pixels or reducing the saturation at the edges. Alternatively, chromatic aberration correction may also involve correction processes such as gradient correction, which applies a gradient to the edges.

[0094] Chromatic aberration occurring in the contours of objects within the peripheral region 72 is mainly caused by axial chromatic aberration, but it can also be caused by lateral chromatic aberration. This chromatic aberration is an unevenness that occurs depending on the distance of the subject to the imaging device 10, and the color and size of the unevenness differ. Therefore, the image processing unit 52 may change the correction content of the chromatic aberration correction depending on the distance of the objects present in the peripheral region 72. That is, the image processing unit 52 may perform correction processing on objects as part of the correction processing performed on the peripheral region, and may change the correction processing on objects depending on the relative distance of the objects in the peripheral region to the focus area. Furthermore, the image processing unit 52 may change the correction content of the chromatic aberration correction depending on whether the objects present in the peripheral region 72 are in front of or behind the subject in the AF area 70 (i.e., whether it is front-focused or back-focused).

[0095] [Third variation] In this modified example, the image processing unit 52 generates a composite image. Figure 19 conceptually shows an example of the composite image generation process. When the image processing unit 52 generates a composite image 82 by combining the captured image 56 and the stereoscopic image 80, it uses distance distribution information 58 to align the captured image 56 and the stereoscopic image 80. The stereoscopic image 80 is, for example, a graphic image used in AR, and the composite image 82 is a so-called AR image. The distance distribution information 58 includes distance information corresponding to multiple pixels that make up the captured image 56. Therefore, since the distance can be determined on a pixel-by-pixel basis, the discrepancy between the captured image 56 and the stereoscopic image 80 can be reduced even when there are many subjects or when the shape of the subjects is complex.

[0096] In the above embodiment, the hardware structure of the control unit, with processor 40 as an example, can be any of the following types of processors. These types of processors include a CPU, which is a general-purpose processor that functions by executing software (programs), as well as processors whose circuit configuration can be changed after manufacturing, such as FPGAs. FPGAs include dedicated electrical circuits, which are processors with circuit configurations specifically designed to perform specific processing, such as PLDs or ASICs.

[0097] The control unit may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, multiple control units may consist of a single processor.

[0098] There are several possible examples of configuring multiple control units with a single processor. A first example is a configuration where a single processor is composed of one or more CPUs and software, as exemplified by client and server computers, and this processor functions as multiple control units. A second example is a configuration where a processor that realizes the functions of the entire system, including multiple control units, on a single IC chip is used, as exemplified by System-on-a-Chip (SOC) systems. Thus, a control unit can be configured as a hardware structure using one or more of the above-mentioned types of processors.

[0099] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices.

[0100] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0101] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference. [Explanation of Symbols]

[0102] 10 Imaging device 11 Main unit 11A Camera-side mount 11B Electrical Contact 12 imaging lenses 12A lens-side mount 12B Electrical Contact 13 Control section 14 Finder 15 displays 20 Imaging sensors 20A light receiving surface 30 objective lenses 31 Focus Lens 32 Rear lens 34 Lens drive control unit 40 processors 42 memory 43 Programs 50 Main control unit 51 Imaging control unit 52 Image Processing Unit 53 Distance distribution information acquisition unit 54 Image file generation unit 56 Captured Images 58 Distance distribution information 59 image files 60 imaging area 62 Difference Map 64 Extraction area 70 AF Area 72 Peripheral area 74 Subject distance information 76. Surrounding distance information 80 Stereoscopic Images 82 Composite Images ΔX shift amount δ shift amount A optical axis CF Color Filter D1 1st phase difference information D2 Second phase difference information EP exit pupil EP1 1st exit pupil EP2 2nd exit pupil H Subject LF luminous flux ML Microlens N image pixels O1,O2,O3 Object P1 First phase difference pixel P2 Second Phase Difference Pixel PD photodiode PI (Primary Object) RD RAW image RL line S1 1st signal S2 2nd signal SF light shielding layer

Claims

1. An image sensor having a plurality of first phase difference pixels and a plurality of second phase difference pixels, and outputting an captured image, At least a processor, An imaging device comprising, The aforementioned processor, The signal is encoded using a local binary encoding method that involves setting an extraction region for each of the signals obtained from the plurality of first phase difference pixels and the plurality of second phase difference pixels, comparing the pixel values ​​of surrounding pixels with the pixel value of the pixel of interest within the extraction region as a threshold to perform binarization, generating a local binary pattern based on the binarized values ​​of the surrounding pixels, and replacing the value of the pixel of interest with the local binary pattern, while changing the extraction region, thereby generating first phase difference information and second phase difference information. While shifting the second phase difference information relative to the first phase difference information, the local binary patterns are read from the corresponding pixels, and a bit count value is calculated by performing a bit count on the exclusive OR of the two read local binary patterns. For each pixel, the shift amount that minimizes the bit count value is determined, and distance distribution information is generated using the shift amount as the pixel value. Based on the distance distribution information, the system is configured to acquire subject distance information representing the distance of a subject located in the area to be focused, and peripheral distance information representing the distance of objects located in the area surrounding the area to be focused. Imaging device.

2. Equipped with a focusing lens, The processor is configured to perform focusing control, which controls the position of the focusing lens based on the subject distance information. The imaging apparatus according to claim 1.

3. The aforementioned processor, Based on the subject distance information and the surrounding distance information, an object existing between the subject and the imaging device is detected. The system is configured to change the focus control when the object within the field of view approaches the subject. The imaging apparatus according to claim 2.

4. The aforementioned processor, The system is configured to estimate the position of the subject based on the past position of the subject when the object obstructs the subject. The imaging device according to claim 3.

5. The aforementioned processor, Move the focus area to the estimated position of the subject, If the subject is not detected within the focus area after it has been moved, the focus area is configured to move to the position of the object. The imaging device according to claim 3.

6. The aforementioned processor, The captured image and the distance distribution information are recorded. The system is configured to acquire the subject distance information and the surrounding distance information based on the distance distribution information. The imaging apparatus according to any one of claims 1 to 5.

7. The aforementioned processor, The system is configured to generate and record an image file containing the captured image and the distance distribution information. The imaging device according to claim 6.

8. The peripheral distance information included in the distance distribution information includes the relative distance of an object in the peripheral region to the focus target region. The imaging device according to claim 6.

9. The aforementioned processor, Based on the distance distribution information, the system is configured to perform correction processing on at least one of the focus target region and the peripheral region of the captured image. The imaging apparatus according to claim 8.

10. The aforementioned processor, The correction process for the object is configured to be changed according to the relative distance. The imaging device according to claim 9.

11. The correction process applied to the object is chromatic aberration correction. The imaging apparatus according to claim 10.

12. The distance distribution information includes distance information corresponding to a plurality of pixels that constitute the captured image, The aforementioned processor, The system is configured to generate a composite image by combining a stereoscopic image with the captured image using the aforementioned distance information. The imaging device according to claim 6.

13. A method for driving an imaging device that includes an image sensor having a plurality of first phase difference pixels and a plurality of second phase difference pixels and outputting an captured image, The signal is encoded using a local binary encoding method that involves setting an extraction region for each of the signals obtained from the plurality of first phase difference pixels and the plurality of second phase difference pixels, comparing the pixel values ​​of surrounding pixels with the pixel value of the pixel of interest within the extraction region as a threshold to perform binarization, generating a local binary pattern based on the binarized values ​​of the surrounding pixels, and replacing the value of the pixel of interest with the local binary pattern, while changing the extraction region, thereby generating first phase difference information and second phase difference information. While shifting the second phase difference information relative to the first phase difference information, the local binary patterns are read from the corresponding pixels, and a bit count value is calculated by performing a bit count on the exclusive OR of the two read local binary patterns. For each pixel, the shift amount that minimizes the bit count value is determined, and distance distribution information is generated using the shift amount as the pixel value. Based on the distance distribution information, subject distance information representing the distance of a subject located in the area to be focused, and peripheral distance information representing the distance of objects located in the area surrounding the area to be focused are obtained. A method for driving an imaging device.

14. A program for operating an imaging device that includes an image sensor having a plurality of first phase difference pixels and a plurality of second phase difference pixels, and outputting an captured image, The signal is encoded using a local binary encoding method that involves setting an extraction region for each of the signals obtained from the plurality of first phase difference pixels and the plurality of second phase difference pixels, comparing the pixel values ​​of surrounding pixels with the pixel value of the pixel of interest within the extraction region as a threshold to perform binarization, generating a local binary pattern based on the binarized values ​​of the surrounding pixels, and replacing the value of the pixel of interest with the local binary pattern, while changing the extraction region, thereby generating first phase difference information and second phase difference information. While shifting the second phase difference information relative to the first phase difference information, the local binary patterns are read from the corresponding pixels, and a bit count value is calculated by performing a bit count on the exclusive OR of the two read local binary patterns. For each pixel, the shift amount that minimizes the bit count value is determined, and distance distribution information is generated using the shift amount as the pixel value. Based on the distance distribution information, subject distance information representing the distance of a subject located in the area to be focused, and peripheral distance information representing the distance of objects located in the area surrounding the area to be focused are obtained. A program that causes the imaging device to perform the processing.

15. The aforementioned processor, Based on the subject distance information and the peripheral distance information, the subject located in the focus area and objects located in the peripheral area are distinguished. A correction is performed to reduce the brightness of the object, among the subject and the object. The imaging apparatus according to claim 1.

Images