Imaging device and imaging method

Abstract

This imaging device comprises: an imaging element equipped with photodiode divisional pixels; and a control unit. The control unit performs control such that: as a reading for one frame of an image in the case of performing rolling shutter reading from the imaging element, a first reading where, with respect to all the pixels involved in generation of the image, an additional value of a first pixel and a second pixel constituting the photodiode divisional pixels is read and a second reading where a first pixel value and a second pixel value can be determined for some of the pixels involved in the generation of the image are performed in a time-sharing manner; and an exposure period for the first reading and an exposure period for the second reading are set independently of each other.

Description

[Technical field]

[0001] The present technology relates to an imaging device including an imaging element having a pixel group that outputs a phase difference signal, and an imaging method. [Background technology]

[0002] Some imaging devices have a function of acquiring focus information about a subject in order to perform autofocus control. Among such imaging devices, for example, those having pixels for detecting the focus are known. Patent document 1 discloses a technology for performing focus detection by reading out multiple pupil-divided image signals at an image sensor, and for changing the readout row of the focus detection pixel in accordance with the accumulation time or sensitivity setting of the image sensor by controlling the readout row that acquires the multiple image signals. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-81224 A Summary of the Invention [Problem to be solved by the invention]

[0004] Consider the case of rolling shutter readout from an image sensor. During the readout corresponding to one frame of an image, multiple pupil-divided image signals (pixel values) can be read out using pixels with a PD (Photodiode) division method. For this readout, it is known that while the sum value of the PD divided pixels is being read out as normal pixel readout, for the PD divided pixels on a certain line, one of the divided pixels is also read out. For this reason, during one vertical period (also written as "1V"), a certain line may be read out twice. This can cause unnatural edges to appear in images, especially images of moving subjects. Therefore, this technology devisees a readout operation for an image sensor having PD divided pixels to prevent the occurrence of unnatural images. [Means for solving the problem]

[0005] The imaging device according to the present technology includes an imaging element having photodiode divided pixels, and a sum value of a first pixel and a second pixel constituting the photodiode divided pixels for all pixels to be generated as a readout corresponding to one frame of an image when performing rolling shutter readout from the imaging element. only and a second readout capable of obtaining the first pixel value and the second pixel value for a portion of pixels to be used for image generation. Both and a control unit that controls the exposure period so that the first readout and the second readout are performed in a time-division manner and an exposure period for the first readout and an exposure period for the second readout are provided separately. A photodiode split pixel (PD split pixel) becomes a pixel used for image generation by reading out the sum of the first pixel and the second pixel. In order to obtain the values ​​of the first pixel and the second pixel, a second readout is performed in addition to the first readout.

[0006] In the imaging device of the present technology described above, it is conceivable that a process for generating an image is performed based on the added value obtained in the first readout, and a phase difference detection process is performed using the value of the first pixel and the value of the second pixel obtained in the second readout. The first readout is a readout for image generation, and the second readout is a readout for phase difference detection.

[0007] In the imaging device of the present technology described above, it is considered that the control unit executes the second readout as a readout corresponding to one frame of an image after the first readout is completed. For example, first, a first readout is performed for image generation, and then a second readout is performed for focus control in the subsequent frame.

[0008] In the imaging device of the present technology described above, it is considered that the control unit executes the first readout as a readout corresponding to one frame of an image after the second readout is completed. For example, first, the second readout is performed for focus control in the subsequent frame, and then the second readout is performed for image generation.

[0009] In the imaging device of the present technology described above, it is considered that the control unit starts exposure for the second readout before the first readout ends. Even if the first readout is in progress, exposure for the lines for which the readout has been completed is possible, and therefore exposure for the second readout is started.

[0010] In the imaging device of the present technology described above, it is considered that the control unit ends the second readout during exposure for the first readout. That is, the second readout is performed in parallel with the exposure for the first readout, and is completed during the period of exposure for the first readout.

[0011] In the imaging device of the present technology described above, the second readout may involve reading out the value of one of the first pixel and the second pixel, and reading out the added value of the first pixel and the second pixel. By reading out the sum of the first and second pixels and the value of one of the pixels, the value of the other pixel can be calculated, and the value of the first pixel and the value of the second pixel can be determined.

[0012] In the imaging device of the present technology described above, the imaging element has a light-shielding pixel having a pupil division function by being equipped with a light-shielding section that blocks one of a pair of light beams that pass through a pair of partial regions that are biased in opposite directions in a predetermined direction at the exit pupil, and a light-receiving element that receives the other light beam, and the control unit controls to perform a third readout that reads out the light-shielding pixel, and it is considered that a phase difference detection process is performed using the value of the light-shielding pixel obtained in the third readout. A light-shielded pixel is, for example, either a pixel into which only light that has passed through a left region defined by the light-shielding portion as the left half of the exit pupil is incident, or a pixel into which only light that has passed through a right region defined by the light-shielding portion as the right half of the exit pupil is incident.

[0013] In the imaging device of the present technology described above, it is conceivable that the third readout is performed prior to the second readout as a readout corresponding to one frame of an image. The readout of the light-shielded pixels as the third readout and the second readout are performed before and after in a time-division manner.

[0014] In the imaging device of the present technology described above, it is conceivable that the second readout is performed prior to the third readout as readout corresponding to one frame of an image. The second readout and the readout of the light-shielded pixels as the third readout are performed before and after in a time-division manner.

[0015] In the imaging device of the present technology described above, it is considered that the third readout is performed together with the first readout. The photodiode divided pixels are used as pixels for pixel generation, and when the photodiode divided pixels are read out, the light-shielded pixels are also read out.

[0016] In the imaging device of the present technology described above, it is conceivable that focus control is performed using the result of the phase difference detection processing based on the second readout or the result of the phase difference detection processing based on the third readout, whichever is more reliable. Focus control is performed by selectively using either the defocus amount calculated as the phase difference detection result by the light-shielded pixels or the defocus amount calculated as the phase difference detection result by the photodiode divided pixels.

[0017] In the imaging device of the present technology described above, it is considered that the thinning rate for setting the part of pixels to be read out in the second readout is variably set. In the second readout, the values ​​of the first and second pixels are obtained from the thinned out pixels and used for autofocus control, but the thinning rate is changed according to various conditions.

[0018] In the imaging device of the present technology described above, when continuous still image imaging is performed, it is possible to configure the first readout and the second readout to be performed in a time-division manner. In the case of continuous shooting of still images, the first readout and the second readout are performed in a time-division manner as processing for each frame that is continuously shot.

[0019] The imaging method according to the present technology is an imaging method for an imaging device including an imaging element having photodiode divided pixels. When performing rolling shutter readout from the imaging element, as readout corresponding to one frame of an image, the method includes: adding values ​​of first and second pixels constituting the photodiode divided pixels for all pixels to be generated as an image; only and a second readout capable of obtaining the first pixel value and the second pixel value for a portion of pixels to be used for image generation. Both is performed in a time-division manner, and an exposure period for the first readout and an exposure period for the second readout are provided separately. This realizes an imaging device that performs the first readout and the second readout separately. [Brief description of the drawings]

[0020] [Figure 1] 1 is a perspective view of an imaging device according to an embodiment of the present technology; [Diagram 2] FIG. 2 is a rear view of the imaging device according to the embodiment. [Diagram 3] 1 is a block diagram of a configuration of an imaging apparatus according to an embodiment. [Figure 4] 2 is an explanatory diagram of the internal configuration of a lens barrel of the imaging device according to the embodiment. FIG. [Diagram 5] 1 is an explanatory diagram of an imaging element having PD divided pixels according to an embodiment. [Figure 6] 1 is an explanatory diagram of a configuration example of PD divided pixels arranged in an on-axis region. FIG. [Figure 7] FIG. 11 is an explanatory diagram of a read operation in a comparative example. [Figure 8] FIG. 11 is an explanatory diagram of a read operation in a comparative example. [Figure 9] FIG. 4 is an explanatory diagram of a read operation according to the first embodiment. [Figure 10] FIG. 2 is a schematic explanatory diagram of a readout state according to an embodiment. [Figure 11] 4 is a flowchart of a read operation control according to the first embodiment. [Figure 12] 4 is a flowchart of AF control according to the first embodiment. [Figure 13] FIG. 11 is an explanatory diagram of a read operation according to the second embodiment. [Figure 14] 1 is an explanatory diagram of an image sensor having light-shielded pixels and PD divided pixels according to an embodiment. [Figure 15] 11 is an explanatory diagram of a configuration example of light-shielded pixels arranged in an on-axis region; FIG. [Figure 16] FIG. 13 is an explanatory diagram of a read operation according to the third embodiment. [Figure 17] 13 is a flowchart of a read operation control according to the third embodiment. [Figure 18] 13 is a flowchart of AF control according to a third embodiment. [Figure 19] FIG. 2 is a block diagram of a configuration related to calculation of an exposure amount according to an embodiment. [Figure 20] FIG. 13 is an explanatory diagram of a read operation according to the fourth embodiment. [Figure 21] FIG. 13 is an explanatory diagram of a read operation according to the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, embodiments will be described in the following order with reference to the accompanying drawings. <1. Configuration of imaging device> 2. Read Operation in Comparative Example 3. Read Operation of the First Embodiment <4. Read Operation of the Second Embodiment> <5. Read Operation of the Third Embodiment> 6. Read Operation of the Fourth Embodiment 7. Read Operation of Fifth Embodiment <8. Summary and variations>

[0022] <1. Configuration of imaging device> The external appearance of an imaging device 1 according to this embodiment is shown in FIG. 1 and FIG. In the following examples, the subject side is described as the front and the photographer side is described as the rear, but these directions are for convenience of explanation, and the implementation of the present technology is not limited to these directions.

[0023] 1 and 2, the imaging device 1 includes a camera housing 2 in which required components are arranged inside and outside, and a lens barrel 3 that is detachable from the camera housing 2 and attached to a front surface 2a. Note that the lens barrel 3 being detachable as a so-called interchangeable lens is just one example, and the lens barrel 3 may be a lens barrel that cannot be removed from the camera housing 2.

[0024] A rear monitor 4 is disposed on the rear portion 2b of the camera housing 2. On the rear monitor 4, a live view image, a reproduced image of a recorded image, and the like are displayed. The rear monitor 4 is, for example, a display device such as a liquid crystal display (LCD) or an organic EL (Electro-Luminescence) display. The rear monitor 4 is rotatable relative to the camera housing 2. For example, the rear monitor 4 is rotatable around an upper end portion of the rear monitor 4 as a rotation axis such that a lower end portion of the rear monitor 4 moves rearward. The right end or the left end of the rear monitor 4 may be the rotation axis. Furthermore, the rear monitor 4 may be rotatable about a plurality of axes.

[0025] An EVF (Electric Viewfinder) 5 is disposed on the top surface 2c of the camera housing 2. The EVF 5 includes an EVF monitor 5a and a frame-shaped surrounding portion 5b that protrudes rearward so as to surround the upper portion and the left and right sides of the EVF monitor 5a. The EVF monitor 5a is formed using an LCD, an organic EL display, etc. Note that an optical view finder (OVF) may be provided instead of the EVF monitor 5a.

[0026] The rear surface 2b and the top surface 2c are provided with various operators 6. The operators 6 include, for example, a playback menu start button, a decision button, a cross key, a cancel button, a zoom key, a slide key, and a shutter button 6S (release button).

[0027] The various types of operators 6 include various types of buttons, dials, composite operators that can be pressed and rotated, etc. The various types of operators 6 enable, for example, menu operation, playback operation, mode selection / switching operation, focus operation, zoom operation, and parameter selection / setting of shutter speed, F-number, etc.

[0028] The internal configuration of such an imaging device 1 is shown in Fig. 3. Also, Fig. 4 shows an example of the layout of part of the configuration in Fig. 3.

[0029] Inside and outside the camera housing 2 of the imaging device 1, an imaging element 7, a camera signal processing unit 8, a recording unit 9, a display unit 10, an output unit 11, an operation unit 12, a power supply unit 13, a camera control unit 14, a memory unit 15, etc. are provided. The lens barrel 3 comprises an optical system 16, a driver section 17, a barrel control section 18, an operation section 19, a memory section 20, and the like.

[0030] The optical system 16 is configured with various lenses such as an entrance end lens, a zoom lens, a focus lens, and a collecting lens, an aperture mechanism that controls exposure by adjusting the opening amount of the lens or iris (aperture) so that sensing is performed in a state where the signal charge is not saturated and is within the dynamic range, and a shutter unit such as a focal plane shutter. It should be noted that a portion of each component of the optical system 16 may be provided in the camera housing 2.

[0031] The imaging element 7 is, for example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal-Oxide Semiconductor) type, and performs exposure control for light from a subject incident via an optical system 16. The imaging element 7 is configured to include a processing unit that performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, and A / D (Analog / Digital) conversion processing for electrical signals photoelectrically converted by the pixels. Therefore, the imaging element 7 outputs a captured image signal as digital data to the camera signal processing unit 8 and the camera control unit 14.

[0032] The sensor surface of the imaging element 7 is configured to have a sensing element in which a plurality of pixels are arranged two-dimensionally. 5, the imaging element 7 is configured with PD split pixels 21 arranged in a matrix in the row and column directions. Each PD split pixel 21 is configured with two split pixels.

[0033] The configuration of the PD divided pixel 21 is shown typically in FIG. The PD split pixel 21 includes two split pixels, namely, a left PD 40L which is the split pixel on the left side and a right PD 40R which is the split pixel on the right side, a pixel boundary metal 41 arranged in front of them, an inner lens 32, a color filter 33, and an on-chip microlens 34. The color filter 33 is one of a color filter 33R having a spectral sensitivity of red (R), a color filter 33G having a spectral sensitivity of green (G), and a color filter 33B having a spectral sensitivity of blue (B). Note that there is also a configuration example in which the inner lens 32, etc. is not provided.

[0034] As shown in the figure, the left PD 40L receives light that has passed through a right pupil region EPR of the exit pupil EP. The right PD 40R receives light that has passed through a left pupil region EPL. This achieves a pupil division function.

[0035] Such PD divided pixels 21 are arranged as R pixels, G pixels, and B pixels depending on the difference in the color filters 33, as shown in FIG. For example, in the case of a G pixel as one PD divided pixel 21, a signal obtained as an added value of the left PD 40L and the right PD 40R becomes a signal of one G pixel. Furthermore, phase difference detection can be performed using the values ​​of the left PD 40L and the right PD 40R.

[0036] Returning to Fig. 3, the camera signal processing unit 8 is composed of, for example, a microprocessor specialized for digital signal processing, such as a DSP (Digital Signal Processor), a microcomputer, and the like.

[0037] The camera signal processing unit 8 performs various types of signal processing on the digital signal (captured image signal) sent from the imaging element 7. Specifically, correction between the R, G, and B color channels, white balance correction, aberration correction, shading correction, and other processes are performed. In addition, the camera signal processing unit 8 performs various processes such as YC generation processing to generate (separate) a luminance (Y) signal and a color (C) signal from the R, G, and B image data, processing to adjust the luminance and color, knee correction, and gamma correction. Furthermore, the camera signal processing unit 8 performs conversion to a final output format by performing resolution conversion processing and codec processing for encoding for recording or communication. The image data converted to the final output format is stored in the memory unit 15. The image data is output to the display unit 10, whereby an image is displayed on the rear monitor 4 or the EVF monitor 5a. Furthermore, the image data is output from an external output terminal, whereby the image is displayed on a device such as a monitor provided outside the imaging device 1.

[0038] The camera signal processing unit 8 includes a phase difference detection unit 8a. The phase difference detection unit 8a performs phase difference detection from output signals of the left PD 40L and the right PD 40R of the PD division pixel 21. The phase difference detection unit 8a then calculates a defocus amount based on the detected phase difference information. The calculated defocus amount may be used for an autofocus (AF) function by being used to drive a focus lens provided in the optical system 16 via the lens barrel control unit 18. The defocus amount may also be used to present information regarding the focus state of a subject to a user.

[0039] The recording unit 9 is made up of, for example, a non-volatile memory, and stores image files (content files) such as still image data and video data, attribute information of the image files, thumbnail images, and the like. Image files are stored in formats such as Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), and Graphics Interchange Format (GIF). There are various possible actual forms for the recording unit 9. For example, the recording unit 9 may be configured as a flash memory built into the imaging device 1, or may be configured as a memory card (e.g., a portable flash memory) that can be attached to and detached from the imaging device 1 and an access unit that accesses the memory card for storage and reading. Also, the recording unit 9 may be realized as a hard disk drive (HDD) built into the imaging device 1.

[0040] The display unit 10 executes a process for providing various displays to the photographer. The display unit 10 is, for example, the rear monitor 4 or the EVF monitor 5a. The display unit 10 executes a process for displaying image data that is converted to an appropriate resolution and input from the camera signal processor 8. This causes a live view image (also called a through image) that is an image captured during release standby to be displayed. Furthermore, the display unit 10 realizes the display of various operation menus, icons, messages, etc. as a GUI (Graphical User Interface) on the screen based on instructions from the camera control unit 14. Moreover, the display unit 10 is capable of displaying a reproduced image of image data read out from a recording medium in the recording unit 9.

[0041] In this example, both the EVF monitor 5a and the rear monitor 4 are provided, but the implementation of the present technology is not limited to this configuration, and only one of the EVF monitor 5a and the rear monitor 4 may be provided, or either one or both of the EVF monitor 5a and the rear monitor 4 may be configured to be detachable.

[0042] The output unit 11 performs data communication and network communication with external devices in a wired or wireless manner, for example, transmitting captured image data (still image files and video files) to an external display device, recording device, playback device, etc. The output unit 11 may also function as a network communication unit. For example, the output unit 11 may communicate via various networks such as the Internet, a home network, and a LAN (Local Area Network) and transmit and receive various data between the output unit 11 and a server or terminal on the network.

[0043] The operation unit 12 provided on the camera housing 2 includes not only the various operators 6 described above, but also a rear monitor 4 that employs a touch panel system, and outputs operation information to the camera control unit 14 in response to various operations such as tapping and swiping by the photographer. The operation unit 12 may function as a receiving unit for an external operation device such as a remote controller separate from the imaging device 1.

[0044] The power supply unit 13 generates a power supply voltage (Vcc) required for each unit, for example from a battery stored inside, and supplies it as an operating voltage. When the lens barrel 3 is attached to the imaging device 1, the power supply voltage Vcc from the power supply unit 13 is also supplied to the circuits within the lens barrel 3. The power supply unit 13 may be provided with a circuit for charging the battery and a circuit for generating a power supply voltage Vcc, using a DC voltage converted and input by an AC adapter connected to a commercial AC power supply as a power source.

[0045] The camera control unit 14 is configured by a microcomputer (arithmetic processing device) equipped with a CPU (Central Processing Unit), and performs overall control of the imaging device 1. For example, it controls the shutter speed in response to the operation of the photographer, issues instructions regarding various signal processes in the camera signal processing unit 8, performs imaging operations and recording operations in response to the user's operations, and performs playback operations of recorded image files. The camera control unit 14 switches between various shooting modes, etc. Examples of the various shooting modes include a still image shooting mode, a moving image shooting mode, and a continuous shooting mode in which still images are captured continuously.

[0046] The camera control unit 14 includes a user interface control unit (UI control unit) 14a for enabling the user to operate these functions. The UI control unit 14a performs a process for detecting operations on the operators 6 provided in the imaging device 1, a display process for the rear monitor 4, an operation detection process, and the like.

[0047] In addition, the camera control unit 14 issues instructions to a lens barrel control unit 18 in order to control various lenses included in the optical system 16 . For example, it performs processing to specify an aperture value to ensure the amount of light required for AF control, and issues instructions to operate the aperture mechanism according to the aperture value.

[0048] The camera control unit 14 is capable of acquiring information on various lenses included in the optical system 16 via the lens barrel control unit 18. The lens information includes, for example, the model number of the lens, the position and F-number of the zoom lens, or the exit pupil position. The camera control unit 14 is also capable of acquiring the aperture value of the aperture mechanism included in the optical system 16.

[0049] The memory unit 15 stores information and the like used in the processing executed by the camera control unit 14. The illustrated memory unit 15 collectively represents, for example, a Read Only Memory (ROM), a Random Access Memory (RAM), a flash memory, and the like. The memory unit 15 may be a memory area built into a microcomputer chip serving as the camera control unit 14, or may be configured by a separate memory chip.

[0050] The ROM, flash memory, etc. of the memory unit 15 store programs and the like used by the camera control unit 14. The ROM, flash memory, etc. store an OS (Operating System) for the CPU to control each unit, content files such as image files, and application programs and firmware for various operations. The camera control unit 14 executes the program to control the entire imaging device 1 and lens barrel 3.

[0051] The RAM of the memory unit 15 is used as a working area for the camera control unit 14 by temporarily storing data, programs, and the like used when the CPU of the camera control unit 14 executes various data processes.

[0052] The lens barrel control unit 18 of the lens barrel 3 is formed by, for example, a microcomputer, and outputs control signals to a driver unit 17 to actually drive various lenses of the optical system 16 based on instructions from the camera control unit 14. In addition, information communication between the camera control unit 14 and the lens barrel control unit 18 may be possible only when the lens barrel 3 is attached to the camera housing 2, or it may be possible via wireless communication when the lens barrel 3 is not attached to the camera housing 2.

[0053] The lens barrel control unit 18 transmits information on the exit pupil position and the pupil distance of the exit pupil to the camera control unit 14 based on the types and drive positions of the various lenses included in the optical system 16. Specifically, the lens barrel control unit 18 obtains information on the pupil distance from information stored in a ROM serving as the memory unit 20, and transmits the information to the camera control unit 14.

[0054] The driver unit 17 includes, for example, a motor driver for a zoom lens drive motor, a motor driver for a focus lens drive motor, an aperture mechanism driver for a motor that drives an aperture mechanism, and the like. Each driver supplies a drive current to the corresponding drive motor in response to an instruction from the lens barrel control unit 18 .

[0055] Operation unit 19 of lens barrel 3 indicates an operator provided on the side of lens barrel 3. Operation information by operation unit 19 is supplied to barrel control unit 18, and notified to camera control unit 14 via barrel control unit 18. In response to the operation of the operation unit 19, the barrel control unit 18 controls the operation of the optical system 16, and the camera control unit 14 performs various settings and controls the operations.

[0056] The operation unit 19 may function as a receiving unit for an external operation device such as a remote controller separate from the lens barrel 3 .

[0057] The memory unit 20 is configured with a ROM, a flash memory, etc., and stores programs, data, etc. used by the lens barrel control unit 18. The memory unit 20 stores an OS (Operating System) for the CPU to control each unit, application programs for various operations, firmware, etc. The information stored in the memory unit 20 also includes information such as the pupil distance of the exit pupil of the optical system 16 .

[0058] 2. Read Operation in Comparative Example As a comparative example, a case will now be described in which unnatural distortion (zigzag lines) occurs in the vertical direction of a subject during rolling shutter readout from an image sensor 7 having PD divided pixels 21.

[0059] In the case of an image sensor 7 in which PD split pixels 21 each having a left PD 40L and a right PD 40R are arranged as shown in Fig. 5, not only can pixel signals for image generation be read out from each PD split pixel 21, but also the pixel values ​​of the left and right PDs can be calculated separately to obtain a phase difference signal. The defocus amount can be calculated based on this phase difference signal, and AF (autofocus) control can be performed. For this reason, it is conceivable to perform two readouts for one horizontal line, such as performing an L readout and an (L+R) readout for one PD divided pixel 21. Note that "(L+R) readout" refers to readout of the added charges from the left PD 40L and right PD 40R. "L readout" refers to readout of charge from the left PD 40L, and "R readout" refers to readout of charge from the right PD 40R. Furthermore, the "L value" is the pixel signal value obtained by direct readout or calculation from the left PD 40L, and the "R value" is the pixel signal value obtained by direct readout or calculation from the right PD 40R. Furthermore, the (L+R) value is the added value of the pixel signal values ​​of the left PD 40L and right PD 40R.

[0060] The (L+R) value, that is, the sum of the charges of the left PD 40L and the right PD 40R, has the meaning as the pixel value of that PD divided pixel 21, and therefore becomes a signal used for image generation. Also, L readout is performed to obtain the signal value (L value) of the left PD 40L, and the R value, which is the signal value of the right PD 40R, is obtained by (L+R)-L. The phase difference of the pixel components divided into pupils can be obtained from this "L value" and "R value", and AF control can be performed based on this.

[0061] Here, the (L+R) readout is performed on all PD divided pixels 21 to be used for image generation, in order to obtain pixel values. Typically, for example, the (L+R) readout is performed on the PD divided pixels 21 of all horizontal lines as effective pixels. Note that the PD divided pixels 21 to be used for image generation may be some pixels after thinning out, but in any case, the (L+R) readout is performed on all PD divided pixels 21 required for image generation. On the other hand, when the purpose is AF control, L readout does not necessarily have to be performed for all pixels (that is, all pixels to be used for image generation). If L readout is performed for all pixels, the readout process will be performed twice for all horizontal lines that include such pixels, which will lengthen the 1V readout time. This can be disadvantageous in terms of the number of frames that can be captured in continuous shooting.

[0062] Therefore, it is conceivable that the L readout is performed on some of the thinned out pixels. For example, for some horizontal lines, the L readout and (L+R) readout are performed on the PD divided pixels 21 of the horizontal lines, and only the (L+R) readout is performed on the PD divided pixels 21 of the other horizontal lines.

[0063] FIG. 7 illustrates a schematic diagram of such a read operation. 7, in the vertical period (V1, V2,...) defined by the vertical synchronization signal XVS, a straight line SL10 is shown, with the vertical axis representing the horizontal line number, showing the read timing of each horizontal line. The first horizontal line number is "HL1", and the last horizontal line number is "HLmax".

[0064] For example, in a 1V period (vertical period), readout is performed sequentially from the first horizontal line (L1) to the last horizontal line (Lmax). Here, ● is marked on the straight line SL10 at intervals of the period of multiple horizontal lines. This indicates that readout is performed twice for the PD divided pixels 21 of the horizontal lines marked with ●, and normal readout is performed for the PD divided pixels 21 of the horizontal lines corresponding to the solid line portions. In this case, normal reading means performing an (L+R) read once, and double reading means performing an L read and an (L+R) read. After the pixel readout is performed in this manner, AF detection is performed based on the L value and R value of the PD divided pixel 21 that has been readout twice, and the lens is driven according to the result. As a result, for example, the AF control is reflected in the next vertical period.

[0065] FIG. 8 shows the exposure operation and the readout operation (shaded area) for some horizontal lines (horizontal lines HL1, HL2, HL3, . . . HLn, HLn+1, HLn+2, . . . HLm, HLm+1, HLm+2) on the straight line SL10. Each horizontal line is exposed for a predetermined period of time, after which the charges are read out. Here, assume that the horizontal line HLn+1 is the line marked with a black circle in FIG. In FIG. 8, for each horizontal line other than horizontal line HLn+1, only (L+R) readout is performed as normal readout. On the horizontal line HLn+1, two readouts are performed, an L readout and an (L+R) readout.

[0066] In this way, by reading out only a partial horizontal line twice, phase difference detection can be performed without making the 1V period too long. However, when horizontal lines are periodically read twice in the vertical direction, the edges of moving objects appear unnatural in the image. In the example of Fig. 8, the part of horizontal line HLn+1 that is read twice is misaligned with the parts before and after it (horizontal lines HL1 to HLn, and the parts after HLn+2). When the imager is read from top to bottom, this misalignment is repeated multiple times, and it appears as a zigzag line. Therefore, in the embodiment, a readout operation is realized that does not cause such periodic edge shifts in an image.

[0067] 3. Read Operation of the First Embodiment The readout operation according to the first embodiment is shown in FIG. 9. In FIG. 9, the horizontal direction is the time axis, and vertical periods V1, V2, and V3 defined by the vertical synchronization signal XVS are shown. These vertical periods V1, V2, V3, etc. are assumed to be the case where, for example, a release operation is performed in a continuous shooting mode, and continuous shooting is performed. In each vertical period, one image constituting the continuous shooting images is captured.

[0068] The vertical axis shows the horizontal line number ("HL1" ​​to "HLmax"), with solid lines RL1, RL2 and dashed lines EL1, EL2. The solid lines RL1, RL2 show the readout timing for each horizontal line, and the dashed lines EL1, EL2 show the exposure start timing for each horizontal line. When looking at one horizontal line, the interval from dashed line EL1 to solid line RL1, and the interval from dashed line EL2 to solid line RL2 are the exposure periods.

[0069] The following description focuses on the vertical period V2. At the timing indicated by the dashed line EL1, exposure starts sequentially from the first horizontal line (HL1) to the last horizontal line (HLmax). After that, from time TM0, as indicated by a solid line RL1 parallel to the dashed line EL1, readout is performed sequentially from the first horizontal line (HL1) to the last horizontal line (HLmax).

[0070] The fact that the dashed line EL1 and the solid line RL1 are parallel means that exposure is performed for a time length shown as an exposure period TEnor on each horizontal line from the first to the last, and then readout is performed. The period during which the read operation of the solid line RL1 is performed is the read period TRnor. This readout period TRnor is a period during which the first readout is performed, that is, (L+R) readout is performed once for each horizontal line. 10A shows a schematic diagram of the first readout operation, in which L value+R value charge is read out from each horizontal line.

[0071] At a certain time point TM1 during the readout period TRnor in Fig. 9, exposure begins as shown by the dashed line EL2. This is exposure for the subsequent second readout, and targets the PD divided pixels 21 of some horizontal lines. Horizontal lines are periodically selected at a predetermined thinning rate, for example, one in every 10 horizontal lines, and the second readout is performed. For example, if horizontal lines HL1, HL11, HL21, . . . are selected, exposure of horizontal line HL1 begins at time point TM1, and exposure thereafter begins sequentially at the timings indicated by dashed lines EL2. The dashed line EL2 has a steeper slope than the dashed line EL1 because the number of targeted horizontal lines is smaller.

[0072] The thinning rate for selecting the horizontal lines HL to be subjected to the second readout may be changed, for example, according to brightness.

[0073] Here, the readout period TRnor has not yet ended at the time TM1 when exposure starts, as indicated by the dashed line EL2, but the horizontal line to be exposed is set to be a horizontal line for which the first readout has already been completed. For example, the time TM1 when exposure of the dashed line EL2 starts is set so that exposure of the last horizontal line (HLmax) starts immediately after the first readout of the last horizontal line (HLmax) is completed.

[0074] At the timing indicated by the dashed line EL2, exposure is started sequentially for each of the thinned out horizontal lines. After that, from time TM2, as shown by a solid line RL2 parallel to the dashed line EL2, each exposed horizontal line is sequentially read out.

[0075] The fact that the dashed line EL2 and the solid line RL2 are parallel means that, for each horizontal line, exposure is performed for a time length shown as the exposure period TEpd, and then readout is performed. The period during which the read operation of the solid line RL2 is performed is the read period TRpd. This readout period TRpd is a period in which the second readout is performed. That is, two readouts, an L readout and an (L+R) readout, are performed for each horizontal line. 10B shows a schematic diagram of the second read operation, in which L read and (L+R) read are performed from a portion of horizontal lines selected after thinning. For example, after the L value is read from the left PD 40L, the image signal of the right PD 40R is read without resetting the charge. In other words, after the L value is read, the L value and the R value are added in the FD (floating diffusion) section in the image sensor, and the (L+R) value is read out.

[0076] After the second readout indicated by the solid line RL2 in Fig. 9 starts, exposure for the first readout of the next vertical period V3 indicated by the dashed line EL1 starts because exposure may start for the horizontal lines (and the thinned horizontal lines) for which the second readout has been performed.

[0077] 9, in one vertical period corresponding to one frame, a first readout in which (L+R) readout is performed from all PD divided pixels 21 to be used for image generation, and a second readout in which L readout and (L+R) readout are performed for some pixels after thinning out are performed in a time-division manner. An exposure period for the first readout (dashed line EL1) and an exposure period for the second readout (dashed line EL2) are provided separately. In this case, in the first readout indicated by the solid line RL1, the problem of some horizontal lines being read out twice is eliminated, which eliminates the unnatural edges that appear in the image as described in the comparative examples of Figures 7 and 8. Furthermore, it becomes possible to perform phase difference detection based on the second readout and perform AF control.

[0078] The readout operation as shown in Fig. 9 is executed under the control of the camera control unit 14. A flowchart of the readout operation under the control of the camera control unit 14 is shown in Fig. 11. Also, the AF processing accompanying the readout operation is shown in Fig. 12. Note that these are examples assuming imaging operations in continuous shooting mode.

[0079] 11, the camera control unit 14 monitors whether the shutter button 6S is fully pressed. If the shutter button 6S is not fully pressed, other processing (not shown) is performed.

[0080] In the continuous shooting mode, when the shutter button 6S is fully pressed, the camera control unit 14 advances from step S101 to step S102 and starts the first readout. In the image sensor 7, sequential readout starts from the top horizontal line HL1. The (L+R) values ​​read out from each PD divided pixel 21 in this first readout are processed as pixel values ​​for generating an image in the camera signal processing unit 8, and data for one still image in continuous shooting is generated.

[0081] In practice, the pulse timing related to the readout operation in the imaging element 7 is defined according to parameters set by the camera control unit 14, and the first readout is started at a predetermined point in time. In this case, the first readout starts from time point TM0 in FIG. 9, for example, with the vertical synchronization signal that defines the vertical period as the starting point. Even before the shutter button 6S is fully pressed, exposure and readout are performed in each vertical period in order to display a live view image, so exposure continues up to time TM0.

[0082] Next, in step S103, the camera control unit 14 starts exposure for the second readout. In the imaging element 7, sequential exposure is started only for the horizontal lines targeted for the second readout. In practice, pulse timing for executing an exposure operation for each horizontal line in the image sensor 7 is defined according to parameters set by the camera control unit 14, and exposure for the second readout starts from time point TM1 in FIG.

[0083] In step S104, the camera control unit 14 starts the second readout. In the imaging element 7, the second readout is sequentially performed only on the horizontal lines targeted for the second readout. In this case too, in practice, pulse timing for executing a readout operation for each horizontal line in the image sensor 7 is defined according to parameters set by the camera control unit 14, and the second readout is executed as the operation after time TM2 in FIG. 9.

[0084] In step S105, the camera control unit 14 starts exposure for the first readout of the next vertical period. In the image sensor 7, exposure is started sequentially for all pixels (horizontal lines) that are to be used for image generation. In practice, the pulse timing for executing the exposure operation for each horizontal line in the image sensor 7 is specified according to parameters set by the camera control unit 14. In this case, it is possible to start exposure sequentially from the first horizontal line HL1 immediately after the second readout is started. For example, if the first horizontal line HL1 is the target of the second readout, exposure from the horizontal line HL1 becomes possible at the time when the second readout of the PD divided pixels 21 of the horizontal line HL1 is completed. Also, if the first horizontal line HL1 is not the target of the second readout, exposure for the next first readout from the horizontal line HL1 becomes possible at the same time as the start of the second readout. The pulse timing is determined taking these circumstances into consideration.

[0085] Thereafter, the camera control unit 14 returns to step S101, and continues the same process if the shutter button 6S is still fully pressed. When the shutter button 6S is released from being fully pressed, the camera control unit 14 exits the process of Fig. 11. Note that even if the shutter button 6S is fully pressed, the process will exit if the maximum number of images captured during continuous shooting has been exceeded.

[0086] During the period in which continuous shooting is being performed in the process of FIG. 11, AF control is performed as shown in FIG. 12 in response to the second readout in step S104.

[0087] While each of the PD divided pixels 21 is being read out twice as the second readout, steps S201, S202, and S203 are performed for each of the PD divided pixels 21 in the camera signal processing unit 8. That is, for one PD divided pixel 21, the first L value as the second readout is stored in step S201, the second (L+R) value as the second readout is stored in step S202, and the R value is calculated and stored in step S203. The R value is calculated by the formula R value = (L+R) value - L value. This process is repeated until the second readout is completed in step S204, that is, until the L value and the R value are obtained for all of the PD divided pixels 21 that are the target of the second readout.

[0088] In this example, in the second readout, the L value is read out for each PD divided pixel 21 the first time, and the (L+R) value is read out the second time, but the L value may be read out the first time, and the R value may be read out the second time (or read out in the reverse order). That is, the values ​​may be read out for each divided PD. In any case, it is sufficient to obtain the L value and the R value for each PD divided pixel 21 that is the target of the second readout.

[0089] Once the L value and R value are stored for each PD divided pixel 21, the process proceeds to step S205, where the camera signal processing unit 8 performs phase difference detection using the function of the phase difference detection unit 8a. Then, the defocus amount is transmitted to the camera control unit 14. In step S206, the camera control unit 14 instructs the lens barrel control unit 18 in response to the defocus amount to drive the focus lens provided in the optical system 16. Then, in step S207, the lens drive is stopped. This results in the AF operation being performed.

[0090] <4. Read Operation of the Second Embodiment> The readout operation according to the second embodiment is shown in Fig. 13. Note that Fig. 13 and Figs. 16, 20, and 21, which will be described later, assume that a release operation is performed in continuous shooting mode, and continuous shooting is performed, and show vertical periods V1, V2, and V3 defined by the vertical synchronization signal XVS in the same format as Fig. 9. Moreover, hereinafter, the meanings of the first readout and the second readout are the same as those in the first embodiment.

[0091] The following description focuses on the vertical period V2. At a time point TM10 during the first readout indicated by the solid line RL1 in the vertical period V1, exposure for the second readout indicated by the dashed line EL2 starts. At the time point TM10 when exposure starts, the horizontal line to be exposed is a horizontal line for which the first readout has already been completed.

[0092] At the timing indicated by dashed line EL2, exposure begins sequentially for each horizontal line after thinning out, and then, from time TM11, readout is performed sequentially for each exposed horizontal line, as indicated by solid line RL2 parallel to dashed line EL2. The interval between the dashed line EL2 and the solid line RL2 is the exposure period TEpd for the second readout. The period during which the read operation of the solid line RL2 is performed is the read period TRpd of the second read.

[0093] At the timing indicated by the dashed line EL1, exposure starts sequentially from the first horizontal line (HL1) to the last horizontal line (HLmax). After that, from time TM12, as shown by a solid line RL1 parallel to the dashed line EL1, readout is performed sequentially from the first horizontal line (HL1) to the last horizontal line (HLmax). The interval between the dashed line EL1 and the solid line RL1 is the exposure period TEnor for the first readout. The period during which the read operation of the solid line RL1 is performed is the read period TRnor of the first read. Incidentally, in the middle of the readout period TRnor, exposure for the second readout of the next vertical period V3 starts.

[0094] As described above, in the example of Fig. 13, in one vertical period corresponding to one frame, the second readout is performed first, followed by the first readout. This also prevents some horizontal lines from being read out twice in the first readout shown by the solid line RL1, eliminating the problem of unnatural edges appearing in the image as described in the comparative examples of Fig. 7 and Fig. 8. Then, phase difference detection is performed based on the second readout, making it possible to perform AF control.

[0095] <5. Read Operation of the Third Embodiment> As the third embodiment, an example in which the imaging element 7 is of a hybrid type having the PD divided pixels 21 and the light-shielding pixels 23 will be described.

[0096] FIG. 14 shows an example of a pixel arrangement of the image sensor 7. A pixel row having light-shielded pixels 23 is formed as the first pixel row 22A. The first pixel rows 22A are arranged discretely in the vertical direction, and a plurality of second pixel rows 22B are arranged between the first pixel rows 22A. The first pixel rows 22A may be arranged regularly or irregularly, however, a regular arrangement can reduce design costs and manufacturing costs involved in manufacturing the image sensor 7.

[0097] Each PD split pixel 21 included in the second pixel row 22B is covered with a Bayer array color filter, and depending on the type of color filter, it has either red (R) spectral sensitivity, green (G) spectral sensitivity, or blue (B) spectral sensitivity.

[0098] The configuration of the light-shielded pixel 23 will be described with reference to the schematic diagram of FIG. The light-shielding pixel 23 includes a PD 30, a light-shielding section 31 arranged in front of the PD 30 (on the subject side), an inner lens 32 arranged in front of the light-shielding section 31, a color filter (cyan) 33 arranged in front of the inner lens 32, and an on-chip microlens 34 arranged in front of the color filter 33. It should be noted that the light-shielded pixel 23 does not necessarily need to be provided with the inner lens 32 or the color filter 33 .

[0099] The PD 30 is a light receiving element on which a portion of the light that has passed through the exit pupil EP is incident, but a light shielding portion 31 disposed in front of the PD 30 allows light to be received only in a portion of the light receiving region of the PD 30. That is, the light shielding portion 31 is formed so as to cover the left half area of ​​the PD 30. The light shielding portion 31 is formed with a right opening 35R.

[0100] The inner lens 32 and the on-chip microlens 34 are optical components provided for efficiently collecting, onto the PD 30, light that has passed through the exit pupil EP and is incident on one pixel. The color filter 33 is, for example, a filter having a spectral sensitivity of cyan (Cy).

[0101] 15, the PD 30 is configured to receive only light passing through a left region (left pupil region) which is the left half of the exit pupil EP. That is, light passing through a right region (right pupil region) which is the right half of the exit pupil EP is blocked by the light blocking portion 31 and does not reach the PD 30. This realizes a pupil division function.

[0102] 15, the light-shielded pixel 23 configured to receive light passing through the left pupil region receives light in a region biased to the right side on the light-receiving surface, and is therefore referred to as a light-shielded pixel 23R. That is, the light-shielded pixel 23R has a right opening 35R formed therein. 15, the light-shielded pixel 23L is configured to receive light passing through the right pupil region, and receives light in a region biased to the left side on the light-receiving surface. As shown in FIG. 14, the light-shielded portion 31 of the light-shielded pixel 23L has a left opening 35L.

[0103] As shown in FIG. 14, the distance between the light-shielded pixels 23R and 23L is, for example, the distance of two pixels, and they are arranged alternately. The signal output from the light-shielding pixel 23R and the signal output from the light-shielding pixel 23L are treated as a pair of phase difference signals by the camera signal processing unit 8 (or the camera control unit 14). That is, the phase difference detection unit 8a of the camera signal processing unit 8 can calculate the defocus amount using the phase difference between the signal output from the light-shielding pixel 23R and the signal output from the light-shielding pixel 23L.

[0104] FIG. 16 shows an example of a readout operation assuming such an image sensor 7. The following description focuses on the vertical period V2. At the timing indicated by the dashed line EL1, exposure starts sequentially from the first horizontal line (HL1) to the last horizontal line (HLmax). After that, from time TM20, the first pixel row 22A is sequentially read out as indicated by the solid line RL3 (read out period TRm). This is to read out the values ​​of the light-shielded pixels 23L and 23R, and is called the third read out to distinguish it from the first and second read outs.

[0105] From time TM21, as indicated by a solid line RL1 parallel to the dashed line EL1, readout is performed sequentially from the first horizontal line (HL1) to the last horizontal line (HLmax). This is the first readout (see FIG. 10A). Note that the first readout is performed on all horizontal lines, and so readout is performed on both the first pixel row 22A and the second pixel row 22B, which include the light-shielded pixels 23. The pixel values ​​obtained by this first readout are used to generate an image, but the light-shielded pixels 23 are treated as invalid pixels, and a pixel value of 1 is formed for the light-shielded pixels 23 by, for example, an interpolation process.

[0106] At a certain time point TM22 in the middle of the readout period TRnor, exposure for the second readout starts as shown by the dashed line EL2. After that, from time TM23, as indicated by a solid line RL2 parallel to the dashed line EL2, readout is performed sequentially for each exposed horizontal line (second readout: see FIG. 10B).

[0107] After the second readout indicated by the solid line RL2 starts, exposure for the first readout of the next vertical period V3 indicated by the dashed line EL1 starts. In the case of Fig. 16, the exposure of the dashed line EL1 also serves as exposure of the light-shielded pixels 23 for the third readout.

[0108] 16, in one vertical period corresponding to one frame, a first readout in which (L+R) readout is performed from all PD divided pixels 21 to be used for image generation, a second readout in which L readout and (L+R) readout are performed on some of the pixels after thinning out, and a third readout targeting the light-shielded pixels 23 are performed in a time-division manner. In addition, an exposure period (dashed line EL1) for the first readout and the third readout, and an exposure period (dashed line EL2) for the second readout are provided separately. In this case, the first readout indicated by the solid line RL1 does not cause some horizontal lines to be read out twice, eliminating the problem of unnatural edges appearing in the image. In addition, phase difference detection is performed based on the second and third readouts, making it possible to perform AF control.

[0109] The readout operation as shown in Fig. 16 is executed under the control of the camera control unit 14. A flowchart of the readout operation under the control of the camera control unit 14 is shown in Fig. 17. Also, the AF processing accompanying the readout operation is shown in Fig. 18. The following description will be given assuming an imaging operation in a continuous shooting mode.

[0110] 17, the camera control unit 14 monitors whether the shutter button 6S is fully pressed. If the shutter button 6S is not fully pressed, other processing (not shown) is performed.

[0111] In the continuous shooting mode, when the shutter button 6S is fully pressed, the camera control unit 14 advances from step S151 to step S152 to start the third readout. In the image sensor 7, sequential readout is started for the first pixel row 22A. In this case, for example, the third readout is started from time point TM20 in FIG. 16, which is based on the vertical synchronization signal that defines the vertical period. The values ​​read out from the light-shielded pixels 23 in this third readout are used for phase difference detection in the camera signal processing unit 8.

[0112] In step S153, the camera control unit 14 starts the first readout from time point TM21 in Fig. 16. In the imaging element 7, sequential readout starts from the top horizontal line HL1. The (L+R) values ​​read out from each PD divided pixel 21 in this first readout are processed as pixel values ​​for generating an image in the camera signal processing unit 8, and data for one still image in continuous shooting is generated.

[0113] Next, in step S154, the camera control unit 14 starts exposure for the second readout from time point TM22. In the imaging element 7, sequential exposure is started only for the horizontal lines targeted for the second readout.

[0114] In step S155, the camera control unit 14 starts the second readout from time TM23. In the imaging element 7, the second readout is sequentially performed only on the horizontal lines targeted for the second readout. In the second readout, the L value is read out the first time and the (L+R) value is read out the second time for each PD divided pixel 21. Note that in this case as well, in the second readout, the L value may be read out the first time and the R value may be read out the second time (or the order may be reversed).

[0115] In step S156, the camera control unit 14 starts exposure for the first readout and the third readout of the next vertical period. In the image sensor 7, exposure is started sequentially for all pixels (horizontal lines) to be used for image generation.

[0116] Thereafter, the camera control unit 14 returns to step S151, and continues the same process if the shutter button 6S is still fully pressed. When the shutter button 6S is released from being fully pressed, the camera control unit 14 exits the process of Fig. 17. Note that even if the shutter button 6S is fully pressed, the process will also exit when the maximum number of images captured during continuous shooting has been exceeded.

[0117] During the period during which continuous shooting is being performed in the process of FIG. 17, AF control is performed as shown in FIG. 18 in response to the third readout in step S152 and the second readout in step S155.

[0118] While the third readout is being performed, the camera signal processing unit 8 performs the process of step S251 for each light-shielded pixel 23. That is, it stores the L value and R value of the light-shielded pixel 23. These are values ​​read out from the pair of light-shielded pixels 23L, 23R.

[0119] Once the third readout is completed and the L values ​​and R values ​​of all the targeted light-shielded pixels 23L, 23R have been stored, the camera signal processing unit 8 then performs the processes of steps S253, S254, and S255 for each PD divided pixel 21 during the second readout.

[0120] That is, for one PD divided pixel 21, the first L value as the second readout is stored in step S253, the second (L+R) value as the second readout is stored in step S254, and the R value is calculated and stored in step S255. The R value is calculated by the formula R value = (L+R) value - L value. This process is repeated until the second readout is completed in step S256, that is, until the L value and the R value are obtained for all of the PD divided pixels 21 that are the target of the second readout.

[0121] Once the L value and R value are stored for each PD divided pixel 21, the process proceeds to step S258, where phase difference detection and defocus amount calculation are performed by the function of the phase difference detection unit 8a in the camera signal processing unit 8. In this case, phase difference detection can be performed from the output of the light-shielded pixel 23, or from the output of the PD divided pixel 21. The defocus amount based on each of these phase difference detections is transmitted from the camera signal processing unit 8 to the camera control unit 14.

[0122] In step S258, the camera control unit 14 determines which one currently has higher reliability, and accordingly selects a defocus amount based on either the light-shielded pixels 23 or the PD divided pixels 21. Then, in step S259, the camera control unit 14 instructs the lens barrel control unit 18 to drive the focus lens provided in the optical system 16 according to the defocus amount. Then, in step S260, the lens drive is stopped. This causes the AF operation to be performed.

[0123] Here, the light-shielded pixel 23 and the PD divided pixel 21 have the following differences. The light-shielded pixels 23 are pixels used only for phase difference detection. On the other hand, the PD divided pixels 21 can be used as normal pixels (pixels for image generation) while being used for phase difference detection. For this reason, the light-shielded pixels 23 are arranged discretely, and the number of pixels cannot be made too large. As a result, the PD divided pixel 21 is superior to the light-shielded pixel 23 in terms of low illumination performance and large F-number performance. On the other hand, the light-shielded pixel 23 is superior to the PD segment pixel 21 in that it has freedom in pupil correction design and has good off-axis performance. Considering this, it is considered that the advantages of the PD divided pixel 21 can be utilized in a low-illumination environment, and the advantages of the light-shielding pixel 23 can be utilized in a high-illumination environment. Therefore, in step S258, it is possible to determine the reliability based on the illumination environment. Specifically, one of the phase difference signals is selected according to the amount of exposure.

[0124] The calculation of the exposure amount will be described with reference to FIG. A normal pixel output for generating a normal image signal and a phase difference pixel output as a phase difference signal for AF control are output from the image sensor 7. The normal pixel output here refers to the (L+R) value from the PD divided pixel 21 by the first readout.

[0125] The normal pixel output and the phase difference pixel output are each input to an output level detection circuit provided in the camera signal processing unit 8. The output level detection circuit calculates an output average value in an exposure calculation target area on the pixel based on the input normal pixel output and phase difference pixel output, and each output average value is output from the camera signal processing unit 8 and input to the camera control unit 14.

[0126] The camera control unit 14 performs exposure calculation according to the detection result output from the camera signal processing unit 8, and determines the shutter speed (or parameters capable of adjusting the exposure, such as the F-number and gain). The camera control unit 14 performs processing to set the determined shutter speed in the image sensor 7.

[0127] The exposure amount calculation performed by the camera control unit 14 may be performed based on only the normal pixel output, or may be performed based on only the phase difference pixel output, or may be performed based on both the normal pixel output and the phase difference pixel output. In any case, the illumination environment can be determined by calculating the amount of exposure, and based on this, it is possible to select whether to use the phase difference detection result by the light-shielded pixel 23 or the phase difference detection result by the PD divided pixel 21 for AF control.

[0128] 6. Read Operation of the Fourth Embodiment A read operation according to the fourth embodiment is shown in FIG. At a time point TM30 during the first readout indicated by the solid line RL1 in the vertical period V1, exposure for the second readout indicated by the dashed line EL2 starts.

[0129] At the timing indicated by the solid line RL3 from time point TM31, a third readout is started for the light-shielded pixels 23. The readout of the light-shielded pixels 23 for each of the first pixel rows 22A is performed sequentially. From time TM32, a second readout is performed as indicated by the solid line RL2. After the second readout starts, exposure starts sequentially from the first horizontal line (HL1) to the last horizontal line (HLmax) at the timing indicated by the dashed line EL1.

[0130] After that, from time TM33, as shown by a solid line RL1 parallel to the dashed line EL1, readout is performed sequentially from the first horizontal line (HL1) to the last horizontal line (HLmax). Incidentally, in the middle of the readout period TRnor, exposure for the second and third readouts of the next vertical period V3 starts.

[0131] As described above, in the example of Fig. 20, the third readout, the second readout, and the first readout are performed in this order in one vertical period corresponding to one frame. This also eliminates the unnatural edges that occur in the image by the first readout, as indicated by the solid line RL1. In addition, it is possible to perform phase difference detection based on the third readout or the second readout, and perform AF control.

[0132] 7. Read Operation of Fifth Embodiment A read operation according to the fifth embodiment is shown in FIG. This is an example in which the first readout and the third readout are performed simultaneously. At a time point TM40 in the middle of the first readout (and third readout) indicated by the solid line RL1 (RL3) in the vertical period V1, exposure for the second readout indicated by the dashed line EL2 starts.

[0133] From time TM41, the second readout is performed as indicated by the solid line RL2. After the second readout starts, exposure starts sequentially from the first horizontal line (HL1) to the last horizontal line (HLmax) at the timing indicated by the dashed line EL1.

[0134] After that, from time TM42, as shown by the solid line RL1 (RL3) parallel to the dashed line EL1, readout is performed sequentially from the first horizontal line (HL1) to the last horizontal line (HLmax). In this case, the (L+R) value is read out from the PD divided pixel 21, and the light-shielded pixel 23 is also read out in the process.

[0135] As described above, in the example of Fig. 21, the second readout, the first readout, and the third readout are performed in this order in one vertical period corresponding to one frame. This also eliminates the occurrence of unnatural edges in the image due to the first readout, as indicated by the solid line RL1. The third readout is performed during the first readout process, and since two readouts do not occur, there is no problem even if the first readout and the third readout are performed simultaneously. In addition, it is possible to perform phase difference detection based on the third readout or the second readout, and perform AF control.

[0136] <8. Summary and variations> The above embodiment provides the following advantages. The imaging device 1 of the embodiment includes an imaging element 7 having PD divided pixels 21, and a camera control unit 14. The camera control unit 14 controls the first readout and the second readout to be performed in a time-division manner as readouts corresponding to one frame (one vertical period) of an image when performing rolling shutter readout from the imaging element 7, and controls the first readout and the second readout to be separately provided as an exposure period for the first readout and an exposure period for the second readout. The first readout is for reading out a (L+R) value that is the sum of the left PD40L (first pixel) and the right PD40R (second pixel) that constitute the PD divided pixels 21 for all pixels to be used for image generation, and the second readout is for obtaining the value of the left PD40L (first pixel) and the value of the right PD40R (second pixel) for some of the pixels to be used for image generation. This makes it possible to avoid the influence of left-right non-additive readout for phase difference detection when reading out all pixels to be used for image generation. In other words, it is possible to avoid an operation such as reading out one horizontal line twice when reading out all pixels to be used for image generation. This makes it possible to prevent the generation of images with misaligned edges for moving subjects when performing rolling shutter readout. Note that the "all pixels to be used for image generation" in the first readout does not refer to all pixels of the image sensor 7. For example, if dummy pixels are provided, they will be excluded. It also does not always refer to all effective pixels. When pixels are thinned out from among all effective pixels as a readout to form an image during imaging, it means all pixels after thinning that are the subject of the readout. For example, when a low-resolution mode is selected for still images, or when capturing a moving image or taking continuous still images, the pixels that form one frame of an image may be thinned out. In these cases, "all pixels to be used for image generation" refers to all pixels that are read out after thinning out.

[0137] In the first to fifth embodiments, an image is generated based on the added value obtained in the first readout, and a phase difference detection process is performed using the value of the left PD40L and the value of the right PD40R obtained in the second readout. The PD divided pixel 21 is an effective pixel for image generation and also a pixel for calculating the amount of defocus by phase difference detection, so it is usually assumed that it will be read out once. In that case, by reading out twice for each horizontal line, that is, reading out the left PD 40L and reading out the left PD 40L and the right PD 40R in addition, in the case of rolling shutter readout, unnatural lines (zigzag lines) will appear on the edges of a moving subject. In the case of the processing of the embodiment, the first readout and the second readout are performed in a time-division manner, so that the first readout related to image generation does not cause two readouts of one horizontal line. This makes it possible to prevent unnatural edges from appearing in the image, and to improve image quality. In addition, the exposure period is set separately for the first readout and the second readout, which means that the (L+R) value from the first readout is not used, but the (L+R) value and the L value from the exposure before the second readout are used. In other words, this is to ensure that the values ​​of the left PD40L and the right PD40 can be obtained as values ​​for the same exposure opportunity. This allows the defocus amount to be calculated correctly, and even if readout is performed in a time-division manner, accurate phase difference information can be obtained, and for example, AF performance can be maintained.

[0138] In the first and third embodiments, the second readout is executed after the first readout is completed as the readout corresponding to one frame of the image. That is, first, a first readout is performed for image generation, and then a second readout is performed for focus control in the subsequent frame. The order of the first readout and the second readout is not limited, but performing the first readout first is advantageous in that, for example, imaging with a small time lag from the release operation can be achieved.

[0139] In the second, fourth and fifth embodiments, the first readout is executed after the second readout is completed as the readout corresponding to one frame of the image. For example, first, the second readout is performed for focus control in the subsequent frame, and then the second readout is performed for image generation. By performing the second readout first within one vertical period, it is possible to increase the possibility that AF control according to defocus will be possible in the next frame, for example, which is advantageous for improving AF responsiveness. In this case, when continuous shooting is considered, the time lag from the release operation becomes large, but this is hardly a problem in the case of, for example, movie shooting.

[0140] In the first to fifth embodiments, the exposure for the second readout is started before the first readout is completed. Even if the first readout is in progress, exposure for the lines for which the readout has been completed is possible, and therefore exposure for the second readout is started. This makes it possible to shorten the period required for the operations that proceed from the first readout, exposure for the second readout, and then the second readout. In other words, this is advantageous for shortening one vertical period during continuous shooting. This is particularly suitable for continuous shooting, since it is desirable to make one frame (one vertical period) as short as possible. Incidentally, the exposure for the second readout may start after the first readout is completed.

[0141] In the first to fifth embodiments, the second readout is completed during exposure for the first readout. That is, the second readout is performed in parallel with the exposure for the first readout, and is completed during the period of exposure for the first readout. This shortens the time required for the second readout, exposure for the first readout, and then the first readout. For example, the first readout can be started when the second readout is finished. This is also advantageous for shortening one vertical period when shooting continuously. Incidentally, after the second readout is completed, exposure for the first readout may be started.

[0142] In the first to fifth embodiments, in the second readout, the value of the left PD 40L and the added value of the left PD 40L and the right PD 40R are read out. For each horizontal line, for example, two readouts are performed, one for the left PD 40L and the other for the right PD 40R, to obtain the left PD 40L and right PD 40R values ​​for calculating the defocus amount. Because the value from the second readout is not used in image generation, performing such a double readout does not affect the image. In the second readout, the value of the right PD 40R may be read out, and the sum of the left PD 40L and the right PD 40R may be read out. In the second readout, for example, two readouts may be performed, one for the left PD 40L and one for the right PD 40R, or the left PD 40L and the right PD 40R may be simultaneously and independently readout.

[0143] In the third, fourth and fifth embodiments, the image sensor 7 has a light-shielding pixel 23 having a pupil division function by comprising a light-shielding portion that blocks one of a pair of light beams that pass through a pair of partial regions biased in opposite directions in a predetermined direction at the exit pupil, and a light-receiving element that receives the other light beam, and the camera control unit 14 controls so that a third readout is performed to read out the light-shielding pixel 23, and phase difference detection processing is performed using the value of the light-shielding pixel 23 obtained in the third readout. By using an imaging element 7 having a PD split pixel 21 and a light-shielding pixel 23, phase difference detection based on the PD split pixel 21 and phase difference detection based on the light-shielding pixel 23 can be performed, and can be selectively used depending on the situation, for example.

[0144] In the third embodiment, an example has been shown in which the third readout is performed before the second readout as the readout corresponding to one frame of an image. In the fourth and fifth embodiments, the example has been shown in which the second readout is performed before the third readout as the readout corresponding to one frame of an image. There is no restriction on the order of the second and third readouts, but by performing the third readout first, for example in situations where the reliability of the defocus amount based on light-shielded pixels is high, AF control can be performed based on the third readout, thereby improving responsiveness. Conversely, by performing the second readout first, in situations where the reliability of the defocus amount based on the PD divided pixels is high, AF control can be executed based on the second readout, thereby improving responsiveness. In this sense, which one is read first may be switched depending on the situation, such as brightness.

[0145] The fifth embodiment has been described as an example in which the third readout is performed together with the first readout. This eliminates the need to execute the third readout in a period separate from the first readout, which is advantageous in shortening one vertical period.

[0146] In the third embodiment, it has been described that focus control is performed using the more reliable result of the phase difference detection processing based on the second readout or the phase difference detection processing based on the third readout (see FIG. 18). This enables AF control that takes advantage of the respective advantages of PD split pixels and light-shielded pixels, improving AF reliability.

[0147] The thinning rate for setting the part of pixels to be read out in the second readout in the embodiment may be variably set. This enables accurate phase difference detection according to the situation. For example, by changing the thinning rate according to brightness, the number of pixels for which the second readout is performed can be reduced in bright conditions, thereby shortening the second readout time.

[0148] In the embodiment, an example has been given in which the first readout and the second readout are performed in a time-division manner when continuous still image shooting is performed. The effect of unnatural edges appearing in the image due to the readout for image generation and the readout for defocus amount calculation being performed together is easily noticeable in the case of continuous shooting imaging. Therefore, in the case of continuous shooting imaging, it is particularly effective to avoid image degradation by performing the first readout and the second readout in a time-division manner.

[0149] The program of the embodiment is a program that causes, for example, a CPU, a DSP, or a device including these to execute each process shown in FIG. 11, FIG. 12, FIG. 17, and FIG. That is, the program of the embodiment is a program that causes a processor in an imaging device 1 that includes an imaging element 7 that includes PD divided pixels 21 to execute a process in which a first readout that reads out the sum of the first and second pixels that constitute the PD divided pixels 21 for all pixels to be image-generated and a second readout that can obtain the values ​​of the first and second pixels for some of the pixels to be image-generated are performed in a time-division manner as readouts corresponding to one frame of an image when performing rolling shutter readout from the imaging element 7, and an exposure period for the first readout and an exposure period for the second readout are provided separately. The imaging device 1 described above can be realized by such a program.

[0150] A program for realizing such an imaging device 1 can be recorded in advance in a HDD serving as a recording medium built into a device such as the imaging device 1, or in a ROM or the like in a microcomputer having a CPU. Alternatively, the software may be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, a memory card, etc. Such removable recording media may be provided as a so-called package software. Furthermore, such a program can be installed in a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.

[0151] Furthermore, such a program is suitable for widespread provision of the imaging device 1 of the embodiment. For example, by downloading the program to a mobile terminal device such as a smartphone or tablet equipped with a camera function, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), or the like, these devices can function as the imaging device 1 of the present disclosure.

[0152] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0153] The present technology can also be configured as follows. (1) An imaging element having photodiode divided pixels; and a control unit that performs time division to perform a first readout for reading out an added value of a first pixel and a second pixel constituting the photodiode divided pixel for all pixels to be image-generated as a readout corresponding to one frame of an image when performing rolling shutter readout from the image sensor, and a second readout capable of obtaining a value of the first pixel and a value of the second pixel for some pixels among the pixels to be image-generated, and controls so that an exposure period for the first readout and an exposure period for the second readout are provided separately. Imaging device. (2) performing a process of generating an image based on the added value obtained in the first readout; A phase difference detection process is performed using the value of the first pixel and the value of the second pixel obtained in the second readout. The imaging device according to (1) above. (3) The control unit executes the second readout as a readout corresponding to one frame of an image after the first readout is completed. The imaging device according to (1) or (2) above. (4) The control unit executes the first readout as a readout corresponding to one frame of an image after the second readout is completed. The imaging device according to (1) or (2) above. (5) The control unit causes exposure for the second readout to start before the first readout ends. An imaging device according to any one of (1) to (4) above. (6) The control unit causes the second readout to end during exposure for the first readout. An imaging device according to any one of (1) to (5) above. (7) In the second readout, a value of one of the first pixel and the second pixel is read out, and a sum value of the first pixel and the second pixel is read out. An imaging device according to any one of (1) to (6) above. (8) The imaging element includes: a light-shielding pixel having a pupil division function by including a light-shielding portion for shielding one of a pair of light beams that have passed through a pair of partial regions that are biased in opposite directions to each other in a predetermined direction in the exit pupil and a light-receiving element for receiving the other light beam, The control unit controls to perform a third readout for reading out the light-shielded pixels, A phase difference detection process is performed using the values ​​of the light-shielded pixels obtained in the third readout. An imaging device according to any one of (1) to (7) above. (9) The third readout is performed prior to the second readout as a readout corresponding to one frame of an image. The imaging device according to (8) above. (10) The second readout is performed before the third readout as a readout corresponding to one frame of an image. The imaging device according to (8) above. (11) The third read is performed together with the first read. An imaging device according to any one of (8) to (10) above. (12) A focus control is performed using the result of the phase difference detection process based on the second readout or the result of the phase difference detection process based on the third readout, whichever is more reliable. An imaging device according to any one of (8) to (11) above. (13) The thinning rate for setting a part of pixels to be read out in the second readout is variably set. An imaging device according to any one of (1) to (12) above. (14) When continuous still image shooting is performed, the first readout and the second readout are performed in a time-division manner. An imaging device according to any one of (1) to (11) above. (15) As an imaging method of an imaging device including an imaging element having photodiode divided pixels, When performing rolling shutter readout from the imaging element, readout corresponding to one frame of an image includes a first readout for reading out the sum of a first pixel and a second pixel constituting the photodiode divided pixel for all pixels to be image-generated, and a second readout for determining the value of the first pixel and the value of the second pixel for some of the pixels to be image-generated, which are performed in a time-division manner, and an exposure period for the first readout and an exposure period for the second readout are provided separately. Imaging method. [Explanation of symbols]

[0154] 1. Imaging device 2 Camera housing 3 Lens barrel 6S Shutter button 7. Image sensor 8 Camera signal processing section 8a Phase difference detector 14 Camera control section 21 PD split pixels 22A 1st pixel row 22B 2nd pixel row 23, 23R, 23L Light-shielding pixels 40L Left PD 40R Right PD

Claims

1. an image sensor including a photodiode divided pixel and a light-shielding pixel having a pupil division function by including a light-shielding portion for blocking one of a pair of light beams that have passed through a pair of partial regions biased in opposite directions to each other in a predetermined direction at an exit pupil and a light-receiving element for receiving the other light beam; a control unit that performs control so that when performing rolling shutter readout from the image sensor, a first readout for reading out an added value of a first pixel and a second pixel constituting the photodiode divided pixel for all pixels to be image generated, a second readout for determining a value of the first pixel and a value of the second pixel for some pixels to be image generated, are performed in a time-division manner as readout corresponding to one frame of an image, and further a third readout for reading out the light-shielded pixels is performed, and an exposure period for the first readout and an exposure period for the second readout are provided separately, A focus control is performed using a result of the phase difference detection process based on the second readout or a result of the phase difference detection process performed using the values ​​of the light-shielded pixels obtained in the third readout, whichever has a higher reliability. Imaging device.

2. An imaging element having photodiode divided pixels; a control unit that performs time division to perform a first readout for reading out a sum value of a first pixel and a second pixel constituting the photodiode divided pixel for all pixels to be image generated, and a second readout for determining a value of the first pixel and a value of the second pixel for some pixels to be image generated, as a readout corresponding to one frame of an image when performing rolling shutter readout from the image sensor, and that controls so that an exposure period for the first readout and an exposure period for the second readout are provided separately; The thinning rate for setting a part of pixels to be read out in the second readout is variably set. Imaging device.

3. performing a process of generating an image based on the added value obtained in the first readout; A phase difference detection process is performed using the value of the first pixel and the value of the second pixel obtained in the second readout. The imaging device according to claim 2 .

4. The control unit executes the second readout as a readout corresponding to one frame of an image after the first readout is completed. The imaging device according to claim 1 or 2.

5. The control unit executes the first readout as a readout corresponding to one frame of an image after the second readout is completed. The imaging device according to claim 1 or 2.

6. The control unit controls the exposure for the second readout to start before the first readout ends. The imaging device according to claim 1 or 2.

7. The control unit causes the second readout to end during exposure for the first readout. The imaging device according to claim 1 or 2.

8. In the second readout, a value of one of the first pixel and the second pixel is read out, and a sum value of the first pixel and the second pixel is read out. The imaging device according to claim 1 or 2.

9. The third readout is performed prior to the second readout as a readout corresponding to one frame of an image. The imaging device according to claim 1 .

10. The second readout is performed prior to the third readout as a readout corresponding to one frame of an image. The imaging device according to claim 1 .

11. The third readout is performed together with the first readout. The imaging device according to claim 1 .

12. When continuous still image shooting is performed, the first readout and the second readout are performed in a time-division manner. The imaging device according to claim 1 or 2.

13. An imaging method for an imaging device including an imaging element having a photodiode divided pixel and a light-shielding pixel having a pupil division function by being provided with a light-shielding part that blocks one of a pair of light beams that pass through a pair of partial regions biased in opposite directions to each other in a predetermined direction in an exit pupil and a light-receiving element that receives the other light beam, When performing rolling shutter readout from the imaging element, readout corresponding to one frame of an image is performed in a time-division manner, with a first readout for reading out the sum of a first pixel and a second pixel constituting the photodiode divided pixel for all pixels to be image-generated, and a second readout capable of determining the value of the first pixel and the value of the second pixel for some of the pixels to be image-generated, a third readout is performed to read out the light-shielded pixels, and an exposure period for the first readout and an exposure period for the second readout are provided separately; A focus control is performed using a result of the phase difference detection process based on the second readout or a result of the phase difference detection process performed using the values ​​of the light-shielded pixels obtained in the third readout, whichever has a higher reliability. Imaging method.

14. An imaging method for an imaging device having an imaging element having photodiode divided pixels, comprising: When performing rolling shutter readout from the imaging element, readout corresponding to one frame of an image includes a first readout for reading out an added value of a first pixel and a second pixel constituting the photodiode divided pixel for all pixels to be image-generated, and a second readout for determining a value of the first pixel and a value of the second pixel for some of the pixels to be image-generated, which are performed in a time-division manner, and an exposure period for the first readout and an exposure period for the second readout are provided separately; A thinning rate for setting a part of pixels to be read out in the second readout is set variably. Imaging method.

Images