Image sensor, imaging device, method of operating the image sensor, and program

Abstract

This invention provides an image sensor, an imaging device, a method for operating the image sensor, and a program that enable high-precision autofocus with a simple configuration. [Solution] The image sensor includes a readout circuit that reads out pixel data obtained by imaging a subject at a first frame rate, a memory that stores the readout pixel data, and an output circuit that outputs image data based on the stored pixel data at a second frame rate. The first frame rate is a higher frame rate than the second frame rate. The pixel data includes phase difference pixel data and non-phase difference pixel data that is different from the phase difference pixel data. The readout circuit reads out the pixel data of each of multiple frames in parallel within an output period defined by the second frame rate as the period during which one frame of image data is output, and performs reading of non-phase difference pixel data and reading of phase difference pixel data multiple times within the output period.

Description

Technical Field

[0001] The technology of the present disclosure relates to an imaging device, an imaging apparatus, a method of operating an imaging device, and a program.

Background Art

[0002] Japanese Patent Application Laid-Open No. 2014-178603 discloses an imaging apparatus including imaging means, a target area determination means, control means, and focus detection means.

[0003] In the imaging apparatus described in Japanese Patent Application Laid-Open No. 2014-178603, the imaging means has a plurality of imaging areas and generates an image signal corresponding to the light rays incident on the imaging areas. The target area determination means determines a target area of the image indicated by the image signal based on the image signal output from the imaging means.

[0004] The control means includes a first control unit and a second control unit. The first control unit controls so as to image an imaging area on which a light image corresponding to the target area among the plurality of imaging areas is incident under a first imaging condition. The second control unit controls so as to image an imaging area other than the imaging area on which a light image corresponding to the target area among the plurality of imaging areas is incident under a second imaging condition different from the first imaging condition. The focus detection means detects a focus adjustment state of the target area. The first control unit controls so as to perform imaging at a higher frame rate than the second control unit.

Summary of the Invention

[0005] One embodiment according to the technology of the present disclosure provides an imaging device, an imaging apparatus, a method of operating an imaging device, and a program that can improve the accuracy of autofocus with a simple configuration as compared with a case where pixel data is read out a plurality of times from pixels dedicated to autofocus during a read period for reading out an image for one frame.

Means for Solving the Problems

[0006] The first aspect of the technology of this disclosure is an image sensor including phase-difference pixels, comprising: a readout unit built into the image sensor that reads out pixel data obtained by imaging a subject at a first frame rate; a storage unit built into the image sensor that stores the pixel data read out by the readout unit; and an output unit built into the image sensor that outputs image data based on the pixel data stored in the storage unit at a second frame rate, wherein the first frame rate is a higher frame rate than the second frame rate, the pixel data includes phase-difference pixel data and non-phase-difference pixel data different from the phase-difference pixel data, and the readout unit reads out the pixel data of each of multiple frames in parallel within an output period defined by the second frame rate as the period during which one frame of image data is output, and performs reading of non-phase-difference pixel data and reading out phase-difference pixel data multiple times within the output period. This makes it possible to improve the accuracy of autofocus with a simpler configuration compared to the case in which pixel data is read out multiple times from an autofocus-dedicated pixel within the readout period for reading out one frame of image.

[0007] A second aspect of the technology of this disclosure is an image sensor according to the first aspect, in which the readout unit performs the reading of non-phase-difference pixel data and the reading of phase-difference pixel data in parallel. This allows for the reading of more phase-difference pixel data and non-phase-difference pixel data within a limited time compared to a case where the reading of one of the non-phase-difference pixel data and phase-difference pixel data is completed before the reading of the other is started.

[0008] A third aspect of the technology of this disclosure is an image sensor according to the first or second aspect, in which the readout unit reads out pixel data in line units. This makes it possible to thin out the pixel data in line units.

[0009] A fourth aspect of the technology of this disclosure is an image sensor relating to any one of the first to third aspects, wherein the phase difference pixel data is the pixel data of a phase difference pixel, and the non-phase difference pixel data is the pixel data of a non-phase difference pixel, which is a pixel different from the phase difference pixel. Compared to the case where the phase difference pixel data is generated from the pixel data of pixels other than phase difference pixels, and the non-phase difference pixel data is generated from the pixel data of pixels other than non-phase difference pixels, the phase difference pixel data and the non-phase difference pixel data can be easily obtained.

[0010] A fifth aspect of the technology of this disclosure is an image sensor relating to a fourth aspect, wherein the readout of non-phase difference pixel data is the readout of non-phase difference pixel data from a non-phase difference pixel, and the multiple readout of phase difference pixel data is the multiple readout of phase difference pixel data from a phase difference pixel. This makes it possible to obtain both non-phase difference pixel data and multiple phase difference pixel data within the output period, compared to the case where the readout of phase difference pixel data from a phase difference pixel and the readout of non-phase difference pixel data from a non-phase difference pixel are performed alternately once each during the output period.

[0011] A sixth aspect of the technology of this disclosure is an image sensor according to the fourth or fifth aspect, which includes an imaging surface in which a plurality of first lines including phase difference pixels and a plurality of second lines consisting only of non-phase difference pixels are arranged, and the readout unit has a first readout unit that reads phase difference pixel data from each of the phase difference pixels included in the plurality of first lines, and a second readout unit that reads non-phase difference pixel data from each of the non-phase difference pixels included in the plurality of second lines. This makes it possible to distribute the load on reading out the phase difference pixel data and the load on reading out the non-phase difference pixel data.

[0012] A seventh aspect of the technology of this disclosure is an image sensor according to the sixth aspect, in which the reading of phase difference pixel data from phase difference pixels by a first readout unit and the reading of non-phase difference pixel data from non-phase difference pixels by a second readout unit are performed independently. This makes it possible to avoid one of the reading of phase difference pixel data and the reading of non-phase difference pixel data affecting the other.

[0013] An eighth aspect of the technology of this disclosure is an image sensor according to the seventh aspect, wherein, within a reading period of one frame, the reading of phase difference pixel data from phase difference pixels by the first reading unit is performed before the reading of non-phase difference pixel data from non-phase difference pixels by the second reading unit. As a result, the phase difference pixel data can be made available for autofocus processing earlier compared to the case where the reading of non-phase difference pixel data is performed before the reading of phase difference pixel data.

[0014] A ninth aspect of the technology of this disclosure is an image sensor according to any one of the sixth to eighth aspects, wherein the first line is a line in which phase-difference pixels and non-phase-difference pixels are arranged periodically. This makes it possible to improve the accuracy of autofocus over a wide area compared to using a line in which phase-difference pixels and non-phase-difference pixels are arranged locally concentrated.

[0015] A tenth aspect of the technology of this disclosure is an image sensor according to any one of the sixth to ninth aspects, wherein a first line and a predetermined number of second lines are alternately arranged on the imaging surface along a direction intersecting the line direction of the first line. This improves the accuracy of autofocus over a wide area compared to the case where the first line and the second line are locally concentrated in the direction intersecting the line direction of the first line. It is possible.

[0016] An eleventh aspect of the technology of this disclosure is an image sensor relating to any one of the first to tenth aspects, wherein the readout unit reads out one frame's worth of non-phase difference pixel data as recording pixel data within a readout period for one frame, and reads out phase difference pixel data while the non-phase difference pixel data is being read out as recording pixel data. This makes it possible to read out more non-phase difference pixel data and phase difference pixel data as recording pixel data within a limited time compared to the case where the phase difference pixel data is read out after waiting for the non-phase difference pixel data to be read out as recording pixel data.

[0017] A twelfth aspect of the technology of this disclosure is an image sensor according to the eleventh aspect, wherein the readout unit reads out non-phase-difference pixel data and phase-difference pixel data as display pixel data within a readout period of one frame, and reads out the non-phase-difference pixel data as recording pixel data when predetermined conditions are met. This increases versatility compared to the case where non-phase-difference pixel data is always read out as recording pixel data.

[0018] A thirteenth aspect of the technology of this disclosure is an image sensor according to an eleventh aspect, wherein the readout unit reads out non-phase-difference pixel data as recording pixel data in continuous shooting mode. This makes it possible to record the non-phase-difference pixel data read out as recording pixel data in continuous shooting mode.

[0019] A fourteenth aspect of the technology of this disclosure further includes a derivation unit that derives a correction coefficient for correcting attenuation characteristics caused by phase difference pixels based on phase difference pixel data, and an output unit is an image sensor according to any one of the first to thirteenth aspects that outputs the correction coefficient derived by the derivation unit. This makes it possible to correct attenuation characteristics that appear in an image based on phase difference pixel data.

[0020] A fifteenth aspect of the technology of this disclosure is an image sensor according to any one of the first to fourteenth aspects, wherein the image data includes first pixel data based on non-phase difference pixel data and second pixel data based on phase difference pixel data obtained by multiple readouts, and the output unit outputs the first pixel data and the second pixel data at different timings when outputting image data for one frame. This contributes to miniaturization of the image sensor compared to the case where a dedicated output circuit is used for each of the phase difference pixel data and the non-phase difference pixel data.

[0021] A sixteenth aspect of the technology of this disclosure is an image sensor according to the fifteenth aspect, wherein the output unit outputs the first pixel data after the output of the second pixel data is completed. This allows the phase-difference pixel data to be used for autofocus processing sooner compared to the case where the phase-difference pixel data is output after the output of the non-phase-difference pixel data is completed.

[0022] A 17th aspect of the technology of this disclosure is an image sensor relating to any one of the first to 16th aspects, wherein the image data includes pixel data based on non-phase difference pixel data and pixel data based on statistical values ​​of phase difference pixel data obtained by multiple readouts. This makes it possible to reduce the amount of output data from the image sensor compared to outputting multiple phase difference pixel data obtained for each of each frame.

[0023] An eighteenth aspect of the technology of this disclosure is an image sensor according to the seventeenth aspect, wherein the statistical value is the summation average of phase difference pixel data. This reduces the amount of output data from the image sensor compared to outputting multiple phase difference pixel data obtained for each of each frame.

[0024] A 19th aspect of the technology of this disclosure includes an A / D converter shared for phase-difference pixel data and non-phase-difference pixel data, wherein the A / D converter is an image sensor according to any one of the first to 18th aspects, which performs A / D conversion on the phase-difference pixel data and non-phase-difference pixel data at different timings. This contributes to miniaturization of the image sensor compared to using a dedicated A / D converter for each of the phase-difference pixel data.

[0025] The 20th aspect related to the technology of the present disclosure is an imaging device according to any one of the 1st to 18th aspects, including a plurality of A / D converters, the plurality of A / D converters including a first A / D converter used only for phase difference pixel data and a second A / D converter used only for non-phase difference pixel data. Thereby, within the output period, the A / D conversion of the phase difference pixel data and the A / D conversion of the non-phase difference pixel data can be performed in parallel.

[0026] The 21st aspect related to the technology of the present disclosure is an imaging device according to any one of the 1st to 20th aspects in which at least a photoelectric conversion element and a storage unit are integrated into one chip. Thereby, the portability of the imaging device is higher than that of an imaging device in which the photoelectric conversion element and the storage unit are not integrated into one chip.

[0027] The 22nd aspect related to the technology of the present disclosure is an imaging device according to the 21st aspect, which is a stacked imaging device in which a storage unit is stacked on a photoelectric conversion element. Thereby, the transfer speed of image data from the photoelectric conversion element to the storage unit can be increased compared to the case where the photoelectric conversion element and the storage unit are not stacked.

[0028] The 23rd aspect related to the technology of the present disclosure is an imaging device including an imaging device according to any one of the 1st to 22nd aspects, and a control unit that performs at least one of control to display an image based on the image data output by the output unit on the display unit and control to store the image data output by the output unit in the storage device. Thereby, compared to the case where pixel data is read out from the autofocus dedicated pixels a plurality of times during the readout period for reading out an image for one frame, autofocus can be made highly accurate with a simple configuration.

[0029] A 24th aspect of the technology of this disclosure is an operating method for an image sensor incorporating a phase-difference pixel, a readout unit that reads out pixel data obtained by imaging a subject at a first frame rate, a storage unit that stores the pixel data read out by the readout unit, and an output unit that outputs image data based on the pixel data stored in the storage unit at a second frame rate, wherein the first frame rate is a higher frame rate than the second frame rate, the pixel data includes phase-difference pixel data and non-phase-difference pixel data different from the phase-difference pixel data, and the readout unit reads out the pixel data of each of multiple frames in parallel within an output period defined by the second frame rate as the period during which one frame of image data is output, and within the output period, includes reading out the non-phase-difference pixel data and reading out the phase-difference pixel data multiple times. This makes it possible to improve the accuracy of autofocus with a simpler configuration compared to the case in which pixel data is read out multiple times from an autofocus-dedicated pixel within the readout period for reading out one frame of image.

[0030] A 25th aspect of the technology of this disclosure includes a phase-difference pixel, a readout unit that reads out pixel data obtained by imaging a subject at a first frame rate, a storage unit that stores the pixel data read out by the readout unit, and an output unit that outputs image data based on the pixel data stored in the storage unit at a second frame rate, wherein the readout unit, storage unit, and output unit are incorporated into an image sensor and a program for causing a computer to function as the readout unit and output unit. The first frame rate is higher than the second frame rate, and the pixel data includes phase-difference pixel data and non-phase-difference pixel data that is different from the phase-difference pixel data. The reading unit is a program that reads the pixel data of multiple frames in parallel within the output period defined by the second frame rate as the period during which one frame of image data is output, and also reads the non-phase-difference pixel data and reads the phase-difference pixel data multiple times within the output period. This makes it possible to improve the accuracy of autofocus with a simpler configuration compared to the case where pixel data is read multiple times from autofocus-dedicated pixels within the reading period for reading one frame of image.

[0031] A 26th aspect of the technology of this disclosure is an image sensor that includes phase-difference pixels and has a built-in processor and memory, wherein the processor reads out pixel data obtained by imaging a subject at a first frame rate, the memory stores the pixel data read out by the processor, and outputs image data based on the pixel data stored in the memory at a second frame rate, the first frame rate being a higher frame rate than the second frame rate, the pixel data includes phase-difference pixel data and non-phase-difference pixel data different from the phase-difference pixel data, and the processor reads out the pixel data of each of multiple frames in parallel within an output period defined by the second frame rate as the period during which one frame of image data is output, and performs reading of non-phase-difference pixel data and reading of phase-difference pixel data multiple times within the output period. [Brief explanation of the drawing]

[0032] [Figure 1] This is a perspective view showing an example of the external appearance of the imaging device according to the first and second embodiments. [Figure 2] This block diagram shows an example of the configuration of an imaging device according to the first and second embodiments. [Figure 3A] This is a conceptual diagram illustrating the imaging frame rate of the image sensor included in the imaging apparatus according to the first and second embodiments. [Figure 3B]This is a conceptual diagram illustrating the output frame rate of the image sensor included in the imaging apparatus according to the first and second embodiments. [Figure 4] This block diagram shows an example of the electrical system configuration of the imaging device body according to the first and second embodiments. [Figure 5] This block diagram shows an example of the stacked structure of an image sensor included in the imaging apparatus according to the first and second embodiments, as well as an example of the connection relationship between the image sensor, the signal processing unit, and the controller. [Figure 6] This is an arrangement diagram showing an example of the arrangement of each pixel included in the photoelectric conversion element of an imaging sensor included in the imaging device according to the first and second embodiments on the imaging surface. [Figure 7] Figure 6 is a conceptual diagram showing an example of the incident light characteristics of the first and second phase difference pixels included in the photoelectric conversion element. [Figure 8] Figure 6 is a schematic diagram showing an example of the configuration of non-phase difference pixels included in the photoelectric conversion element. [Figure 9] This block diagram shows an example of the electrical system configuration of an image sensor included in the imaging device according to the first embodiment. [Figure 10] This is a time chart showing an example of the readout timing for each frame of analog phase-difference pixel data and analog non-phase-difference pixel data by the readout circuit of the image sensor included in the imaging device according to the first embodiment. [Figure 11] This is a time chart showing an example of the timing for reading analog pixel data, A / D conversion, storage to memory, and output of digital pixel data within the image sensor included in the imaging device according to the first embodiment. [Figure 12] This flowchart shows an example of the timing control process flow according to the first embodiment. [Figure 13] This flowchart shows an example of the phase difference pixel processing flow according to the first embodiment. [Figure 14] This flowchart shows an example of the non-phase-difference pixel processing flow according to the first embodiment. [Figure 15]This flowchart shows an example of the pixel data processing flow according to the first embodiment. [Figure 16] This is a time chart showing a first modified example of the timing of reading analog pixel data, A / D conversion, storage to memory, and output of digital pixel data within the image sensor included in the imaging device according to the first embodiment. [Figure 17] This is a time chart showing a second modified example of the timing of reading analog pixel data, A / D conversion, storage to memory, and output of digital pixel data within the image sensor included in the imaging device according to the first embodiment. [Figure 18] This is a time chart showing a third modified example of the timing of reading analog pixel data, A / D conversion, storage to memory, and output of digital pixel data within the image sensor included in the imaging device according to the first embodiment. [Figure 19] This block diagram shows an example of the electrical system configuration of an image sensor included in the imaging device according to the second embodiment. [Figure 20] This is a time chart showing an example of the timing of reading analog pixel data for each frame by the image sensor readout circuit included in the imaging device according to the second embodiment. [Figure 21] This is a time chart showing an example of the timing for reading analog pixel data, A / D conversion, storage to memory, and outputting digital pixel data within the image sensor included in the imaging device according to the second embodiment. [Figure 22] This flowchart shows an example of the in-image sensor processing flow according to the second embodiment. [Figure 23] This is a continuation of the flowchart shown in Figure 22. [Figure 24] This flowchart shows an example of the pixel data processing flow according to the second embodiment. [Figure 25]This graph shows an example of the attenuation characteristics of a first phase difference image based on first phase difference pixel data from a first phase difference pixel included in the image sensor according to the second embodiment, and the attenuation characteristics of a second phase difference image based on second phase difference pixel data from a second phase difference pixel. [Figure 26] This block diagram shows an example of a calculation circuit for calculating a correction coefficient to correct the attenuation characteristics shown in Figure 25. [Figure 27] This is a conceptual diagram showing an example of the before and after states of correction of the first phase difference image based on first phase difference pixel data from the first phase difference pixel included in the image sensor according to the second embodiment, and the second phase difference image based on second phase difference pixel data from the second phase difference pixel. [Figure 28] This is a block diagram showing modified configurations of the electrical system of an image sensor included in the imaging apparatus according to the first and second embodiments. [Figure 29] This is a conceptual diagram illustrating an example of how various programs are installed from a storage medium containing them into a computer within an image sensor. [Figure 30] This block diagram shows an example of a schematic configuration of a smart device incorporating an image sensor according to the first and second embodiments. [Modes for carrying out the invention]

[0033] Hereinafter, an example of an embodiment of an imaging device relating to the technology of this disclosure will be described with reference to the attached drawings.

[0034] First, let's explain the meaning of the terms used in the following explanation.

[0035] CPU stands for "Central Processing Unit." RAM stands for "Random Access Memory." ROM stands for "Read Only Memory." DRAM stands for "Dynamic This refers to an abbreviation for "Random Access Memory." SRAM is an abbreviation for "Static Random Access Memory."

[0036] LSI stands for "Large-Scale Integration." ASIC stands for "Application Specific Integrated Circuit." PLD stands for "Programmable Logic Device." FPGA stands for "Field-Programmable Logic Device." This refers to an abbreviation for "Gate Array".

[0037] SSD stands for "Solid State Drive." DVD-ROM stands for "Digital Versatile Disc Read Only Memory." USB stands for "Universal Serial Bus." HDD stands for "Hard Disk Drive." EEPROM stands for "Electrically Erasable and Programmable Read Only Memory."

[0038] CCD stands for "Charge Coupled Device". CMOS stands for "Complementary Metal Oxide Semiconductor". EL stands for "Electro-Luminescence". A / D stands for "Analog / Digital". I / F stands for "Interface". UI stands for "User Interface". PC stands for "Personal Computer". AF stands for "Auto-Focus". AE stands for "Automatic Exposure". SoC stands for "System-on-a-chip".

[0039] [First Embodiment] As an example, as shown in Figure 1, the imaging device 10 is a lens-interchangeable camera. The imaging device 10 comprises an imaging device body 12 and an interchangeable lens 14 that is interchangeably attached to the imaging device body 12.

[0040] The imaging device body 12 is equipped with an image sensor 44. When the interchangeable lens 14 is attached to the imaging device body 12, the subject light indicating the subject passes through the interchangeable lens 14 and is imaged onto the imaging surface 44A of the image sensor 44.

[0041] A release button 20 and a dial 22 are provided on the top surface of the imaging device body 12. The dial 22 is operated when setting the operating mode of the imaging system and the operating mode of the playback system. The release button 20 functions as an imaging preparation indicator and an imaging indicator, and can detect two stages of pressing operation: an imaging preparation indicator state and an imaging indicator state. The imaging preparation indicator state refers to the state in which the button is pressed from the standby position to an intermediate position (half-press position), for example, and the imaging indicator state refers to the state in which the button is pressed beyond the intermediate position to the final pressed position (full-press position). Hereafter, the state in which the button is pressed from the standby position to the half-press position will be referred to as the "half-press state," and the state in which the button is pressed from the standby position to the full-press position will be referred to as the "full-press state."

[0042] In the imaging device 10, the imaging mode and playback mode are selectively set according to the user's instructions. The imaging mode is broadly divided into an imaging mode for display video and an imaging mode for recording. In each of the imaging modes, the AF mode is set according to the user's instructions.

[0043] In the video display mode, when the AF mode is set, the AE camera will... The camera's functions activate to set the exposure, the autofocus (AF) function activates to control focus, and image capture for the display video is performed. This image capture generates the live view image. Generally, the live view image is also referred to as a through image.

[0044] The recording imaging modes are broadly divided into motion image recording imaging modes and still image recording imaging modes, and these modes are selectively set according to the user's instructions. In the imaging device 10, when the AF mode is set in motion image recording imaging mode, the AE function works to set the exposure state for each frame, and the AF function works to control focus, and imaging for recording motion is performed. The motion images obtained by performing imaging for recording motion are recorded on a predetermined recording medium such as a memory card or USB memory (hereinafter also simply referred to as the "predetermined recording medium"). The motion images obtained by performing imaging for recording motion are an example of "recording pixel data" related to the technology of this disclosure.

[0045] In the still image recording mode, when the AF mode is set, the shooting conditions are adjusted by half-pressing the release button 20, and then, by fully pressing it, still image capture is performed. In other words, by half-pressing the release button 20, the AE function is activated and the exposure state is set, then the AF function is activated and focus control is performed, and when the release button 20 is fully pressed, still image capture for recording is performed. The still image obtained by performing still image capture for recording is recorded on a predetermined recording medium. The still image obtained by performing still image capture for recording is an example of "recording pixel data" related to the technology of this disclosure.

[0046] As an example, as shown in Figure 2, the interchangeable lens 14 has an imaging lens 40. The imaging lens 40 comprises an objective lens 40A, a focusing lens 40B, and an aperture 40C. The objective lens 40A, focusing lens 40B, and aperture 40C are arranged in that order along the optical axis L1 from the subject side to the imaging device body 12 side. The aperture 40C is operated by receiving power from a drive source such as a motor (not shown). This changes the opening of the aperture 40C. By changing the opening of the aperture 40C, the exposure is adjusted.

[0047] The focus lens 40B is attached to the slide mechanism 15. A motor 17 is connected to the slide mechanism 15. The motor 17 generates power and transmits the generated power to the slide mechanism 15, thereby operating the slide mechanism 15. The slide mechanism 15 moves the focus lens 40B along the optical axis L1 in accordance with the power supplied by the motor 17.

[0048] The motor 17 is connected to the controller 46 of the imaging device body 12 via a communication line 55. The motor 17 is controlled by the controller 46. In AF mode, the focus lens 40B moves along the optical axis L1 under the control of the controller 46, so that the subject light is imaged onto the imaging surface 44A of the image sensor 44 at a focus position corresponding to the subject distance. Herein, "focus position" refers to the position of the focus lens 40B on the optical axis L1 when in focus. For the sake of explanation, the control that adjusts the focus lens 40B to the focus position will be referred to as "AF control" below.

[0049] The imaging device body 12 includes a mechanical shutter 42 and an image sensor 44. The mechanical shutter 42 operates by receiving power from a drive source such as a motor (not shown). When the interchangeable lens 14 is attached to the imaging device body 12, the subject light passes through the imaging lens 40 and is imaged onto the imaging surface 44A of the image sensor 44 via the mechanical shutter 42.

[0050] The imaging device body 12 comprises a controller 46, a UI system device 48, and a signal processing unit 50. Both the controller 46 and the signal processing unit 50 are implemented using LSIs. Furthermore, since both the controller 46 and the signal processing unit 50 are located after the image sensor 44, they can also be considered the downstream circuits of the image sensor 44.

[0051] The controller 46 controls the entire imaging device 10. The UI device 48 is a device that presents information to the user and receives instructions from the user. The UI device 48 is connected to the controller 46, and the controller 46 acquires various information from the UI device 48 and controls the UI device 48.

[0052] The image sensor 44 is connected to the controller 46 via a communication line 57, and under the control of the controller 46, it captures an image of the subject, thereby generating image data 69 that represents the image of the subject.

[0053] The image sensor 44 is connected to the signal processing unit 50 via a communication line 53. The signal processing unit 50 is a device including an ASIC. A controller 46 is connected to the signal processing unit 50 via a communication line 60.

[0054] The signal processing unit 50 receives image data 69 from the image sensor 44 via the communication line 53. The signal processing unit 50 performs various signal processing on the image data 69 received from the image sensor 44 via the communication line 53. These various signal processing processes include, for example, known signal processing such as white balance adjustment, sharpness adjustment, gamma correction, color space conversion processing, and color difference correction.

[0055] In this first embodiment, a device including an ASIC is given as an example of the signal processing unit 50, but the technology of this disclosure is not limited thereto, and the signal processing unit 50 may be a device including an ASIC, FPGA, and / or PLD. Also, the signal processing unit 50 may be a computer including a CPU, ROM, and RAM. There may be one CPU or more CPUs. Furthermore, the signal processing unit 50 may be implemented by a combination of hardware and software configurations.

[0056] The image sensor 44 is an example of a "stacked image sensor" according to the technology of this disclosure. In this embodiment, the image sensor 44 is a CMOS image sensor. Although a CMOS image sensor is used as an example of the image sensor 44 here, the technology of this disclosure is not limited to this, and for example, the technology of this disclosure can also be established even if the image sensor 44 is a CCD image sensor.

[0057] In the image sensor 44, the subject is imaged at the imaging frame rate, generating multiple image data 69, each representing an image of the subject, as shown in Figure 3A as an example. The image sensor 44 also outputs the generated multiple image data 69 at the output frame rate. Both the imaging frame rate and the output frame rate are variable frame rates. The imaging frame rate is an example of the "first frame rate" related to the technology of this disclosure, and the output frame rate is an example of the "second frame rate" related to the technology of this disclosure.

[0058] The imaging frame rate and the output frame rate have the relationship "imaging frame rate > output frame rate". In other words, the imaging frame rate is higher than the output frame rate. For example, as shown in Figure 3A, the imaging frame rate is the frame rate at which 8 frames of imaging are performed within period T, and as shown in Figure 3B, the output frame rate is the frame rate at which 2 frames of output are performed within period T. Specifically, an example of an imaging frame rate is 240 fps (frames per second), and an example of an output frame rate is 60 fps.

[0059] As an example, as shown in Figure 4, the controller 46 includes a CPU 46A, ROM 46B, RAM 46C, a first communication interface 46D1, a second communication interface 46D2, and a third communication interface 46D3. The CPU 46A, ROM 46B, RAM 46C, first communication interface 46D1, second communication interface 46D2, and third communication interface 46D3 are interconnected via a bus line 88.

[0060] ROM46B stores various programs. CPU46A reads the programs from ROM46B and loads them into RAM46C. CPU46A controls the entire imaging device 10 according to the programs loaded into RAM46C.

[0061] The various programs stored in ROM 46B include a timing control program 46B1 and a pixel data processing program 46B2. The CPU 46A executes the timing control processing (see Figure 12) described later according to the timing control program 46B1. The CPU 46A also executes the pixel data processing (see Figure 15) described later according to the pixel data processing program 46B2.

[0062] The first communication interface 46D1, the second communication interface 46D2, and the third communication interface 46D3 are each communication devices having an FPGA. The first communication interface 46D1 is connected to the signal processing unit 50 via a communication line 60. Image data 69 (see Figures 2, 3A, and 3B) that has undergone various signal processing by the signal processing unit 50 is input to the first communication interface 46D1 via the communication line 60. The first communication interface 46D1 transfers the image data 69 input from the signal processing unit 50 to the CPU 46A.

[0063] The second communication interface 46D2 is connected to the image sensor 44 via the communication line 57. The CPU 46A controls the image sensor 44 via the second communication interface 46D2.

[0064] The third communication interface 46D3 is connected to the motor 17 via the communication line 55. The CPU 46A controls the motor 17 via the third communication interface 46D3.

[0065] Bus line 88 is connected to the secondary storage device 80 and the external interface 82. The secondary storage device 80 is a non-volatile memory such as an SSD, HDD, or EEPROM. The CPU 46A reads and writes various types of information to the secondary storage device 80.

[0066] The external I / F82 is a communication device equipped with an FPGA. A predefined recording medium is connected to the external I / F82. External devices such as a PC or server (not shown) are also connected to the external I / F82. The external I / F82 is responsible for the exchange of various types of information between the CPU46A and the external devices.

[0067] The UI device 48 includes a touch panel display 26 and a reception device 84. The display 32 is connected to a bus line 88. An example of the display 32 is a liquid crystal display. The display 32 may be a liquid crystal display or other type of display such as an organic EL display. Under the control of the CPU 46A, the display 32 displays various images such as live view images and still images, as well as text information. The display 32 is an example of a "display unit" related to the technology of this disclosure. The CPU 46A is an example of a "control unit (processor)" related to the technology of this disclosure.

[0068] The reception device 84 includes a hard key section 25 and a touch panel 34. The unit 25 consists of multiple hard keys, including a release button 20 and a dial 22. The touch panel 34 is a transmissive touch panel superimposed on the surface of the display area of ​​the display 32. The touch panel 34 detects contact by an indicator, such as a finger or a stylus pen. The hard key unit 25 and the touch panel 34 are connected to a bus line 88, and the CPU 46A operates according to the various instructions received by the hard key unit 25 and the touch panel 34, respectively.

[0069] As an example, as shown in Figure 5, the image sensor 44 incorporates a photoelectric conversion element 61, a processing circuit 62, and a memory 64. The image sensor 44 is an image sensor in which the photoelectric conversion element 61, processing circuit 62, and memory 64 are integrated into a single chip. That is, the photoelectric conversion element 61, processing circuit 62, and memory 64 are packaged together. In the image sensor 44, the processing circuit 62 and memory 64 are stacked on top of the photoelectric conversion element 61. Specifically, the photoelectric conversion element 61 and the processing circuit 62 are electrically connected to each other by conductive bumps made of copper or the like (not shown), and the processing circuit 62 and memory 64 are also electrically connected to each other by conductive bumps made of copper or the like (not shown). Here, a three-layer structure of the photoelectric conversion element 61, processing circuit 62, and memory 64 is exemplified, but the technology of this disclosure is not limited to this, and a two-layer structure of a memory layer consisting of the processing circuit 62 and memory 64 as one layer, and the photoelectric conversion element 61, is also possible. Memory 64 is an example of a "memory" related to the technology disclosed herein.

[0070] The processing circuit 62 is, for example, an LSI, and the memory 64 is, for example, DRAM. However, the technology of this disclosure is not limited to these, and SRAM may be used instead of DRAM as the memory 64.

[0071] The processing circuit 62 is a device including an ASIC and an FPGA, and controls the entire image sensor 44 according to instructions from the controller 46. While this example shows the processing circuit 62 being implemented by a device including an ASIC and an FPGA, the technology of this disclosure is not limited to this, and may include, for example, a device including an ASIC, FPGA, and / or PLD. Furthermore, a computer including a CPU, ROM, and RAM may be used as the processing circuit 62. The CPU may be single or multiple. Also, the processing circuit 62 may be implemented by a combination of hardware and software configurations.

[0072] The photoelectric conversion element 61 has multiple photodiodes arranged in a matrix. An example of multiple photodiodes is a set of photodiodes for "4896 × 3265" pixels.

[0073] Each photodiode in the photoelectric conversion element 61 is fitted with a color filter. The color filter includes a G filter corresponding to G (green), which contributes most to obtaining the luminance signal; an R filter corresponding to R (red); and a B filter corresponding to B (blue). The photoelectric conversion element 61 has R pixels, G pixels, and B pixels (see Figure 6). An R pixel is a pixel corresponding to a photodiode fitted with an R filter, a G pixel is a pixel corresponding to a photodiode fitted with a G filter, and a B pixel is a pixel corresponding to a photodiode fitted with a B filter.

[0074] The image sensor 44 has a so-called electronic shutter function, and by operating the electronic shutter function under the control of the controller 46, the charge accumulation time of each photodiode in the photoelectric conversion element 61 is controlled. The charge accumulation time refers to the so-called shutter speed.

[0075] The imaging device 10 uses a rolling shutter system for capturing still images and for capturing moving images. The following is performed. In the still image recording mode, still image capture is achieved by activating the electronic shutter function and the mechanical shutter 42 (see Figure 2). In the still image recording mode, continuous shooting is achieved by activating the electronic shutter function without activating the mechanical shutter 42. In the video recording mode, video capture is also achieved by activating the electronic shutter function without activating the mechanical shutter 42. Furthermore, in the display video capture mode, live view image capture is also achieved by activating the electronic shutter function without activating the mechanical shutter 42. Although a rolling shutter method is exemplified here, the technology of this disclosure is not limited to this, and a global shutter method may be applied instead of the rolling shutter method.

[0076] The processing circuit 62 reads out the image data 69 obtained when the subject is imaged by the photoelectric conversion element 61. The image data 69 is the signal charge stored in the photoelectric conversion element 61. The processing circuit 62 performs A / D conversion on the analog image data 69 read out from the photoelectric conversion element 61. The processing circuit 62 stores the digital image data 69 obtained by performing A / D conversion on the analog image data 69 in the memory 64.

[0077] The processing circuit 62 is connected to the signal processing unit 50 via the communication line 53. The processing circuit 62 is also connected to the second communication interface 46D2 of the controller 46 via the communication line 57.

[0078] Memory 54 includes multiple storage areas, including a first storage area 64A, a second storage area 64B, a third storage area 64C, a fourth storage area 64D, and a fifth storage area 64E. In each of the multiple storage areas, for example, digital image data 69 is stored pixel by pixel at an address corresponding to a pixel of the photoelectric conversion element 61 for each frame. Random access to the multiple storage areas is performed by the processing circuit 62.

[0079] As an example, as shown in Figure 6, on the imaging surface 44A of the photoelectric conversion element 61, R pixels, G pixels, and B pixels are arranged in a predetermined periodicity in both the row direction (horizontal direction) and the column direction (vertical direction). In this first embodiment, the R pixels, G pixels, and B pixels are arranged in a periodicity corresponding to the X-Trans® array. Although the example shown in Figure 6 illustrates the X-Trans array, the technology of this disclosure is not limited thereto, and the array of R pixels, G pixels, and B pixels may be a Bayer array, a honeycomb array, or the like.

[0080] In the example shown in Figure 6, in the first row, R pixels, G pixels, and B pixels are arranged in a cyclical manner in the row direction, in the order of G pixels, B pixels, R pixels, G pixels, R pixels, and B pixels. In the second row, R pixels, G pixels, and B pixels are arranged in a cyclical manner in the row direction, in the order of R pixels, G pixels, G pixels, B pixels, G pixels, and G pixels. In the third row, R pixels, G pixels, and B pixels are arranged in a cyclical manner in the row direction, in the order of B pixels, G pixels, G pixels, R pixels, G pixels, and G pixels. In the fourth row, R pixels, G pixels, and B pixels are arranged in a cyclical manner in the row direction, in the order of G pixels, R pixels, B pixels, G pixels, B pixels, and R pixels. In the fifth row, R pixels, G pixels, and B pixels are arranged in a cyclical manner in the row direction, in the order of B pixels, G pixels, G pixels, R pixels, G pixels, and G pixels. Furthermore, in the sixth row, the R pixels, G pixels, and B pixels are arranged in a cyclical manner in the row direction, in the order of R pixels, G pixels, G pixels, B pixels, G pixels, and G pixels. The arrangement pattern of the R pixels, G pixels, and B pixels in rows 1 to 6 is repeated in the column direction in units of 6 rows, thereby forming the overall arrangement pattern of the R pixels, G pixels, and B pixels of the photoelectric conversion element 61.

[0081] The photoelectric conversion element 61 is formed by two types of photosensitive pixels: phase difference pixels and non-phase difference pixels N, which are pixels different from phase difference pixels. Generally, non-phase difference pixels N are normal pixels. It is also referred to as [another term]. Multiple phase-difference pixel lines 61A and multiple non-phase-difference pixel lines 61B are arranged on the imaging surface 44A. Phase-difference pixel lines 61A are horizontal lines containing phase-difference pixels. Specifically, phase-difference pixel lines 61A are horizontal lines in which phase-difference pixels and non-phase-difference pixels N are mixed. Non-phase-difference pixel lines 61B are horizontal lines containing only multiple non-phase-difference pixels N, that is, horizontal lines consisting of multiple non-phase-difference pixels N. Note that phase-difference pixel lines 61A are an example of the "first line" relating to the technology of this disclosure, and non-phase-difference pixel lines 61B are an example of the "second line" relating to the technology of this disclosure.

[0082] As an example, as shown in Figure 6, the imaging surface 44A has phase-difference pixel lines 61A and a predetermined number of non-phase-difference pixel lines 61B arranged alternately along the column direction. Here, "determined number of lines" refers to, for example, 2 lines. Here, 2 lines are used as an example of the predetermined number of lines, but the technology of this disclosure is not limited to this, and the predetermined number of lines may be 3 or more lines, or a dozen or so lines, tens of lines, or hundreds of lines, etc. Also, in the example shown in Figure 6, the row direction is an example of the "line direction of the first line" according to the technology of this disclosure, and the column direction is an example of the "direction intersecting the line direction of the first line" according to the technology of this disclosure.

[0083] The phase difference pixel line 61A is arranged in a column direction with two rows skipped from the first row to the last row. Some of the pixels in the phase difference pixel line 61A are phase difference pixels. Specifically, the phase difference pixel line 61A is a horizontal line in which phase difference pixels and non-phase difference pixels N are arranged periodically. Phase difference pixels are broadly classified into first phase difference pixels L and second phase difference pixels R. In the phase difference pixel line 61A, first phase difference pixels L and second phase difference pixels R are arranged alternately as G pixels at intervals of several pixels in the line direction.

[0084] The first phase difference pixels L and the second phase difference pixels R are arranged to appear alternately in the column direction. In the example shown in Figure 6, in the fourth column, the first phase difference pixels L, second phase difference pixels R, first phase difference pixels L, and second phase difference pixels R are arranged in that order from the first row along the column direction. That is, the first phase difference pixels L and second phase difference pixels R are arranged alternately from the first row along the column direction. Also in the example shown in Figure 6, in the tenth column, the second phase difference pixels R, first phase difference pixels L, second phase difference pixels R, and first phase difference pixels L are arranged in that order from the first row along the column direction. That is, the second phase difference pixels R and first phase difference pixels L are arranged alternately from the first row along the column direction.

[0085] As an example, as shown in Figure 7, the first phase difference pixel L comprises a microlens 19, a light-shielding member 21A, and a photodiode PD. In the first phase difference pixel L, the light-shielding member 21A is positioned between the microlens 19 and the light-receiving surface of the photodiode PD. The left half in the row direction of the light-receiving surface of the photodiode PD (the left side when viewing the subject from the light-receiving surface (in other words, the right side when viewing the light-receiving surface from the subject)) is shielded by the light-shielding member 21A.

[0086] The second phase-difference pixel R comprises a microlens 19, a light-shielding member 21B, and a photodiode PD. In the second phase-difference pixel R, the light-shielding member 21B is positioned between the microlens 19 and the light-receiving surface of the photodiode PD. The right half of the photodiode PD's light-receiving surface in the row direction (the right side when viewing the subject from the light-receiving surface (in other words, the left side when viewing the light-receiving surface from the subject)) is shielded by the light-shielding member 21B.

[0087] The light beam passing through the exit pupil of the imaging lens 40 is broadly divided into left-region passing light 300L and right-region passing light 300R. Left-region passing light 300L refers to the left half of the light beam passing through the exit pupil of the imaging lens 40 when viewed from the phase difference pixel side towards the subject side, and right-region passing light 300R refers to the light beam passing through the exit pupil of the imaging lens 40 from the phase difference pixel side towards the subject side This refers to the light beam in the right half when viewing the image. The light beam passing through the exit pupil of the imaging lens 40 is split into left and right by the microlens 19 and light-shielding members 21A and 21B, which function as pupil division parts. The first phase difference pixel L receives the light 300L passing through the left region as subject light, and the second phase difference pixel R receives the light 300R passing through the right region as subject light. As a result, the image sensor 44 generates a first phase difference image corresponding to the subject image corresponding to the light 300L passing through the left region, and a second phase difference image corresponding to the subject image corresponding to the light 300R passing through the right region.

[0088] For the sake of clarity, in the following explanation, the first phase difference pixel L and the second phase difference pixel R will be referred to simply as "phase difference pixels" unless it is necessary to distinguish between them. Also, for the sake of clarity, in the following explanation, the light-shielding members 21A and 21B will be referred to simply as "light-shielding members" without any reference numerals unless it is necessary to distinguish between them.

[0089] As an example, as shown in Figure 8, non-phase difference pixels differ from phase difference pixels in that they do not have a light-shielding element. The photodiode PD of the non-phase difference pixel receives light passing through the left region 300L and light passing through the right region 300R as the subject light.

[0090] As an example, as shown in Figure 9, the processing circuit 62 includes a read circuit 62A, a digital processing circuit 62B, an image processing circuit 62C, an output circuit 62D, and a control circuit 62E. The read circuit 62A is an example of a "read unit (read circuit)" according to the technology of this disclosure. The output circuit 62D is an example of an "output unit (output circuit)" according to the technology of this disclosure.

[0091] The read circuit 62A comprises a first read circuit 62A1 and a second read circuit 62A2. The first read circuit 62A1 is an example of a "first read unit (first read circuit)" according to the technology of this disclosure, and the second read circuit 62A2 is an example of a "second read unit (second read circuit)" according to the technology of this disclosure.

[0092] The first readout circuit 62A1 and the second readout circuit 62A2 are each connected to the photoelectric conversion element 61, the digital processing circuit 62B, and the control circuit 62E. The memory 64 is connected to the control circuit 62E. The image processing circuit 62C is also connected to the control circuit 62E. The output circuit 62D is also connected to the control circuit 62E. The control circuit 62E is connected to the controller 46 via the communication line 57. The output circuit 62D is connected to the signal processing unit 50 via the communication line 53.

[0093] As an example, as shown in Figure 9, the image data 69 described above can be broadly divided into analog pixel data 69A and digital pixel data 69B. For the sake of explanation, in the following, when it is not necessary to distinguish between analog pixel data 69A and digital pixel data 69B, they will be referred to simply as "image data 69".

[0094] Analog pixel data 69A is broadly divided into analog phase-difference pixel data 69A1, which is the analog pixel data of phase-difference pixels, and analog non-phase-difference pixel data 69A2, which is the analog pixel data of non-phase-difference pixels N. Digital pixel data 69B is broadly divided into digital phase-difference pixel data 69B1 and digital non-phase-difference pixel data 69B2. Digital phase-difference pixel data 69B1 is the pixel data obtained by digitizing analog phase-difference pixel data 69A1. Digital non-phase-difference pixel data 69B2 is the pixel data obtained by digitizing analog non-phase-difference pixel data 69A2.

[0095] Analog pixel data 69A and digital pixel data 69B are examples of "pixel data" relating to the technology of this disclosure. Analog phase-difference pixel data 69A1 and digital phase-difference pixel data 69B1 are examples of "second pixel data" relating to the technology of this disclosure. Analog non-phase-difference pixel data 69A2 and digital non-phase-difference pixel data 69B2 are examples of the technology of this disclosure. This is an example of the "first pixel data" related to this.

[0096] The controller 46 supplies timing control signals to the control circuit 62E via the communication line 57. The timing control signals include an imaging vertical synchronization signal and an output vertical synchronization signal. The output vertical synchronization signal is a synchronization signal that defines the output timing in units of one frame. In other words, the output vertical synchronization signal is a synchronization signal that defines the output frame rate. That is, the output period (hereinafter simply referred to as the "output period") in which one frame of digital pixel data 69B is output is defined by the output vertical synchronization signal. The imaging vertical synchronization signal is a synchronization signal that defines the imaging timing in units of one frame. In other words, the imaging vertical synchronization signal is a synchronization signal that defines the imaging frame rate. That is, the interval in which imaging is performed is defined by the imaging vertical synchronization signal.

[0097] In this first embodiment, multiple vertical synchronization signals for imaging are supplied from the controller 46 to the control circuit 62E during the output period, thereby enabling imaging of multiple frames within the output period.

[0098] The readout circuit 62A reads out analog pixel data 69A obtained by imaging the subject at the imaging frame rate in horizontal line units. That is, under the control of the control circuit 62E, the readout circuit 62A controls the photoelectric conversion element 61 and reads out analog pixel data 69A from the photoelectric conversion element 61 in horizontal line units. The readout circuit 62A is a circuit capable of selectively reading out analog phase difference pixel data 69A1 and analog non-phase difference pixel data 69A2 from all pixels included in the photoelectric conversion element 61. As will be described in detail later, the selective reading out of analog phase difference pixel data 69A1 and analog non-phase difference pixel data 69A2 is realized by the first readout circuit 62A1 and the second readout circuit 62A2. Here, an example of a configuration in which analog phase difference pixel data 69A1 and analog non-phase difference pixel data 69A2 are selectively read out from all pixels included in the photoelectric conversion element 61 is given, but the technology of this disclosure is not limited to this. For example, analog phase-difference pixel data 69A1 and analog non-phase-difference pixel data 69A2 may be selectively read out from a specified group of pixels among all the pixels included in the photoelectric conversion element 61.

[0099] The readout of one frame's worth of analog pixel data 69A from the photoelectric conversion element 61 is initiated according to the imaging vertical synchronization signal. The control circuit 62E transfers the imaging vertical synchronization signal supplied from the controller 46 to the readout circuit 62A. When the reading circuit 62A receives the imaging vertical synchronization signal from the control circuit 62E, it starts reading one frame's worth of analog pixel data 69A from the photoelectric conversion element 61.

[0100] Specifically, the control circuit 62E transfers the imaging vertical synchronization signal supplied from the controller 46 to the first readout circuit 62A1 and the second readout circuit 62A2. When the imaging vertical synchronization signal is input to the first readout circuit 62A1 and the second readout circuit 62A2, the reading of analog phase difference pixel data 69A1 from the phase difference pixels and the reading of analog non-phase difference pixel data 69A2 from the non-phase difference pixels N are performed independently. The reading of analog phase difference pixel data 69A1 is performed by the first readout circuit 62A1, and the reading of analog non-phase difference pixel data 69A2 is performed by the second readout circuit 62A2.

[0101] More specifically, when the vertical synchronization signal for imaging is input from the control circuit 62E, the first readout circuit 62A1 reads out one frame's worth of analog phase difference pixel data from each phase difference pixel in each phase difference pixel line 61A to the photoelectric conversion element 61. When the readout start condition (described later) is satisfied, the second readout circuit 62A2 reads out one frame's worth of analog non-phase difference pixel data from each non-phase difference pixel N in each non-phase difference pixel line 61B to the photoelectric conversion element 61. The "readout start condition" here refers to when the vertical synchronization signal for imaging is input from the control circuit 62E. This refers to the condition that input has been received and that the reading of one frame's worth of analog phase difference pixel data by the first readout circuit 62A1 has been completed.

[0102] The readout circuit 62A performs analog signal processing on the analog pixel data 69A read from the photoelectric conversion element 61. Analog signal processing includes known processes such as noise cancellation and analog gain processing. Noise cancellation is a process that cancels noise caused by variations in the characteristics between pixels in the photoelectric conversion element 61. Analog gain processing is a process that applies a gain to the analog pixel data 69A. The analog pixel data 69A that has undergone analog signal processing in this manner is output to the digital processing circuit 62B by the readout circuit 62A.

[0103] The digital processing circuit 62B includes an A / D converter 62B1. The digital processing circuit 62B performs digital signal processing on the analog pixel data 69A input from the readout circuit 62A. Digital signal processing includes, for example, correlated double sampling, A / D conversion by the A / D converter 62B1, and digital gain processing.

[0104] Correlated double sampling is performed on the analog pixel data 69A by the digital processing circuit 62B. After the correlated double sampling signal processing is performed on the analog pixel data 69A, A / D conversion is performed by the A / D converter 62B1, thereby digitizing the analog pixel data 69A and obtaining digital pixel data 69B. Then, digital gain processing is performed on the digital pixel data 69B by the digital processing circuit 62B. Digital gain processing refers to the process of applying gain to the digital pixel data 69B.

[0105] The control circuit 62E acquires the digital pixel data 69B obtained by the digital signal processing from the digital processing circuit 62B and stores the acquired digital pixel data 69B in the memory 64.

[0106] Memory 64 is a memory capable of storing digital pixel data 69B for multiple frames. Memory 64 has multiple storage areas (see Figure 5), and the digital pixel data 69B is stored pixel by pixel in the corresponding storage area of ​​memory 64 by the control circuit 62E.

[0107] The control circuit 62E has random access to memory 64 and acquires digital non-phase-difference pixel data 69B2 from memory 64. The control circuit 62E outputs the digital non-phase-difference pixel data 69B2 acquired from memory 64 to the image processing circuit 62C. The image processing circuit 62C performs image processing on the digital non-phase-difference pixel data 69B2 input from the control circuit 62E. "Image processing" here refers to processes such as demosaicing, digital decimation, digital addition, and data embedding.

[0108] Demosaicing is the process of calculating all color information for each pixel from a mosaic image corresponding to an array of color filters. For example, in the case of an image sensor consisting of RGB color filters, all RGB color information is calculated for each pixel from a mosaic image consisting of RGB.

[0109] Digital decimation is the process of decimating pixels in the digital non-phase difference pixel data 69B2 on a line-by-line basis. "Line-by-line basis" refers, for example, to horizontal line units and / or vertical line units. Digital addition is the process of adding and averaging the pixel values ​​of multiple pixels in the digital non-phase difference pixel data 69B2. Data embedding is the process of embedding specific data into the lower-order empty bits of the digital non-phase difference pixel data 69B2. "Specific data" here refers to, for example, digital non-phase difference Examples include information that identifies the method of digital decimation performed on the pixel data 69B2, or a frame number that identifies the frame.

[0110] The image processing circuit 62C outputs the processed digital non-phase-difference pixel data 69B2 to the control circuit 62E. The control circuit 62E stores the digital non-phase-difference pixel data 69B2 input from the image processing circuit 62C in the memory 64. The processed digital non-phase-difference pixel data 69B2 is an example of the "image data based on pixel data" and "first pixel data based on phase-difference pixel data" related to the technology of this disclosure.

[0111] The control circuit 62E acquires digital pixel data 69B from memory 64. The control circuit 62E outputs the digital pixel data 69B acquired from memory 64 to the output circuit 62D.

[0112] Specifically, whenever one frame's worth of digital phase-difference pixel data 69B1 is stored in memory 64, the control circuit 62E retrieves one frame's worth of digital phase-difference pixel data 69B1 from memory 64 without waiting for digital non-phase-difference pixel data to be stored in memory 64. Then, the control circuit 62E outputs the latest frame's worth of digital phase-difference pixel data 69B1 retrieved from memory 64 to the output circuit 62D.

[0113] When the control circuit 62E has finished acquiring the digital phase difference pixel data 69B1 corresponding to one frame of analog phase difference pixel data 69A1 read by the first read circuit 62A1 within the output period, it acquires digital non-phase difference pixel data 69B2 from memory 64. The digital non-phase difference pixel data 69B2 acquired from memory 64 is the digital non-phase difference pixel data 69B2 corresponding to one frame of analog non-phase difference pixel data 69A2 read by the second read circuit 62A2 within the output period. Then, the control circuit 62E outputs the digital non-phase difference pixel data 69B2 acquired from memory 64 to output circuit 62D.

[0114] When outputting digital pixel data 69B for one frame, the output circuit 62D outputs digital phase-difference pixel data 69B1 and digital non-phase-difference pixel data 69B2 at different timings. Specifically, the output circuit 62D first outputs the digital phase-difference pixel data 69B1, and only after the output of the digital phase-difference pixel data 69B1 is complete does it output the digital non-phase-difference pixel data 69B2.

[0115] The digital phase-difference pixel data 69B1 for all frames read within the output period is output to the output circuit 62D by the control circuit 62E before the digital non-phase-difference pixel data 69B2. The output circuit 62D outputs the digital pixel data 69B input from the control circuit 62E to the signal processing unit 50 in the order in which it was input. First, the output circuit 62D outputs the digital phase-difference pixel data 69B1 for all frames read within the output period to the signal processing unit 50, and then outputs the digital non-phase-difference pixel data 69B2 for all frames read within the output period to the signal processing unit 50.

[0116] Incidentally, imaging is performed using a rolling shutter method in the imaging device 10. Therefore, as shown in Figure 10 as an example, there is a discrepancy between the readout start timing and the readout end timing. The readout start timing is the timing at which the first horizontal line to be read out of all the horizontal lines in the imaging area of ​​the image sensor 44 (hereinafter also referred to as the "first horizontal line") is read out. The readout end timing is the timing at which the last horizontal line to be read out of all the horizontal lines in the imaging area of ​​the image sensor 44 (hereinafter also referred to as the "last horizontal line") is read out. In the rolling shutter method, imaging of the image sensor 44 is performed according to the horizontal synchronization signal. For all horizontal lines in the region, analog pixel data 69A is read out sequentially, one horizontal line at a time. Therefore, the readout completion timing occurs later than the readout start timing. Consequently, a discrepancy arises between the timing of exposure for the first horizontal line and the timing of exposure for the last horizontal line.

[0117] Figure 10 shows examples of readout and reset timings for four frames within a single output period. In the example shown in Figure 10, "F" is an abbreviation for "frame". In the example shown in Figure 10, imaging for four frames is performed within a single output period. That is, the readout of multiple analog pixel data 69A is performed in parallel within a single output period. For example, of the readouts from the first to the fourth frame within a single output period, the readout of the first frame and the readout of the second frame are performed in parallel. Consequently, the reset of the first frame and the reset of the second frame are also performed in parallel. In other words, the readout period of the first frame and the readout period of the second frame partially overlap, and the reset period of the first frame and the reset period of the second frame also partially overlap. The same can be said for the second and third frames, and also for the third and fourth frames.

[0118] Thus, within a single output period, four frames of analog non-phase difference pixel data 69A2 and four frames of analog phase difference pixel data 69A1 are read out. In other words, within a single output period, in addition to the reading of analog non-phase difference pixel data 69A2, multiple readings of analog phase difference pixel data 69A1 are performed. Here, "reading of analog non-phase difference pixel data 69A2" refers to the reading of analog non-phase difference pixel data 69A2 from a non-phase difference pixel N. Also, "multiple readings of analog phase difference pixel data 69A1" refers to multiple readings of analog phase difference pixel data 69A1 from a phase difference pixel. In other words, this means that within a single output period, analog phase difference pixel data 69A1 is read out multiple times from the same phase difference pixel.

[0119] As an example, as shown in Figure 10, in each frame, first, the analog phase difference pixel data 69A1 is read from the photoelectric conversion element 61 by the first readout circuit 62A1. That is, all phase difference pixel lines 61A for one frame are to be read out, and the analog phase difference pixel data 69A1 is read out line by line by the first readout circuit 62A1. After the first readout circuit 62A1 has finished reading out a phase difference pixel line 61A, a reset is performed by the first readout circuit 62A1.

[0120] Once the reading of analog phase difference pixel data 69A1 for all phase difference pixel lines 61A is complete, the second readout circuit 62A2 then reads out analog non-phase difference pixel data 69A2 line by line, targeting all non-phase difference pixel lines 61B. After each non-phase difference pixel line 61B has been read out by the second readout circuit 62A2, a reset is performed by the second readout circuit 62A2.

[0121] Thus, during the readout period for one frame, the reading of analog phase difference pixel data 69A1 from phase difference pixels by the first readout circuit 62A1 is performed before the reading of analog non-phase difference pixel data 69A2 from non-phase difference pixels N by the second readout circuit 62A2.

[0122] Furthermore, in the example shown in Figure 10, the reading of the analog non-phase-difference pixel data 69A2 for the first frame and the reading of the analog phase-difference pixel data 69A1 for the second frame are performed in parallel. In other words, the reading period for the analog non-phase-difference pixel data 69A2 for the first frame and the reading period for the analog phase-difference pixel data 69A1 for the second frame overlap. The same can be said for the second and third frames, and also for the third and fourth frames.

[0123] As an example, as shown in Figure 11, the analog phase difference pixel data 69A1 read out by the first readout circuit 62A1 for each frame within one output period is digitized by the A / D converter 62B1 and converted into digital phase difference pixel data 69B1. Once the digitization of all the analog phase difference pixel data 69A1 read out within one output period is complete, A / D conversion is performed on the analog non-phase difference pixel data 69A2 read out by the second readout circuit 62A2 for each frame within one output period. That is, the analog non-phase difference pixel data 69A2 read out by the second readout circuit 62A2 for each frame is digitized by the A / D converter 62B1 and converted into digital non-phase difference pixel data 69B2.

[0124] Thus, the analog phase-difference pixel data 69A1 and the analog non-phase-difference pixel data 69A2 are converted by the A / D converter 62B1 at different timings. This is because the A / D converter 62B1 is shared between the analog phase-difference pixel data 69A1 and the analog non-phase-difference pixel data 69A2.

[0125] As an example, as shown in Figure 11, each digital phase difference pixel data 69B1 obtained by A / D conversion for each frame is stored in the corresponding memory area of ​​memory 64. In the example shown in Figure 11, the digital phase difference pixel data 69B1 for the first frame is stored in the first memory area 64A. The digital phase difference pixel data 69B1 for the second frame is stored in the second memory area 64B. The digital phase difference pixel data 69B1 for the third frame is stored in the third memory area 64C. The digital phase difference pixel data 69B1 for the fourth frame is stored in the fourth memory area 64D.

[0126] The digital non-phase-difference pixel data 69B2 obtained by A / D conversion for each frame is combined into one frame's worth and stored in the fifth memory area 64E. Here, "combination" refers to the pixel-level averaging of the digital non-phase-difference pixel data 69B2 of multiple frames (frames 1 to 4 in the example shown in Figure 11) obtained by imaging within one output period. In this way, the digital non-phase-difference pixel data 69B2 of multiple frames is averaged pixel by pixel to generate one frame's worth of digital non-phase-difference pixel data 69B2.

[0127] Here, we have provided an example in which digital non-phase difference pixel data 69B2 for multiple frames is combined into one frame, but the technology of this disclosure is not limited to this. For example, all of the digital non-phase difference pixel data 69B2 for multiple frames may be stored in memory 64. Alternatively, the analog non-phase difference pixel data 69A2 may be read out by the second readout circuit 62A2 only from one representative frame among the multiple frames (for example, the first frame). As will be explained in detail later as an example of a modified version, in this case, the analog non-phase difference pixel data 69A2 is not read out by the second readout circuit 62A2 from the other frames (for example, the second to fourth frames) (see Figure 16). Therefore, only the analog non-phase difference pixel data 69A2 for one frame is A / D converted, and the "combination" described above becomes unnecessary.

[0128] When digital phase difference pixel data 69B1 is stored in memory 64 on a frame-by-frame basis, the digital phase difference pixel data 69B1 in memory 64 is transferred to output circuit 62D when the storage of one frame's worth of digital phase difference pixel data 69B1 in memory 64 is complete. In other words, the digital phase difference pixel data 69B1 in memory 64 is retrieved from memory 64 by the control circuit 62E and output to output circuit 62D. Output circuit 62D is controlled by the control circuit The digital phase difference pixel data 69B1 input from path 62E is output to the signal processing unit 50.

[0129] After the digital phase-difference pixel data 69B1 in memory 64 is output by the output circuit 62D, the digital non-phase-difference pixel data 69B2 in memory 64 is retrieved from memory 64 by the control circuit 62E and output to the output circuit 62D. The output circuit 62D outputs the digital non-phase-difference pixel data 69B2 input from the control circuit 62E to the signal processing unit 50.

[0130] Next, the operation of the imaging device 10 will be explained.

[0131] First, the timing control process executed by the CPU 46A according to the timing control program 46B1 when the conditions for starting the timing control process are met will be explained with reference to Figure 12. For example, the condition for starting the timing control process is that the imaging mode has been set.

[0132] In the timing control process shown in Figure 12, first, in step ST10, the CPU 46A determines whether or not the read start timing has arrived. The read start timing refers to the timing at which the image sensor 44 is instructed to start reading out the analog pixel data 69A from the photoelectric conversion element 61. If the read start timing has not arrived in step ST10, the determination is denied, and the timing control process proceeds to step ST18. If the read start timing has arrived in step ST10, the determination is affirmed, and the timing control process proceeds to step ST12.

[0133] In step ST12, the CPU 46A outputs a vertical synchronization signal for imaging to the image sensor 44, and then the timing control process proceeds to step ST14. The vertical synchronization signal for imaging output by the execution of step ST12 is received by the control circuit 62E of the image sensor 44. The image sensor 44 performs imaging at an imaging frame rate determined according to the vertical synchronization signal for imaging received by the control circuit 62E.

[0134] In step ST14, the CPU 46A determines whether the number of times the vertical synchronization signal for imaging has been output has reached a predetermined number (for example, 4 times). If the number of times the vertical synchronization signal for imaging has been output has not reached the predetermined number in step ST14, the determination is denied, and the timing control process proceeds to step ST18. If the number of times the vertical synchronization signal for imaging has been output has reached the predetermined number in step ST14, the determination is affirmed, and the timing control process proceeds to step ST16.

[0135] In step ST16, the CPU 46A outputs a vertical sync signal for output to the image sensor 44, and then the timing control process proceeds to step ST18. The vertical sync signal for output output as a result of the execution of step ST16 is received by the control circuit 62E of the image sensor 44. The image sensor 44 outputs digital pixel data 69B at an output frame rate determined according to the vertical sync signal for output received by the control circuit 62E via the output circuit 62D.

[0136] In step ST18, the CPU 46A determines whether the conditions for terminating the timing control process (hereinafter referred to as "timing control process termination conditions") have been met. For example, one of the timing control process termination conditions is that an instruction to terminate the timing control process has been received by the receiving device 84 (see Figure 4). If the timing control process termination conditions are not met in step ST18, the determination is denied, and the timing control process proceeds to step ST10. If the timing control process termination conditions are met in step ST18, the determination is affirmed, and the timing control process terminates.

[0137] Next, the phase difference pixel processing performed by the processing circuit 62 during the output period will be explained with reference to Figure 13.

[0138] In the phase difference pixel processing shown in Figure 13, first, in step ST30, the control circuit 62E determines whether or not it has received the imaging vertical synchronization signal output by the processing in step ST12 of the timing control process described above. If the imaging vertical synchronization signal has not been received in step ST30, the determination is denied, and the phase difference pixel processing proceeds to step ST40. If the imaging vertical synchronization signal has been received in step ST30, the determination is affirmed, and the phase difference pixel processing proceeds to step ST32.

[0139] In step ST32, the first readout circuit 62A1 reads analog phase difference pixel data 69A1 from the phase difference pixels for all phase difference pixel lines 61A for one frame, and then the phase difference pixel processing proceeds to step ST34.

[0140] In step ST34, the digital processing circuit 62B performs digital signal processing on the analog phase difference pixel data 69A1 read out by the first readout circuit 62A1, thereby converting the analog phase difference pixel data 69A1 into digital phase difference pixel data 69B1.

[0141] In the next step, ST36, the control circuit 62E acquires digital phase difference pixel data 69B1 from the digital processing circuit 62B, stores the acquired digital phase difference pixel data 69B1 in the memory 64, and then the phase difference pixel processing proceeds to step ST38.

[0142] When digital phase difference pixel data 69B1 is stored in memory 64, the control circuit 62E retrieves the digital phase difference pixel data 69B1 from memory 64, and the retrieved digital phase difference pixel data 69B1 is transferred to output circuit 62D.

[0143] In the next step, ST38, the output circuit 62D outputs the digital phase difference pixel data 69B1 input from the control circuit 62E to the signal processing unit 50, and then the phase difference pixel processing proceeds to step ST40.

[0144] In step ST40, the control circuit 62E determines whether the conditions for terminating the phase difference pixel processing (hereinafter referred to as the "phase difference pixel processing termination conditions") have been met. For example, the phase difference pixel processing termination conditions include the condition that an instruction to terminate the phase difference pixel processing has been received by the receiving device 84 (see Figure 4). If the phase difference pixel processing termination conditions are not met in step ST40, the determination is denied, and the phase difference pixel processing proceeds to step ST30. If the phase difference pixel processing termination conditions are met in step ST40, the determination is affirmed, and the phase difference pixel processing is terminated.

[0145] Next, the non-phase difference pixel processing performed by the processing circuit 62 during the output period will be explained with reference to Figure 14.

[0146] In the non-phase difference pixel processing shown in Figure 14, first, in step ST60, it is determined whether the reading of one frame's worth of analog phase difference pixel data 69A1 by the first reading circuit 62A1 has finished. If, in step ST60, the reading of one frame's worth of analog phase difference pixel data 69A1 by the first reading circuit 62A1 has not finished, the determination is denied, and the non-phase difference pixel processing proceeds to step ST72. If, in step ST60, the reading of one frame's worth of analog phase difference pixel data 69A1 by the first reading circuit 62A1 has finished, the determination is affirmed, and the non-phase difference pixel processing proceeds to step ST62.

[0147] In step ST62, the second readout circuit 62A2 reads out analog non-phase difference pixel data 69A2 from non-phase difference pixels N for all non-phase difference pixel lines 61B for one frame, and then the non-phase difference pixel processing proceeds to step ST64.

[0148] In step ST64, the digital processing circuit 62B performs digital signal processing on the analog non-phase difference pixel data 69A2 read out by the second readout circuit 62A2, thereby converting the analog non-phase difference pixel data 69A2 into digital non-phase difference pixel data 69B2.

[0149] In the next step, ST66, the control circuit 62E acquires digital non-phase difference pixel data 69B2 from the digital processing circuit 62B, stores the acquired digital non-phase difference pixel data 69B2 in the memory 64, and then the non-phase difference pixel processing proceeds to step ST68. The memory 64 stores the digital non-phase difference pixel data 69B2 that has been processed by the image processing circuit 62C.

[0150] In step ST68, the control circuit 62E determines whether the non-phase difference pixel data output timing has arrived. The non-phase difference pixel data output timing refers to the timing at which the digital non-phase difference pixel data 69B2 in memory 64 is output to the signal processing unit 50. The non-phase difference pixel data output timing does not need to overlap with the timing at which the digital phase difference pixel data 69B1 for all frames within the same output period is output to the signal processing unit 50. An example of a non-phase difference pixel data output timing is the timing at which the output of the digital phase difference pixel data 69B1 for all frames within the same output period to the signal processing unit 50 has been completed.

[0151] In step ST68, if the non-phase difference pixel data output timing has not yet arrived, the determination is denied, and the determination in step ST68 is repeated. In step ST68, if the non-phase difference pixel data output timing has arrived, the determination is affirmed, and the non-phase difference pixel processing proceeds to step ST70.

[0152] In step ST70, the control circuit 62E acquires digital non-phase difference pixel data 69B2 from memory 64 and transfers the acquired digital non-phase difference pixel data 69B2 to the output circuit 62D. The output circuit 62D outputs the digital non-phase difference pixel data 69B2 input from the control circuit 62E to the signal processing unit 50, and then the non-phase difference pixel processing proceeds to step ST72.

[0153] In step ST72, the control circuit 62E determines whether the conditions for terminating non-phase difference pixel processing (hereinafter referred to as "non-phase difference pixel processing termination conditions") have been met. For example, one of the non-phase difference pixel processing termination conditions is that an instruction to terminate non-phase difference pixel processing has been received by the receiving device 84 (see Figure 4). If the non-phase difference pixel processing termination conditions are not met in step ST72, the determination is denied, and the non-phase difference pixel processing proceeds to step ST60. If the non-phase difference pixel processing termination conditions are met in step ST72, the determination is affirmed, and the non-phase difference pixel processing is terminated.

[0154] In the signal processing unit 50, various signal processing is performed on the digital pixel data 69B input from the image sensor 44, and the digital pixel data 69B that has undergone various signal processing is output to the controller 46.

[0155] Next, when digital pixel data 69B is input from the signal processing unit 50 to the controller 46, the pixel data processing performed by the CPU 46A according to the pixel data processing program 46B2 will be explained with reference to Figure 15.

[0156] In the pixel data processing shown in Figure 15, first, in step ST150, the CPU 46A determines whether the digital pixel data 69B input from the signal processing unit 50 is digital non-phase difference pixel data 69B2. If the digital pixel data 69B input from the signal processing unit 50 is digital phase difference pixel data 69B1 in step ST150, the determination is denied, and the pixel data processing proceeds to step ST154. If the digital pixel data 69B input from the signal processing unit 50 is digital non-phase difference pixel data 69B2 in step ST150, the determination is affirmed, and the pixel data processing proceeds to step ST152.

[0157] In step ST152, the CPU 46A outputs digital non-phase difference pixel data 69B2 to the display 32, and then the pixel data processing proceeds to step ST156.

[0158] When the digital non-phase-difference pixel data 69B2 is output to the display 32, the display 32 displays an image based on the digital non-phase-difference pixel data 69B2.

[0159] In step ST154, the CPU 46A performs AF control using the digital phase-difference pixel data 69B1 input from the signal processing unit 50, and then the pixel data processing proceeds to step ST156.

[0160] In step ST156, the CPU 46A determines whether the conditions for terminating pixel data processing (hereinafter referred to as "pixel data processing termination conditions") have been met. For example, one of the pixel data processing termination conditions is that an instruction to terminate pixel data processing has been received by the receiving device 84 (see Figure 4). If the pixel data processing termination conditions are not met in step ST156, the determination is denied, and pixel data processing proceeds to step ST150. If the pixel data processing termination conditions are met in step ST156, the determination is affirmed, and pixel data processing is terminated.

[0161] As described above, in the imaging device 10 according to this first embodiment, within the output period, the readout circuit 62A reads out the analog pixel data 69A of each of the multiple frames in parallel. Also within the output period, the readout circuit 62A reads out the analog non-phase difference pixel data 69A2 and reads out the analog phase difference pixel data 69A1 multiple times.

[0162] By the way, if, for example, pixel data is read multiple times from an AF-dedicated pixel within the readout period for reading out one frame's worth of analog pixel data 69A, then an A / D converter is required for each AF-dedicated pixel, which complicates the image sensor configuration.

[0163] In contrast, the configuration of the image sensor 44 according to this first embodiment is simpler than the configuration of an image sensor in which an A / D converter is mounted on each AF-dedicated pixel (e.g., a phase-difference pixel), because it does not have an A / D converter mounted on each phase-difference pixel. Moreover, within the output period, the analog pixel data 69A of each of the multiple frames are read out in parallel, and the analog non-phase-difference pixel data 69A2 is read out, and the analog phase-difference pixel data 69A1 is read out multiple times.

[0164] Multiple digital phase-difference pixel data 69B1 obtained by digitizing multiple analog phase-difference pixel data 69A1 are used for AF control. It is clear that multiple digital phase-difference pixel data 69B1 contribute to higher AF accuracy compared to a single digital phase-difference pixel data 69B1. Therefore, according to the image sensor 44 of this first embodiment, AF accuracy can be improved with a simpler configuration compared to the case where pixel data is read multiple times from AF-dedicated pixels within the readout period for reading out one frame of image.

[0165] Furthermore, in the image sensor 44 according to this first embodiment, the readout circuit 62A performs the readout of analog non-phase difference pixel data 69A2 and the readout of analog phase difference pixel data 69A1 in parallel. Therefore, compared to the case where the readout of one of the analog phase difference pixel data 69A1 and analog non-phase difference pixel data 69A2 is completed before the readout of the other is started, a larger amount of analog phase difference pixel data 69A1 and analog non-phase difference pixel data 69A2 can be read out within a limited time.

[0166] Furthermore, in the image sensor 44 according to this first embodiment, the analog pixel data 69A is read out by the readout circuit 62A in units of horizontal lines. Therefore, decimation in units of horizontal lines can be achieved.

[0167] Furthermore, in the image sensor 44 according to this first embodiment, the pixel data of a phase difference pixel is used as the analog phase difference pixel data 69A1, and the pixel data of a non-phase difference pixel N is used as the analog non-phase difference pixel data 69A2. Therefore, compared to the case where the analog phase difference pixel data 69A1 is generated from the pixel data of pixels other than phase difference pixels, and the analog non-phase difference pixel data 69A2 is generated from the pixel data of pixels other than non-phase difference pixels, the analog phase difference pixel data 69A1 and the analog non-phase difference pixel data 69A2 can be easily obtained.

[0168] Furthermore, in the image sensor 44 according to this first embodiment, within the output period, analog non-phase difference pixel data 69A2 is read from non-phase difference pixels N, and analog phase difference pixel data 69A1 is read from phase difference pixels multiple times. Therefore, compared to the case where the reading of analog non-phase difference pixel data 69A2 from non-phase difference pixels N and the reading of analog phase difference pixel data 69A1 from phase difference pixels are performed alternately once each during the output period, it is possible to obtain analog non-phase difference pixel data 69A2 and multiple analog phase difference pixel data 69A1 within the output period.

[0169] Furthermore, in the image sensor 44 according to this first embodiment, the first readout circuit 62A1 reads out analog phase difference pixel data 69A1 from each of the phase difference pixels included in the plurality of phase difference pixel lines 61A. In addition, the second readout circuit 62A2 reads out analog non-phase difference pixel data 69A2 from each of the non-phase difference pixels N included in the plurality of non-phase difference pixel lines 61B. Therefore, the load required for reading out the analog phase difference pixel data 69A1 and the load required for reading out the analog non-phase difference pixel data 69A2 can be distributed.

[0170] Furthermore, in the image sensor 44 according to this first embodiment, the reading of analog phase-difference pixel data 69A1 by the first readout circuit 62A1 and the reading of analog non-phase-difference pixel data 69A2 by the second readout circuit 62A2 are performed independently. Therefore, it is possible to avoid one of the readings of analog phase-difference pixel data 69A1 and the other affecting the other.

[0171] Furthermore, in the image sensor 44 according to this first embodiment, the reading of analog phase-difference pixel data 69A1 from phase-difference pixels by the first readout circuit 62A1 is performed before the reading of analog non-phase-difference pixel data 69A2 from non-phase-difference pixels N by the second readout circuit 62A2. Therefore, compared to the case where the reading of analog non-phase-difference pixel data 69A2 is performed before the reading of analog phase-difference pixel data 69A1, the digital phase-difference pixel data 69B1 can be provided for AF control earlier.

[0172] Furthermore, in the image sensor 44 according to this first embodiment, a horizontal line in which phase-difference pixels and non-phase-difference pixels N are periodically arranged is used as the phase-difference pixel line 61A. Therefore, Compared to using a horizontal line in which phase-detection pixels and non-phase-detection pixels N are locally concentrated, this method can improve the accuracy of autofocus over a wide area.

[0173] Furthermore, in the image sensor 44 according to this first embodiment, phase-difference pixel lines 61A and a predetermined number of non-phase-difference pixel lines 61B are alternately arranged along the column direction on the imaging surface 44A. Therefore, compared to the case where the phase-difference pixel lines 61A and non-phase-difference pixel lines 61B are locally concentrated in the column direction, the accuracy of AF over a wide area can be improved.

[0174] Furthermore, in the image sensor 44 according to this first embodiment, when digital pixel data 69B for one frame is output by the output circuit 62D, the digital phase-difference pixel data 69B1 and the digital non-phase-difference pixel data 69B2 are output at different timings. Therefore, compared to the case where a dedicated output circuit is used for each of the digital phase-difference pixel data 69B1 and the digital non-phase-difference pixel data 69B2, it is possible to contribute to miniaturization of the image sensor 44.

[0175] Furthermore, in the image sensor 44 according to this first embodiment, the digital non-phase-detection pixel data 69B2 is output only after the output of the digital phase-detection pixel data 69B1 is complete. Therefore, compared to the case where the digital phase-detection pixel data 69B1 is output only after the output of the digital non-phase-detection pixel data 69B2 is complete, the digital phase-detection pixel data 69B1 can be made available for AF control sooner.

[0176] Furthermore, in the image sensor 44 according to this first embodiment, A / D conversion is performed at different timings for the analog phase-difference pixel data 69A1 and the analog non-phase-difference pixel data 69A2. Therefore, compared to the case where a dedicated A / D converter is used for each of the analog phase-difference pixel data 69A1 and the analog non-phase-difference pixel data 69A2, it is possible to contribute to miniaturization of the image sensor 44.

[0177] Furthermore, the image sensor 44 according to this first embodiment is an image sensor in which the photoelectric conversion element 61, processing circuit 62, and memory 64 are integrated on a single chip. This makes the image sensor 44 more portable than an image sensor in which the photoelectric conversion element 61, processing circuit 62, and memory 64 are not integrated on a single chip. It also allows for greater design flexibility compared to an image sensor in which the photoelectric conversion element 61, processing circuit 62, and memory 64 are not integrated on a single chip. Moreover, it contributes to miniaturization of the imaging device body 12 compared to an image sensor in which the photoelectric conversion element 61, processing circuit 62, and memory 64 are not integrated on a single chip.

[0178] Furthermore, as shown in Figure 5, a stacked image sensor 44 is employed, in which a memory 64 is stacked on a photoelectric conversion element 61. This allows for shorter wiring between the photoelectric conversion element 61 and the memory 64, thereby reducing wiring delay. As a result, the transfer speed of image data 69 from the photoelectric conversion element 61 to the memory 64 can be increased compared to a case where the photoelectric conversion element 61 and the memory 64 are not stacked. This improvement in transfer speed also contributes to faster processing of the entire processing circuit 62. In addition, the degree of design flexibility can be increased compared to a case where the photoelectric conversion element 61 and the memory 64 are not stacked. Moreover, it can contribute to miniaturization of the imaging device body 12 compared to a case where the photoelectric conversion element 61 and the memory 64 are not stacked.

[0179] Furthermore, the imaging device 10 displays live view images and the like based on the digital pixel data 69B on the display 32. The digital pixel data 69B is also stored in the secondary storage device 80. Therefore, the versatility of the digital pixel data 69B can be increased.

[0180] In the first embodiment described above, the image sensor 44 is a photoelectric conversion element 61 and a processing circuit 62 Although an image sensor in which the photoelectric conversion element 61 and memory 64 are integrated into a single chip has been given as an example, the technology of this disclosure is not limited thereto. For example, it is sufficient if at least the photoelectric conversion element 61 and memory 64 among the photoelectric conversion element 61, processing circuit 62, and memory 64 are integrated into a single chip.

[0181] Furthermore, in the first embodiment described above, the digital non-phase difference pixel data 69B2 obtained by A / D conversion is processed by the image processing circuit 62C, and the processed digital non-phase difference pixel data 69B2 is the output target. However, the technology of this disclosure is not limited to this. The digital non-phase difference pixel data 69B2 obtained by A / D conversion may be output by the output circuit 62D without performing image processing on the digital non-phase difference pixel data 69B2. In this case, instead of the image processing circuit 62C, the signal processing unit 50 and / or controller 46, which are downstream circuits of the image sensor 44, may perform image processing on the digital non-phase difference pixel data 69B2.

[0182] Furthermore, in the first embodiment described above, no image processing is performed on the digital phase difference pixel data 69B1 by the image processing circuit 62C, but the technology of this disclosure is not limited thereto. For example, the digital phase difference pixel data 69B1 may be processed by the image processing circuit 62C. In this case, the digital phase difference pixel data 69B1 that has been processed by the image processing circuit 62C is output to the signal processing unit 50 by the output circuit 62D. The digital non-phase difference pixel data 69B2 that has been processed by the image processing circuit 62C is an example of "image data" and "second pixel data based on phase difference pixel data" related to the technology of this disclosure.

[0183] Furthermore, although the first embodiment described above included an example of a configuration in which communication between the image sensor 44 and the signal processing unit 50 is performed in a wired manner via a communication line 53, the technology of this disclosure is not limited thereto. For example, communication between the image sensor 44 and the signal processing unit 50 may be performed in a wireless manner. Similarly, communication between the image sensor 44 and the controller 46 may be performed in a wireless manner, or communication between the signal processing unit 50 and the controller 46 may be performed in a wireless manner.

[0184] Furthermore, although the first embodiment described above included an example in which analog phase difference pixel data 69A1 and analog non-phase difference pixel data 69A2 are read for all frames within the output period, the technology of this disclosure is not limited thereto. For example, as shown in Figure 16, analog phase difference pixel data 69A1 and analog non-phase difference pixel data 69A2 may be read for only one representative frame (the first frame in the example shown in Figure 16) among all frames within the output period. In this case, only analog phase difference pixel data 69A1 may be read for each of the other frames (frames 2 through 4 in the example shown in Figure 16). As a result, analog non-phase difference pixel data 69A2 is not read for frames other than the representative frame, and power consumption can be reduced compared to the case in which analog phase difference pixel data 69A1 and analog non-phase difference pixel data 69A2 are read for each of all frames.

[0185] Furthermore, in the first embodiment described above, an example was given in which the digital phase difference pixel data 69B1 is output to the signal processing unit 50 each time the digital phase difference pixel data 69B1 of the next frame is stored in the memory 64, without waiting for the digital phase difference pixel data 69B1 of the next frame to be stored in the memory 64. However, the technology of this disclosure is not limited to this. For example, as shown in Figure 17, the digital phase difference pixel data 69B1 of all frames may be stored in the memory 64 within one output period before the digital phase difference pixel data 69B1 of all frames is output to the signal processing unit 50 by the output circuit 62D. In this case, the digital non-phase difference pixel data 69B2 is output by the output circuit 62D after waiting for the output of the digital phase difference pixel data 69B1 of all frames to the signal processing unit 50 to be completed. The output should be directed to the processing unit 50. This makes it possible to make the output timing of the digital phase-difference pixel data 69B1 and the output timing of the digital non-phase-difference pixel data 69B2 different.

[0186] Furthermore, in the first embodiment described above, an example was given in which the digital phase difference pixel data 69B1 for each frame is output to the signal processing unit 50 by the output circuit 62D, but the technology of this disclosure is not limited thereto. For example, pixel data based on statistical values ​​of a plurality of digital phase difference pixel data 69B1 obtained for each frame may be output to the signal processing unit 50 by the output circuit 62D. This reduces the amount of data output from the image sensor 44 to the signal processing unit 50 compared to the case in which a plurality of digital phase difference pixel data 69B1 obtained for each frame is output to the signal processing unit 50.

[0187] The above statistical value is the summation average of multiple digital phase difference pixel data 69B1 obtained for each frame within the output period. Here, for example, by calculating the summation average of the pixels corresponding to the positions of the digital phase difference pixel data 69B1 of multiple frames, the digital phase difference pixel data 69B1 of multiple frames is combined into the digital phase difference pixel data 69B1 of one frame. This reduces the amount of data output from the image sensor 44 to the signal processing unit 50 compared to outputting multiple digital phase difference pixel data 69B1 obtained for each frame to the signal processing unit 50. In addition to the summation average, other examples of the above statistical value include the median or mode.

[0188] [Second Embodiment] In the first embodiment described above, an example of how analog phase-difference pixel data 69A1 used for AF control and analog non-phase-difference pixel data 69A2 used for purposes other than AF control are read out was given. In this second embodiment, however, a case in which analog phase-difference pixel data 69A1 is also used for purposes other than AF control will be described. In this second embodiment, the same reference numerals are used for components that are the same as in the first embodiment, and their descriptions are omitted. The following will mainly describe the parts that differ from the first embodiment.

[0189] As an example, as shown in Figure 4, the imaging device 10 according to this second embodiment differs from the imaging device 10 according to the first embodiment in that the ROM 46B of the controller 46 stores a pixel data processing program 46B3 instead of the pixel data processing program 46B2.

[0190] As an example, as shown in Figure 19, in the imaging device 10 according to this second embodiment, when the receiving device 84 (see Figure 4) receives an instruction to set the operating mode of the imaging device 10 to continuous shooting mode, the controller 46 supplies a continuous shooting instruction signal to the control circuit 62E via the communication line 57. The continuous shooting instruction signal refers to a signal that instructs the image sensor 44 to take images in continuous shooting mode.

[0191] When the imaging device 10 is set to continuous shooting mode, as shown in Figure 20 as an example, in each output period, all analog pixel data 69A for one representative frame (the first frame in the example shown in Figure 20) is read out as recording pixel data by the second readout circuit 62A2. In the example shown in Figure 20, the second readout circuit 62A2 reads out the analog pixel data 69A for the first frame sequentially from the first row to the last row of the photoelectric conversion element 61. That is, analog pixel data 69A is read out for all horizontal lines, including the phase difference pixel lines 61A and the non-phase difference pixel lines 61B.

[0192] For the sake of explanation, we will assume that all horizontal lines are read out in this example. Although examples are provided for explanation, the analog pixel data 69A may be read out by downsampling the phase-difference pixels and non-phase-difference pixels N in units of several rows and / or several columns.

[0193] In the example shown in Figure 20, within a single output period, from the second frame onward, the first readout circuit 62A1 reads out the analog phase difference pixel data 69A1, and the analog non-phase difference pixel data 69A2 is not read out.

[0194] Furthermore, as shown in Figure 20 as an example, in the image sensor 44, during the readout period of analog pixel data 69A for one frame, the first readout circuit 62A1 reads out analog phase-difference pixel data 69A1 for other frames. Here, "analog pixel data 69A for one frame" includes analog phase-difference pixel data 69A1 and analog non-phase-difference pixel data 69A2.

[0195] In the example shown in Figure 20, while the analog pixel data 69A is being read out as recording pixel data during the readout period of the first frame, the analog phase difference pixel data 69A1 for multiple other frames is read out by the first readout circuit 62A1. The other multiple frames refer to frames 2 through 4 in the example shown in Figure 20.

[0196] In this case, while analog pixel data 69A is read out as recording pixel data within the reading period for one frame, analog phase difference pixel data 69A1 for multiple other frames is read out by the first reading circuit 62A1. However, the technology of this disclosure is not limited to this. For example, while analog pixel data 69A is read out as recording pixel data within the reading period for one frame, analog phase difference pixel data 69A1 for the next frame may be read out by the first reading circuit 62A1.

[0197] Thus, when the analog pixel data 69A is read out as recording pixel data by the second readout circuit 62A2 within the readout period of the first frame, A / D conversion is performed on the analog pixel data 69A read out as recording pixel data, as shown in Figure 21 as an example. As a result, the analog pixel data 69A is converted into digital pixel data 69B and stored in the memory 64. The digital pixel data 69B stored in the memory 64 is then output to the signal processing unit 50 by the output circuit 62D.

[0198] Next, the operation of the imaging device 10 according to this second embodiment will be described.

[0199] First, the in-image sensor processing performed by the processing circuit 62 of the image sensor 44 during the output period will be explained with reference to Figures 22 and 23.

[0200] In the image sensor processing shown in Figure 22, first, in step ST100, the control circuit 62E determines whether or not it has received the imaging vertical synchronization signal output by the execution of the timing control process in step ST12 described in the first embodiment. If the imaging vertical synchronization signal has not been received in step ST100, the determination is denied, and the image sensor processing proceeds to step ST112. If the imaging vertical synchronization signal has been received in step ST100, the determination is affirmed, and the image sensor processing proceeds to step ST102.

[0201] In step ST102, the control circuit 62E determines whether the reading of analog pixel data 69A by the readout circuit 62A is the reading of the first frame. If, in step ST102, the reading of analog pixel data 69A by the readout circuit 62A is the reading of the second frame or later, the determination is rejected, and the in-image sensor processing is performed as shown in Figure 23. The process proceeds to step ST114 shown. In step ST102, if the reading of analog pixel data 69A by the readout circuit 62A is the reading of the first frame, the determination is affirmed, and the in-image sensor processing proceeds to step ST104. Here, the readout analog pixel data 69A includes analog phase-difference pixel data 69A1 for one frame and analog non-phase-difference pixel data 69A2 for one frame.

[0202] In step ST104, the second readout circuit 62A2 reads out one frame's worth of analog pixel data 69A from the photoelectric conversion element 61, and then the image sensor processing proceeds to step ST106.

[0203] In step ST106, the digital processing circuit 62B converts the analog pixel data 69A read out by the second readout circuit 62A2 into digital pixel data 69B by performing digital signal processing on the analog pixel data 69A.

[0204] In the next step, ST107, the control circuit 62E acquires digital pixel data 69B from the digital processing circuit 62B, stores the acquired digital pixel data 69B in the memory 64, and then the in-image sensor processing proceeds to step ST110. The memory 64 stores the digital pixel data 69B that has been processed by the image processing circuit 62C.

[0205] When digital pixel data 69B is stored in memory 64, the control circuit 62E retrieves the digital pixel data 69B from memory 64, and the retrieved digital pixel data 69B is transferred to output circuit 62D.

[0206] In the next step, ST110, the output circuit 62D outputs the digital pixel data 69B input from the control circuit 62E to the signal processing unit 50, and then the in-image sensor processing proceeds to step ST112.

[0207] In step ST114 shown in Figure 23, the first readout circuit 62A1 reads out analog phase difference pixel data 69A1 for one frame from the photoelectric conversion element 61, and then the in-image sensor processing proceeds to step ST116.

[0208] In step ST116, the digital processing circuit 62B performs digital signal processing on the analog phase difference pixel data 69A1 read out by the first readout circuit 62A1, thereby converting the analog phase difference pixel data 69A1 into digital phase difference pixel data 69B1.

[0209] In the next step, ST118, the control circuit 62E acquires digital phase difference pixel data 69B1 from the digital processing circuit 62B, stores the acquired digital phase difference pixel data 69B1 in the memory 64, and then the in-image sensor processing proceeds to step ST120.

[0210] In step ST120, the control circuit 62E determines whether the phase difference pixel data output timing has arrived. The phase difference pixel data output timing refers to the timing at which the digital phase difference pixel data 69B1 in memory 64 is output to the signal processing unit 50. The phase difference pixel data output timing does not need to overlap with the timing at which the digital pixel data 69B of the first frame within the same output period is output to the signal processing unit 50. An example of a phase difference pixel data output timing is the timing at which the output of the digital pixel data 69B of the first frame within the same output period to the signal processing unit 50 is completed.

[0211] In step ST120, if the phase difference pixel data output timing has not yet arrived, the determination is denied and the determination in step ST120 is repeated. In step ST120, if the phase difference pixel data output timing has arrived, the determination is affirmed and imaging is performed. The in-device processing proceeds to step ST122.

[0212] In step ST122, the control circuit 62E acquires digital phase difference pixel data 69B1 from memory 64 and transfers the acquired digital phase difference pixel data 69B1 to the output circuit 62D. The output circuit 62D outputs the digital phase difference pixel data 69B1 input from the control circuit 62E to the signal processing unit 50, and then the in-image sensor processing proceeds to step ST112 shown in Figure 22.

[0213] In step ST112, the control circuit 62E determines whether the conditions for terminating the image sensor processing (hereinafter referred to as "image sensor processing termination conditions") have been met. For example, the image sensor processing termination conditions include the condition that an instruction to terminate the image sensor processing has been received by the receiving device 84 (see Figure 4). If the image sensor processing termination conditions are not met in step ST112, the determination is denied, and the image sensor processing proceeds to step ST100. If the image sensor processing termination conditions are met in step ST112, the determination is affirmed, and the image sensor processing is terminated.

[0214] In the signal processing unit 50, various signal processing is performed on the digital pixel data 69B input from the image sensor 44, and the digital pixel data 69B that has undergone various signal processing is output to the controller 46.

[0215] Next, when digital pixel data 69B is input from the signal processing unit 50 to the controller 46, the pixel data processing performed by the CPU 46A according to the pixel data processing program 46B3 will be explained with reference to Figure 24.

[0216] In the pixel data processing shown in Figure 24, first, in step ST250, the CPU 46A determines whether the digital pixel data 69B input from the signal processing unit 50 is recording pixel data or not. Here, "recording pixel data" refers to the pixel data that has undergone various signal processing by the signal processing unit 50 on the digital pixel data 69B of the first frame (the digital pixel data 69B output in step ST110). Furthermore, here, "digital pixel data 69B of the first frame" refers to the digital pixel data 69B obtained by digitizing the analog pixel data 69A of the first frame (the analog pixel data 69A read in step ST104) that was read by the second readout circuit 62A2 within the output period.

[0217] In step ST250, if the digital pixel data 69B input from the signal processing unit 50 is digital phase difference pixel data 69B1, the determination is denied and the pixel data processing proceeds to step ST256. In step ST250, if the digital pixel data 69B input from the signal processing unit 50 is recording pixel data, the determination is affirmed and the pixel data processing proceeds to step ST252.

[0218] In step ST252, the CPU46A records the pixel data for recording onto a predetermined recording medium, and then the pixel data processing proceeds to step ST254.

[0219] In step ST254, the CPU 46A retrieves the recording pixel data recorded on the predetermined recording medium in step ST250, thins out the recording pixel data, and outputs it to the display 32. After that, the pixel data processing proceeds to step ST254. One method for thinning out the recording pixel data is to thin out even-numbered lines or odd-numbered lines along the column direction. Once the recording pixel data is output to the display 32, the display 32 displays an image based on the recording pixel data.

[0220] In step ST256, the CPU 46A performs AF control using the digital phase-difference pixel data 69B1 input from the signal processing unit 50, and then the pixel data processing proceeds to step ST260.

[0221] In step ST260, the CPU 46A determines whether the pixel data processing termination conditions described in the first embodiment above have been satisfied. If the pixel data processing termination conditions are not satisfied in step ST260, the determination is denied, and the pixel data processing proceeds to step ST250. If the pixel data processing termination conditions are satisfied in step ST260, the determination is affirmed, and the pixel data processing is terminated.

[0222] As described above, in the image sensor 44 according to this second embodiment, analog pixel data 69A, including analog phase-difference pixel data 69A1 and analog non-phase-difference pixel data 69A2 for one frame, is read out as recording pixel data by the second readout circuit 62A2 within the readout period for one frame. Then, while the analog pixel data 69A is being read out as recording pixel data, the analog phase-difference pixel data 69A1 of other frames is read out by the first readout circuit 62A. Therefore, compared to the case where the analog phase-difference pixel data 69A1 is read out only after the analog pixel data 69A has been read out as recording pixel data, more analog pixel data 69A and analog phase-difference pixel data 69A1 can be read out as recording pixel data within a limited time.

[0223] Furthermore, in the image sensor 44 according to this second embodiment, analog non-phase-difference pixel data 69A2 is read out as recording pixel data when in continuous shooting mode. Therefore, in continuous shooting mode, the digital non-phase-difference pixel data 69B2 obtained by digitizing the analog non-phase-difference pixel data 69A2 read out as recording pixel data can be recorded on a predetermined recording medium.

[0224] In the second embodiment described above, an example was given in which analog non-phase-difference pixel data 69A2 is read out as recording pixel data in continuous shooting mode, but the technology of this disclosure is not limited thereto. For example, analog non-phase-difference pixel data 69A2 may be read out as recording pixel data when a predetermined condition is satisfied. A first example of the above predetermined condition is that a predetermined recording medium is electrically connected to an external I / F 82. A second example of the above predetermined condition is that the imaging device 10 is set to a motion image recording imaging mode. A third example of the above predetermined condition is that the imaging device 10 is set to a still image recording imaging mode. A fourth example of the above predetermined condition is that the imaging frame rate is set to a frame rate higher than a first predetermined frame rate (e.g., 120fps). A fifth example of the above predetermined condition is that the output frame rate is set to a frame rate higher than a second predetermined frame rate (e.g., 30fps). In this way, by reading out the analog non-phase-difference pixel data 69A2 as recording pixel data when the predetermined conditions are met, versatility can be increased compared to when the analog non-phase-difference pixel data 69A2 is always read out as recording pixel data.

[0225] For example, as shown in Figure 7, the light 300L passing through the left region after passing through the exit pupil of the imaging lens 40 passes through the microlens 19 corresponding to the first phase difference pixel L and is received by the photodiode PD of the first phase difference pixel L. However, even after passing through the microlens 19 corresponding to the second phase difference pixel R, the light 300L passing through the left region is blocked by the light-shielding member 21B and is therefore not received by the photodiode PD of the second phase difference pixel R.

[0226] On the other hand, the light 300R passing through the right region after passing through the exit pupil of the imaging lens 40 passes through the microlens 19 corresponding to the second phase difference pixel R, and is then processed by the photodiode PD of the second phase difference pixel R. This is how the light is received. However, the light 300R passing through the right region is blocked by the light-shielding member 21A even after passing through the microlens 19 corresponding to the first phase difference pixel L, and is therefore not received by the photodiode PD of the first phase difference pixel L.

[0227] As described above, light-shielding members are placed on half of the pixels, and the centers of the light passing through the left region 300L and the light passing through the right region 300R are offset from the optical axis of the imaging lens 40, the attenuation characteristics of the first phase difference pixel L and the second phase difference pixel R change linearly according to the pixel position on the horizontal line. This change in attenuation characteristics manifests as a change in the output of the image based on the recording pixel data, for example, when digital phase difference pixel data 69B1 is used as the recording pixel data. For example, as shown in Figure 25, the first phase difference image based on the digital phase difference pixel data 69B1 obtained from the first phase difference pixel L shows that the brightness decreases as the pixel position moves to the right. Similarly, the second phase difference image based on the digital phase difference pixel data 69B1 obtained from the second phase difference pixel R shows that the brightness decreases as the pixel position moves to the left.

[0228] Therefore, a correction coefficient for correcting the attenuation characteristics caused by the phase difference pixels may be derived by the control circuit 62E. To derive the correction coefficient, as shown in Figure 26 as an example, the control circuit 62E includes a calculation circuit 62E1. The calculation circuit 62E1 is an example of a "derivation unit (derivation circuit)" related to the technology of this disclosure and calculates the correction coefficient. The correction coefficient is broadly classified into a correction coefficient for the first phase difference pixel and a correction coefficient for the second phase difference pixel.

[0229] The correction coefficient for the first phase difference pixel is the ratio A of the digital phase difference pixel data 69B1 of the first phase difference pixel L to the digital non-phase difference pixel data 69B2 of the non-phase difference pixel N when considering adjacent non-phase difference pixels N and the first phase difference pixel L. n This refers to the proportion A shown in Figure 26. n As an example, percentages A0 and A1 are shown.

[0230] The correction coefficient for the second phase difference pixel is the ratio B of the digital phase difference pixel data 69B1 of the second phase difference pixel R to the digital non-phase difference pixel data 69B2 of the non-phase difference pixel N, when considering adjacent non-phase difference pixels N and the second phase difference pixel R. n This refers to the proportion B shown in Figure 26. n As an example, percentages B0 and B1 are shown.

[0231] The correction coefficient calculated by the calculation circuit 62E1 is output to the signal processing unit 50 by the output circuit 62D. In the signal processing unit 50, the attenuation characteristics are corrected using the correction coefficient, as shown in Figure 27 as an example. In this way, the correction coefficient for correcting the attenuation characteristics is calculated by the calculation circuit 62E1, and the calculated correction coefficient is output to the signal processing unit 50 by the output circuit 62D, thereby correcting the attenuation characteristics that appear in the image based on the digital phase difference pixel data 69B1.

[0232] Although examples of how the correction coefficient is calculated have been given here, the technology of this disclosure is not limited thereto. For example, the control circuit 62E may derive the correction coefficient using a mapping table in which the digital non-phase difference pixel data 69B2 and digital phase difference pixel data 69B1 of the non-phase difference pixel N are associated with the correction coefficient.

[0233] Furthermore, although an example of how the correction coefficient is output to the signal processing unit 50 has been described here, the technology of this disclosure is not limited thereto. For example, the image processing circuit 62C and / or the control circuit 62E may use the correction coefficient to correct the attenuation characteristics that appear in the image based on the digital phase difference pixel data 69B1.

[0234] In the embodiments described above, an A / D converter 62B1 shared for analog phase-difference pixel data 69A1 and analog non-phase-difference pixel data 69A2 was illustrated, but the technology of this disclosure is not limited thereto, and multiple A / D converters may be applied. In this case, for example, as shown in Figure 28. Accordingly, the digital processing circuit 620B is applied in place of the digital processing circuit 62B (see Figures 9 and 19) described in each of the above embodiments. The digital processing circuit 620B has a plurality of A / D converters. The "plural of A / D converters" referred to here include a first A / D converter 620B1 and a second A / D converter 620B2. The first A / D converter 620B1 is an A / D converter used only for analog phase difference pixel data 69A1, and the second A / D converter 620B2 is an A / D converter used only for analog non-phase difference pixel data 69A2. Each of the first A / D converter 620B1 and the second A / D converter 620B2 performs A / D conversion independently of each other. Therefore, by using the first A / D converter 620B1 and the second A / D converter 620B2, the A / D conversion of the analog phase-difference pixel data 69A1 and the A / D conversion of the analog non-phase-difference pixel data 69A2 can be performed in parallel within the output period.

[0235] In the embodiments described above, examples were given in which the output circuit 62D outputs the digital phase difference pixel data 69B1 itself from the image sensor 44 to the signal processing unit 50, but the technology of this disclosure is not limited thereto. For example, the output circuit 62D may output correlation data to the signal processing unit 50. In this case, the control circuit 62E acquires various information from the controller 46, such as information indicating the characteristics of the incident angle of subject light, aperture value, and defocus amount. The control circuit 62E corrects the brightness of the digital phase difference pixel data 69B1 related to the first phase difference pixel L and the brightness of the digital phase difference pixel data 69B1 related to the second phase difference pixel R according to the acquired information. That is, the control circuit 62E performs sensitivity ratio correction on the digital phase difference pixel data 69B1 related to the first phase difference pixel L and the digital phase difference pixel data 69B1 related to the second phase difference pixel R. The control circuit 62E performs a correlation calculation using the sensitivity ratio corrected digital phase difference pixel data 69B1, and the output circuit 62D outputs the correlation data, which is the result of the correlation calculation, to the signal processing unit 50.

[0236] In this way, by outputting correlation data from the image sensor 44 to the signal processing unit 50, the amount of data output from the output circuit 62D to the signal processing unit 50 can be reduced compared to the case where the digital phase-difference pixel data 69B1 itself is output from the output circuit 62D. As a result, it becomes possible to speed up autofocus.

[0237] Furthermore, in correlation calculations, pattern matching is performed, so it is preferable that the brightness of the digital phase difference pixel data 69B1 related to the first phase difference pixel L and the brightness of the digital phase difference pixel data 69B1 related to the second phase difference pixel R are the same or approximately the same. For this reason, the correlation calculation performed by the control circuit 62E may be a normalized cross-correlation calculation. In this case, even if there are fluctuations in the brightness of the subject, the similarity between the digital phase difference pixel data 69B1 related to the first phase difference pixel L and the digital phase difference pixel data 69B1 related to the second phase difference pixel R can be calculated stably.

[0238] In the embodiments described above, the case where the imaging frame rate is fixed has been explained, but the technology of this disclosure is not limited thereto. For example, the imaging frame rate may be changed in conjunction with the exposure time. Alternatively, the imaging frame rate may be increased as the exposure time decreases. Furthermore, exposure to the photoelectric conversion element 61 may be restarted after the start of exposure, once the readout circuit 62A has completed reading out at least one pixel's worth of analog pixel data 69A. The readout speed of the analog pixel data 69A by the readout circuit 62A may be changed according to the number of frames from which the analog pixel data 69A is read out in parallel. Furthermore, the readout circuit 62A may change the amount of analog pixel data 69A when the analog pixel data 69A is A / D converted, according to the number of frames from which the analog pixel data 69A is read out in parallel and the number of A / D converters that perform A / D conversion on the readout analog pixel data 69A.

[0239] Furthermore, although the above embodiments have described examples in which the processing circuit 62 is implemented by a device including an ASIC and an FPGA, the technology of this disclosure is not limited thereto. For example, the imaging process described above may be implemented by a software configuration on a computer.

[0240] In this case, for example, as shown in Figure 29, the computer 852 built into the image sensor 44 stores various programs in the storage medium 900 for executing the phase-difference pixel processing, non-phase-difference pixel processing, and in-image-sensor processing described above.

[0241] The various programs refer to the phase-difference pixel processing program 902, the non-phase-difference pixel processing program 904, and the image sensor in-processing program 906. The phase-difference pixel processing program 902 is a program that causes the computer 852 to perform the phase-difference pixel processing described above. The non-phase-difference pixel processing program 904 is a program that causes the computer 852 to perform the non-phase-difference pixel processing described above. The image sensor in-processing program 906 is a program that causes the computer 852 to perform the image sensor in-processing described above.

[0242] As an example, as shown in Figure 29, the computer 852 is equipped with a CPU 852A, a ROM 852B, and a RAM 852C. Various programs stored in the storage medium 900 are installed in the computer 852. The CPU 852A performs the phase difference pixel processing described above according to the phase difference pixel processing program 902. The CPU 852A also performs the non-phase difference pixel processing described above according to the non-phase difference pixel processing program 904. Furthermore, the CPU 852A performs the image sensor processing described above according to the image sensor processing program 906.

[0243] Here, a single CPU is given as CPU852A, but the technology of this disclosure is not limited to this, and multiple CPUs may be used instead of CPU852A. The storage medium is a non-temporary storage medium. An example of the storage medium 900 is any portable storage medium such as an SSD or USB memory.

[0244] In the example shown in Figure 29, various programs are stored in the storage medium 900, but the technology of this disclosure is not limited to this. For example, various programs may be pre-stored in ROM 852B, and the CPU 852A may read the various programs from ROM 852B, expand them in RAM 852C, and execute the expanded programs.

[0245] Alternatively, various programs may be stored in the memory of another computer or server device connected to the computer 852 via a communication network (not shown), and these programs may be downloaded to the computer 852 in response to requests from the imaging device 10. In this case, the downloaded programs are executed by the CPU 852A of the computer 852.

[0246] Alternatively, the computer 852 may be located outside the image sensor 44. In this case, the computer 852 can control the processing circuit 62 according to various programs.

[0247] As hardware resources that perform the phase-difference pixel processing, non-phase-difference pixel processing, in-image sensor processing, timing control processing, and pixel data processing (hereinafter referred to as "various processes") described in each of the above embodiments, the following types of processors can be used. As processors, for example, as mentioned above, a CPU is a general-purpose processor that functions as a hardware resource that performs various processes by executing software, i.e., a program. As processors, for example, FPGAs, PLDs, or ASICs can be used. One example is a dedicated electrical circuit, which is a processor with a circuit configuration specifically designed to perform a particular process.

[0248] The hardware resources that perform various processes may consist of one of these various processors, or a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resources that perform various processes may consist of a single processor.

[0249] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, as exemplified by client and server computers, and this processor functions as a hardware resource that executes processing within the imaging device. Secondly, a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that execute various processes, on a single IC chip, as exemplified by SoCs. Thus, processing within the imaging device is realized using one or more of the above types of processors as hardware resources.

[0250] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices.

[0251] Furthermore, although a lens-interchangeable camera was used as an example of the imaging device 10 in the above embodiments, the technology of this disclosure is not limited to this. For example, the technology of this disclosure may be applied to the smart device 950 shown in Figure 30. As an example, the smart device 950 shown in Figure 30 is an example of an imaging device relating to the technology of this disclosure. The smart device 950 is equipped with the image sensor 44 described in the above embodiments. Even with a smart device 950 configured in this way, the same operation and effects as the imaging device 10 described in the above embodiments can be obtained. It should be noted that the technology of this disclosure is applicable not only to the smart device 950, but also to personal computers or wearable terminal devices.

[0252] Furthermore, although the above embodiments illustrate a display 32, the technology of this disclosure is not limited thereto. For example, a separate display attached to the imaging device body 12 may be used as the "display unit (display)" according to the technology of this disclosure.

[0253] Furthermore, the various processes described above are merely examples. Therefore, it goes without saying that you may remove unnecessary steps, add new steps, or change the order of processes, as long as you do not deviate from the main purpose.

[0254] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0255] In this specification, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0256] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

Claims

1. An image sensor including pixels capable of acquiring data regarding the focus state of a subject, A readout circuit reads out pixel data obtained by imaging the subject at a first frame rate, A memory for storing the pixel data read out by the aforementioned reading circuit, The output circuit includes outputting image data based on pixel data stored in the memory at a second frame rate, The aforementioned first frame rate is a higher frame rate than the aforementioned second frame rate. The pixel data includes focus-related pixel data, which is data relating to the focus state of the subject, and out-of-focus related pixel data, which is different from the focus-related pixel data. The readout circuit performs the reading of the out-of-focus related pixel data and the reading of the focused related pixel data multiple times within the period of one frame defined by the second frame rate. Image sensor.

2. The image sensor according to claim 1, wherein the readout circuit performs the reading of the out-of-focus related pixel data and the reading of the in-focus related pixel data in parallel.

3. The image sensor according to claim 1 or claim 2, wherein the readout circuit reads out the pixel data in line units.

4. The aforementioned focusing-related pixel data is the pixel data of the aforementioned pixel, The image sensor according to any one of claims 1 to 3, wherein the out-of-focus related pixel data is pixel data of an out-of-focus related pixel different from the aforementioned pixel.

5. The reading of the out-of-focus related pixel data is the reading of the out-of-focus related pixel data from the out-of-focus related pixels, The image sensor according to claim 4, wherein the reading of the focus-related pixel data multiple times is the reading of the focus-related pixel data multiple times from the pixel.

6. The imaging surface includes a plurality of first lines containing the aforementioned pixels and a plurality of second lines consisting only of the aforementioned out-of-focus related pixels, The image sensor according to claim 4 or 5, wherein the readout circuit comprises a first readout circuit that reads the focus-related pixel data from each of the pixels included in the plurality of first lines, and a second readout circuit that reads the out-of-focus related pixel data from each of the out-of-focus related pixels included in the plurality of second lines.

7. The image sensor according to claim 6, wherein the reading of the focus-related pixel data from the pixel by the first reading circuit and the reading of the out-of-focus related pixel data from the out-of-focus related pixel by the second reading circuit are performed independently.

8. The image sensor according to claim 7, wherein, within the reading period for one frame, the reading of the focus-related pixel data from the pixel by the first reading circuit is performed before the reading of the out-of-focus related pixel data from the out-of-focus related pixel by the second reading circuit.

9. The image sensor according to any one of claims 6 to 8, wherein the first line is a line in which the pixels and the out-of-focus related pixels are arranged periodically.

10. The image sensor according to any one of claims 6 to 9, wherein the imaging surface is arranged alternately with the first line and a predetermined number of the second lines along a direction intersecting the line direction of the first line.

11. The image sensor according to any one of claims 1 to 10, wherein the readout circuit reads out one frame's worth of out-of-focus related pixel data as recording pixel data within a readout period for one frame, and reads out the in-focus related pixel data while the out-of-focus related pixel data is being read out as recording pixel data.

12. The image sensor according to claim 11, wherein the readout circuit reads out the out-of-focus related pixel data and the in-focus related pixel data as display pixel data within a readout period of one frame, and reads out the out-of-focus related pixel data as recording pixel data when a predetermined condition is satisfied.

13. The image sensor according to claim 11, wherein the readout circuit reads out the out-of-focus related pixel data as the recording pixel data in the continuous shooting mode.

14. The system further includes a derivation circuit that derives a correction coefficient for correcting the attenuation characteristics caused by the aforementioned pixels, based on the focusing-related pixel data. The image sensor according to any one of claims 1 to 13, wherein the output circuit outputs the correction coefficient derived by the derivation circuit.

15. The image data includes first pixel data based on the out-of-focus related pixel data and second pixel data based on the focused related pixel data obtained by reading the focused related pixel data multiple times. The image sensor according to any one of claims 1 to 14, wherein the output circuit outputs the first pixel data and the second pixel data at different timings when outputting the image data for one frame.

16. The image sensor according to claim 15, wherein the output circuit outputs the first pixel data after the output of the second pixel data is completed.

17. The image sensor according to any one of claims 1 to 16, wherein the image data includes pixel data based on the non-focused related pixel data and pixel data based on statistical values ​​of the focused related pixel data obtained by reading the focused related pixel data multiple times.

18. The image sensor according to claim 17, wherein the statistical value is the summation average of the focusing-related pixel data.

19. Includes an A / D converter shared for the aforementioned focus-related pixel data and the aforementioned out-of-focus pixel data, The image sensor according to any one of claims 1 to 18, wherein the A / D converter performs A / D conversion on the in-focus related pixel data and the out-of-focus related pixel data at different timings.

20. Includes multiple A / D converters, The image sensor according to any one of claims 1 to 18, wherein the plurality of A / D converters include a first A / D converter used only for the focusing-related pixel data and a second A / D converter used only for the out-of-focus-related pixel data.

21. An image sensor according to any one of claims 1 to 20, wherein at least a photoelectric conversion element and the memory are integrated into a single chip.

22. The image sensor according to claim 21, wherein the image sensor is a stacked image sensor in which the memory is stacked on the photoelectric conversion element.

23. An image sensor according to any one of claims 1 to 22, A processor that performs at least one of the following: control to display an image based on the image data output by the output circuit on a display, and control to store the image data output by the output circuit in a storage device. An imaging device including an imaging device.

24. A method for operating an image sensor, comprising: pixels capable of acquiring data relating to the focus state of a subject; a readout circuit that reads out pixel data obtained by imaging the subject at a first frame rate; a memory that stores the pixel data read out by the readout circuit; and an output circuit that outputs image data based on the pixel data stored in the memory at a second frame rate, The aforementioned first frame rate is a higher frame rate than the aforementioned second frame rate. The pixel data includes focus-related pixel data, which is data relating to the focus state of the subject, and out-of-focus related pixel data, which is different from the focus-related pixel data. The method for operating an image sensor includes, within the period of one frame defined by the second frame rate, reading out the out-of-focus related pixel data and reading out the in-focus related pixel data multiple times.

25. A program for causing a computer to function as the read circuit and output circuit included in an image sensor, the read circuit and output circuit being the read circuit and the image sensor, the read circuit and output circuit being the read circuit and the image sensor being included in the image sensor, the read circuit and output circuit being the read circuit and the output circuit being the read circuit and output The aforementioned first frame rate is a higher frame rate than the aforementioned second frame rate. The pixel data includes focus-related pixel data, which is data relating to the focus state of the subject, and out-of-focus related pixel data, which is different from the focus-related pixel data. The readout circuit performs the reading of the out-of-focus related pixel data and the reading of the focused related pixel data multiple times within the period of one frame defined by the second frame rate. program.

Images