Imaging element, imaging device, and imaging method

Abstract

Provided is an imaging element comprising: a first photoelectric conversion unit that converts light emitted from an optical system having a focus lens into electric charge; a second photoelectric conversion unit that converts light emitted from the optical system into electric charge; and a drive control unit that performs control such that, in a period in which the focus lens moves from a first position to a second position, a first timing at which the accumulation of the electric charge converted by the first photoelectric conversion unit is started and a second timing at which the accumulation of the electric charge converted by the second photoelectric conversion unit is started are different.

Description

Image pickup element, image pickup device, and image pickup method

[0001] The present invention relates to an imaging element, an imaging device, and an imaging method.

[0002] Solid-state imaging devices in which pixels are arranged two-dimensionally in a matrix in the vertical and horizontal directions are known (for example, see Patent Document 1). Conventionally, a problem has been that control of readout timing and the like is uniformly applied to all pixels on the imaging surface. [Prior art documents] [Patent documents] [Patent document 1] JP 2007-214832 A General disclosure

[0003] A first aspect of the present invention provides an image sensor including a first photoelectric conversion unit that converts light emitted from an optical system having a focus lens into electric charges, a second photoelectric conversion unit that converts light emitted from the optical system into electric charges, and a drive control unit that controls the first and second photoelectric conversion units so that a first timing at which accumulation of the electric charges converted by the first photoelectric conversion unit starts is different from a second timing at which accumulation of the electric charges converted by the second photoelectric conversion unit starts during a period in which the focus lens moves from a first position to a second position.

[0004] The above summary of the invention does not list all of the features of the present invention, and subcombinations of these features may also be inventions.

[0005] 1 is a block diagram of an imaging device 10 according to a first embodiment. It is a schematic diagram showing functional blocks of an entire image sensor 100 according to the first embodiment. It is a schematic diagram showing the positional relationship between an area in which a pixel unit 50 is arranged and an area in which a processing circuit unit 160 is arranged in the image sensor 100 according to the first embodiment. It is a schematic diagram showing one of a plurality of unit blocks 200 and a peripheral circuit unit 500 according to the first embodiment. It shows an example of the circuit configuration of a pixel 202. It shows an example of a timing chart illustrating the operation timing of one pixel block 142 having a pixel 202. It is a diagram illustrating an example of an image captured of a subject 55 with depth. It is an example of a processing flow of the image sensor 100 according to the first embodiment. It shows an example of a timing chart illustrating the operation timing of the image sensor 100 according to the first embodiment. It shows an example of the circuit configuration of a pixel 202-1. It shows an example of a timing chart illustrating the operation timing of one pixel block 142 having a pixel 202-1. It shows another example of a timing chart illustrating the operation timing of the image sensor 100 according to the first embodiment. 1 is a diagram illustrating an example of an image obtained by capturing images of a plurality of subjects 55A, 55B, and 55C having different reflectances. 2 is another example of a processing flow of the image sensor 100 according to the first embodiment. 3 is another example of a timing chart illustrating an outline of the operation timing of the image sensor 100 according to the first embodiment.

[0006] The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the scope of the invention as claimed. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

[0007] 1 is a block diagram of an imaging device 10 according to a first embodiment. The imaging device 10 has an optical system 20, a lens driving unit 23, an imaging element 100, an image processing unit 24, a control unit 26, a recording unit 28, an operation unit 30, a display unit 32, and a power supply unit 34. The lens driving unit 23, the imaging element 100, the image processing unit 24, the control unit 26, the recording unit 28, the operation unit 30, the display unit 32, and the power supply unit 34 are interconnected via a bus line 36.

[0008] The optical system 20 has a group of lenses including a focus lens, and focuses image light, which is incident light from a subject, on the imaging surface of the image sensor 100 along an optical axis 22. The focus lens is, for example, a liquid lens. Note that in FIG. 1, a Z axis parallel to the optical axis 22 and X and Y axes perpendicular to the Z axis are shown, and this also applies to the subsequent figures.

[0009] The lens driver 23 controls the focus position of the optical system 20. The lens driver 23 changes the focus position by moving the focus lens from a first position to a second position. As shown in Fig. 1 , the first position of the focus lens may be, for example, on the positive side of the Z axis in Fig. 1 , i.e., in front of the image sensor 100. In this case, the second position may be on the negative side of the Z axis in Fig. 1 , i.e., behind the image sensor 100.

[0010] In the following description, the first position may be referred to as the "front" and the second position may be referred to as the "back." The basic time unit for obtaining one image's worth of data may be referred to as one frame. One frame mainly includes the charge accumulation time (exposure time) of the photoelectric conversion unit and the pixel signal readout time for one image capture. One frame is, for example, approximately 1 / 60 of a second. The lens driver 23 moves the focus lens from the first position to the second position within one frame period. In the following description, the period during which the focus lens moves from the first position to the second position may be simply referred to as the "movement period." Note that after the end of the movement period, the focus lens may be returned from the second position to the first position by the lens driver 23 before the end of one frame, for example, during pixel signal readout. Note that the first and second positions may be reversed, and the time it takes for the focus lens to move from the back to the front may be the same as or different from the time it takes for the focus lens to move from the front to the back. In this specification, when it is defined that the focus lens moves from a first position to a second position, it is intended that the focus lens moves in one direction along the optical axis 22 from the first position to the second position.

[0011] The image sensor 100 captures an image of a subject. As will be described in detail later, the image sensor 100 includes a plurality of stacked semiconductor substrates. The plurality of semiconductor substrates include semiconductor substrates including a plurality of photoelectric conversion units that convert light emitted from the optical system 20 into electric charges. More specifically, the image sensor 100 converts the light emitted from the optical system 20 into an electric signal and outputs it as a pixel signal. Unless otherwise specified, the "pixel signal" may be in either an analog or digital state.

[0012] The multiple semiconductor substrates included in the image sensor 100 also include a semiconductor substrate that includes a circuit that controls the timing at which the accumulation of electric charges converted by the multiple photoelectric conversion units begins to occur at different times during the period in which the focus lens moves from the first position to the second position, as will be described in detail later.

[0013] The image processing unit 24 is, for example, an ISP (Image Signal Processor), and receives pixel signals output from the image sensor 100. The image processing unit 24 performs image processing on the received pixel signals. This image processing includes color interpolation, tone conversion, compression, and the like.

[0014] The control unit 26 controls the entire imaging device 10, and controls each of the lens driving unit 23, the imaging element 100, the image processing unit 24, the recording unit 28, the operation unit 30, the display unit 32, and the power supply unit 34 via a bus line 36.

[0015] The recording unit 28 records moving or still images of the subject captured by the image sensor 100 on a recording medium such as a hard disk or semiconductor memory. The operation unit 30 issues operation commands for various functions of the image sensor 10 under user operation. The display unit 32 is a panel-type display device such as a liquid crystal panel or an organic EL (Electro Luminescence) panel, and displays moving or still images of the subject captured by the image sensor 100. The power supply unit 34 supplies various types of power to the lens drive unit 23, image processing unit 24, control unit 26, recording unit 28, operation unit 30, and display unit 32 as operating power sources.

[0016] 2 is a schematic diagram showing the overall functional blocks of the image sensor 100 according to the first embodiment. Fig. 2 shows the relationship between the functional blocks and the transmission and reception of signals between them, but does not show the spatial arrangement within the image sensor 100.

[0017] The imaging device 100 has a pixel section 50 , a drive control section 60 , a vertical drive section 62 , a signal processing section 64 , a horizontal drive section 68 , and an output section 70 .

[0018] The pixel unit 50 has a plurality of pixels arranged along a first direction and a second direction intersecting the first direction. The first direction is, for example, the row direction. The second direction is, for example, the column direction. In the example of FIG. 2 , the first direction is the X-axis direction, and the second direction is the Y-axis direction. The pixel unit 50 has a plurality of pixels arranged two-dimensionally. Each of the plurality of pixels includes at least one photoelectric conversion unit. These pixels convert image light from the optical system 20 into electric charges through photoelectric conversion.

[0019] The drive control unit 60 receives an input clock and signals instructing the operation mode and the like from the control unit 26, and drives the image sensor 100. For example, the drive control unit 60 generates clock signals and control signals that serve as references for the operations of the vertical drive unit 62, signal processing unit 64, horizontal drive unit 68, and the like, based on a vertical synchronization signal, a horizontal synchronization signal, and a master clock, and outputs these to the vertical drive unit 62, the signal processing unit 64, and the horizontal drive unit 68. The drive control unit 60 receives image information, phase difference information, and the like from the pixel unit 50 via the signal processing unit 64, and controls the multiple pixels of the pixel unit 50 via the vertical drive unit 62, etc.

[0020] The drive control unit 60 controls the timings at which the accumulation of charges converted by the multiple photoelectric conversion units starts to differ from each other during a period in which the focus lens of the optical system 20 moves from the first position to the second position. Specifically, the drive control unit 60 controls the timings at which the accumulation of charges converted by the first photoelectric conversion unit starts to differ from the timings at which the accumulation of charges converted by the second photoelectric conversion unit starts in the multiple pixels of the pixel unit 50 during the period.

[0021] During the above-mentioned period, drive control unit 60 causes each of the first photoelectric conversion unit and the second photoelectric conversion unit to sequentially start accumulating electric charges according to the first timing and the second timing, in synchronization with the movement of the focus lens of optical system 20 from the first position to the second position. In other words, drive control unit 60 controls at least two photoelectric conversion units to have different exposure timings, in synchronization with the movement of the focus lens from the first position to the second position, within the movement period.

[0022] The vertical drive unit 62 drives the pixels, for example, row by row, based on the control signal. It can also be said that the vertical drive unit 62 controls the exposure time of the pixels using the control signal. The signal processing unit 64 performs signal processing on the pixel signals read out from the pixels. For example, the signal processing unit 64 has a conversion unit that converts the pixel signals read out from the pixels into digital signals. The signal processing unit 64 also has a noise removal unit that removes noise contained in the pixel signals read out from the pixels. The signal processing unit 64 temporarily stores the pixel signals after signal processing. The horizontal drive unit 68 sequentially reads out the pixel signals stored in the signal processing unit 64 and outputs them to the output unit 70.

[0023] The output unit 70 performs specific processing on the pixel signals sequentially supplied from the signal processing unit 64 and outputs the signals to the image processing unit 24. The specific processing performed by the output unit 70 is, for example, buffering.

[0024] 3 schematically shows the positional relationship between an area where the pixel unit 50 is arranged and an area where the processing circuit unit 160 is arranged in the image sensor 100 according to the first embodiment. The image sensor 100 has a first semiconductor substrate 140 and a second semiconductor substrate 150, which are stacked in this order in the positive direction of the Z axis in FIG.

[0025] The pixel unit 50 is disposed on a first semiconductor substrate 140. In the following description, the first semiconductor substrate 140 may be referred to as a pixel chip. The pixel unit 50 has a plurality of pixel blocks 142 arranged along a first direction and a second direction intersecting the first direction. The first direction is, for example, the row direction. The second direction is, for example, the column direction. The pixel unit 50 has a plurality of pixel blocks 142 arranged two-dimensionally. Each of the pixel blocks 142 includes a plurality of pixels arranged along the first direction and the second direction. Each of the pixel blocks 142 includes a plurality of pixels arranged two-dimensionally. Each of the plurality of pixels includes at least one photoelectric conversion unit.

[0026] It can be said that the first semiconductor substrate includes at least a first pixel block having a plurality of photoelectric conversion units including the above-described first photoelectric conversion unit, and a second pixel block having a plurality of photoelectric conversion units including the above-described second photoelectric conversion unit. Furthermore, in this embodiment, all of the pixels in the pixel unit 50 are image plane phase difference pixels for calculating the defocus amount of the optical system 20 in pixel block units, and are also image pixels. It can be said that the plurality of photoelectric conversion units in each of the first pixel block and the second pixel block include image plane phase difference pixels for calculating the defocus amount of each of the first pixel block and the second pixel block. Note that pixel signals output from the image plane phase difference pixels can be used as phase difference information for calculating the defocus amount of the optical system 20 in pixel block units, and can also be used as image information for generating an image of a subject.

[0027] The processing circuit section 160, the drive control section 60, the output section 58, the global vertical drive section 152, and the global horizontal drive section 154 are provided on a second semiconductor substrate 150. The second semiconductor substrate 150 may also be called a signal processing chip.

[0028] The processing circuit unit 160 has a plurality of signal processing blocks 162 arranged along a first direction and a second direction intersecting the first direction. The first direction is, for example, the row direction. The second direction is, for example, the column direction. In the example of FIG. 3 , the first direction is the X-axis direction and the second direction is the Y-axis direction. Corresponding to the pixel unit 50 having a plurality of pixel blocks 142 arranged two-dimensionally, the processing circuit unit 160 has a plurality of signal processing blocks 162 arranged two-dimensionally. The signal processing blocks 162 include circuits that perform drive and signal processing on the pixel blocks 142, and may include, for example, some or all of the vertical drive unit 62, signal processing unit 64, and horizontal drive unit 68 of FIG. 2 .

[0029] The processing circuit unit 160 in this example is disposed at a position facing the pixel unit 50 on the second semiconductor substrate 150. That is, the processing circuit unit 160 is disposed so as to at least partially overlap with the pixel unit 50 in the stacking direction in which the first semiconductor substrate 140 and the second semiconductor substrate 150 are stacked, i.e., in the optical axis direction. In this case, the pixel unit 50 and the processing circuit unit 160 may overlap so as to encompass the other in the optical axis direction.

[0030] Furthermore, the signal processing block 162 is disposed on the second semiconductor substrate 150 at a position facing the corresponding pixel block 142. That is, the signal processing block 162 that performs drive and signal processing for a pixel block 142 at a certain position is disposed so as to at least partially overlap with the pixel block 142 in the stacking direction in which the first semiconductor substrate 140 and the second semiconductor substrate 150 are stacked, i.e., in the optical axis direction. In this case, the pixel block 142 and the corresponding signal processing block 162 may overlap so as to encompass the other in the optical axis direction.

[0031] The drive control unit 60, the output unit 58, the global vertical drive unit 152, and the global horizontal drive unit 154 are arranged on the periphery of the processing circuit unit 160 on the second semiconductor substrate. Here, the global vertical drive unit 152, the global horizontal drive unit 154, and the output unit 58 perform common control for a plurality of pixel blocks 142, and may be, for example, parts of the vertical drive unit 62, the horizontal drive unit 68, and the output unit 70 other than those included in the signal processing block 162.

[0032] The drive control unit 60 sets the first timing and the second timing to be different timings during a period in which the focus lens of the optical system 20 moves from the first position to the second position, i.e., the exposure timing of the first pixel block and the exposure timing of the second pixel block, based on the defocus amount of the optical system 20 for each pixel block corresponding to the first pixel block and the second pixel block. The drive control unit 60 starts accumulating the charges converted by the first photoelectric conversion unit and the charges converted by the second photoelectric conversion unit based on the defocus amount of the optical system 20. The drive control unit 60 may have a storage unit that stores information regarding the relationship between the period and the focus position of the optical system 20. The drive control unit 60 may start accumulating the charges converted by the first photoelectric conversion unit and the charges converted by the second photoelectric conversion unit based on the information. The drive control unit 60 may set the first timing and the second timing based on the information. The information may be a function capable of calculating the focus position from the defocus amount, a function capable of calculating the time within one frame from the focus position, or a combination of these.

[0033] The drive control unit 60 may set the exposure timing within the period for all pixel blocks 142 in the pixel unit 50 based on the defocus amount of the optical system 20 for each corresponding pixel block. In this case, if the focus positions of the subject indicated in the image information from any pixel block 142 are the same or approximately the same, the drive control unit 60 may set the same exposure timing for these pixel blocks 142.

[0034] In this embodiment, the drive control unit 60 calculates the defocus amount of the optical system 20 for each of the first pixel block and the second pixel block from the output of the pixel for image plane phase difference, i.e., from the above-mentioned phase difference information. The drive control unit 60 may, for example, store a function for calculating the defocus amount from the phase difference information in the above-mentioned storage unit and calculate the defocus amount using the function.

[0035] Note that part or all of the drive control unit 60 may be included in the signal processing block 162. In this case, the drive control unit 60 may be provided corresponding to each of the above-described first pixel blocks and second pixel blocks, or may be provided corresponding to each of all pixel blocks 142 in the pixel unit 50. In this case, in the stacking direction of the multiple semiconductor substrates, the drive control unit 60 corresponding to the first pixel block may be arranged in a region of the second semiconductor substrate 150 that overlaps with the first pixel block, and the drive control unit 60 corresponding to the second pixel block may be arranged in a region of the second semiconductor substrate 150 that overlaps with the second pixel block.

[0036] The image sensor 100 may further include a stacked third semiconductor substrate in addition to the first semiconductor substrate 140 and the second semiconductor substrate 150. For example, the third semiconductor substrate performs image processing on the pixel signals output from the second semiconductor substrate 150. The third semiconductor substrate may also include a memory unit that stores the pixel signals output from the second semiconductor substrate 150. The structure of the image sensor 100 may be either a back-illuminated type or a front-illuminated type.

[0037] 4 is a schematic diagram illustrating one of the unit blocks 200 and the peripheral circuit unit 500 in the first embodiment. In this figure and the following figures, for the purpose of making the drawings easier to understand, functional blocks included in the unit block 200 and the peripheral circuit unit 500 that are not described in the figure and signal lines connected to those functional blocks may be omitted.

[0038] Each unit block 200 includes one of the plurality of pixel blocks 142 and a local circuit portion provided corresponding to the pixel block 142. Therefore, there are a number of unit blocks 200 corresponding to, for example, the same number as, the plurality of pixel blocks 142. In addition, all or part of the unit blocks 200 other than the pixel blocks 142 correspond to the signal processing block 162 in Figure 3. On the other hand, the peripheral circuit portion 500 is a global circuit portion provided in common to the plurality of unit blocks 200.

[0039] The pixel block 142 shown in FIG. 4 includes pixels 202. As an example, nine pixels 202 arranged in three rows and three columns are shown. The pixels 202 are pixels for image plane phase difference used to calculate the defocus amount, and are also pixels used to form an image. For simplicity of symbols, the pixels 202 may be represented as Aij (i is the number of rows, j is the number of columns). The number of rows, the number of columns, and the total number of pixels 202 are not limited to those described above. For example, the pixels 202 may be arranged in 16 rows and 16 columns, or 32 rows and 32 columns.

[0040] The unit block 200 has a block vertical drive unit 210 and a signal processing circuit 222. The block vertical drive unit 210 drives the pixels 202 row by row based on control signals, such as drive pulses, from the peripheral circuit unit 500. Therefore, it can be said that the block vertical drive unit 210 constitutes at least a part of the vertical drive unit 62 in FIG. 2 .

[0041] The signal processing circuit 222 performs signal processing on pixel signals read from the pixels 202 based on control signals from the peripheral circuit unit 500. As the signal processing circuit 222, FIG. 4 shows a conversion unit 224 that converts analog pixel signals into digital pixel signals and a storage unit 226 that stores the digital pixel signals. The conversion unit 224 may be, for example, a single-slope type, but other types may also be used. The signal processing circuit 222 may also include other processing circuits, such as a CDS circuit (correlated double sampling circuit). In FIG. 4, a signal processing circuit 222 is arranged for each pixel column, and three signal processing circuits 222 are provided corresponding to the three columns of pixels 202. The pixel signals from the signal processing circuit 222 are output to the peripheral circuit unit 500 via horizontal lines 502.

[0042] The signal processing circuits 222 included in one unit block 200 may be collectively referred to as a signal processing circuit group 220. It can also be said that the signal processing circuit group 220 constitutes at least a part of the signal processing unit 64 and the horizontal driving unit 68 in FIG.

[0043] 3 and 4 , the image sensor 100 according to this embodiment may include multiple pixels 202 in one pixel block 140. Alternatively, one pixel block 140 may include one pixel 202.

[0044] FIG. 5 shows an example of the circuit configuration of a pixel 202. FIG. 6 shows an example of a timing chart illustrating the operation timing of one pixel block 142 having pixels 202. In FIG. 6, of the first to N rows included in one pixel block 142, only the timing charts for the first row (Row 1), the second row (Row 2), and the Nth row (Row N) are representatively shown, and the timing charts for the third to N-1th rows are omitted. Furthermore, the timing charts for the first to Nth rows included in one pixel block 142 are identical to each other except that readout is performed sequentially from the first row to the Nth row, and therefore only the timing chart for one row will be described below.

[0045] As shown in FIG. 5 , in this embodiment, a pixel 202 has two photoelectric conversion units 120L and 120R that are adjacently arranged in a first direction and can output pixel signals separately. In the example of FIG. 3 described above, the first direction is the X-axis direction. The two photoelectric conversion units 120L and 120R can provide image information by adding together the pixel signals that they output separately, in addition to providing phase difference information for calculating the defocus amount. Note that the photoelectric conversion units 120L and 120R are an example of a first photoelectric conversion unit or a second photoelectric conversion unit.

[0046] 5, the pixel 202 includes a first transfer unit 123L-1, a discharge unit 123L-2, a second transfer unit 123L-3, and a memory 124L associated with the photoelectric conversion unit 120L located on the left side. Similarly, the pixel 202 includes a first transfer unit 123R-1, a discharge unit 123R-2, a second transfer unit 123R-3, and a memory 124R associated with the photoelectric conversion unit 120R located on the right side. The pixel 202 further includes a reset unit 126 and a pixel output unit 127 as components common to the two photoelectric conversion units 120L and 120R. The pixel output unit 127 includes an amplifier unit 128 and a selector unit 129. In this example, the first transfer units 123L-1 and 123R-1, the discharge units 123L-2 and 123R-2, the second transfer units 123L-3 and 123R-3, the reset unit 126, the amplifier unit 128, and the selector unit 129 are described as N-channel FETs, but the type of transistor is not limited to this.

[0047] The photoelectric conversion units 120L and 120R have a photoelectric conversion function of converting light into electric charges. The photoelectric conversion units 120L and 120R convert light emitted from the optical system 20 into electric charges and output the electric charges. The electric charges output from the photoelectric conversion units 120L and 120R are accumulated in the circuit on the output side of the photoelectric conversion units 120L and 120R when the circuit is closed, and flow to the low potential side when the circuit is open. The photoelectric conversion units 120L and 120R are, for example, photodiodes.

[0048] The first transfer units 123L-1 and 123R-1 transfer the charges accumulated in the circuits in the closed state to the memories 124L and 124R. The first transfer units 123L-1 and 123R-1 are examples of transfer gates that transfer the charges of the photoelectric conversion units 120L and 120R. In other words, the first transfer units 123L-1 and 123R-1 serve as gates, the photoelectric conversion units 120L and 120R serve as sources, and the memories 124L and 124R serve as drains, which together form a so-called transfer transistor.

[0049] When the photoelectric conversion units 120L and 120R are exposed, local control signals φTX1L and φTX1R specific to the pixel block 142 corresponding to the block vertical drive unit 210 are input from the block vertical drive unit 210 to the gate terminals of the first transfer units 123L-1 and 123R-1. More specifically, as indicated by TX1L and TX1R in FIG. 6 , once during exposure in each frame, the control signals φTX1L and φTX1R are simultaneously input to the gate terminals of the first transfer units 123L-1 and 123R-1 in accordance with exposure timing that is set differently for each pixel block 142. This causes the first transfer units 123L-1 and 123R-1 to go high, and the accumulated charges are transferred to the memories 124L and 124R.

[0050] In this way, the two photoelectric conversion units 120L and 120R in the pixel 202 are exposed simultaneously. In this embodiment, at least some of the multiple pixel blocks 142 are exposed at different exposure timings, but the photoelectric conversion units 120L and 120R in each pixel block 142 are exposed simultaneously.

[0051] The memories 124L and 124R receive the charges transferred from the photoelectric conversion units 120L and 120R by the first transfer units 123L-1 and 123R-1. The memories 124L and 124R are, for example, capacitors.

[0052] The second transfer units 123L-3 and 123R-3 transfer the charges accumulated in the memories 124L and 124R to the accumulation unit 125. The second transfer units 123L-3 and 123R-3 are an example of transfer gates that transfer the charges of the photoelectric conversion units 120L and 120R via the memories 124L and 124R. In other words, the second transfer units 123L-3 and 123R-3 serve as gates, the memories 124L and 124R on the photoelectric conversion units 120L and 120R side serve as sources, and the accumulation unit 125 serves as a drain, constituting a so-called transfer transistor.

[0053] When the photoelectric conversion units 120L and 120R are exposed, local control signals φTX3L and φTX3R that are specific to the pixel block 142 corresponding to the block vertical drive unit 210 are input from the block vertical drive unit 210 to the gate terminals of the second transfer units 123L-3 and 123R-3.

[0054] 6, once during exposure in each frame, control signals φTX3L and φTX3R are input to the gate terminals of the second transfer units 123L-3 and 123R-3 immediately before the charges accumulated on the output sides of the photoelectric conversion units 120L and 120R are transferred to the memories 124L and 124R in accordance with exposure timing that is set differently for each pixel block 142. This causes the second transfer units 123L-3 and 123R-3 to go high, the charges remaining in the memories 124L and 124R are transferred to the accumulation unit 125, and the memories 124L and 124R are reset.

[0055] When pixel signals are read out, global control signals φTX3L and φTX3R, which are common to the plurality of pixel blocks 142, are input from the vertical drive unit 62 to the gate terminals of the second transfer units 123L-3 and 123R-3.

[0056] 6, once during readout of pixel signals in each frame, control signals φTX3L and φTX3R are input sequentially to the gate terminals of the second transfer units 123L-3 and 123R-3 in accordance with a readout timing common to the plurality of pixel blocks 142. As a result, the second transfer units 123L-3 and 123R-3 go high in this order, and the charges accumulated in the memories 124L and 124R during exposure are transferred sequentially to the accumulation unit 125.

[0057] The discharge units 123L-2 and 123R-2 discharge the charges accumulated in the photoelectric conversion units 120L and 120R to power supply wiring to which the power supply voltage VDD is supplied. Immediately before the photoelectric conversion units 120L and 120R are exposed to light, local discharge control signals φTX2L and φTX2R that are specific to the pixel block 142 corresponding to the block vertical drive unit 210 are input to the gate terminals of the discharge units 123L-2 and 123R-2 from the block vertical drive unit 210.

[0058] 6, once in each frame immediately before the exposure of the photoelectric conversion units 120L, 120R, discharge control signals φTX2L, φTX2R are simultaneously input to the gate terminals of the discharge units 123L-2, 123R-2 in accordance with a readout timing common to the plurality of pixel blocks 142. As a result, the discharge units 123L-2, 123R-2 become high in synchronization, and the charges remaining in the photoelectric conversion units 120L, 120R are discharged to the power supply wiring.

[0059] Charges from the memories 124L and 124R are transferred to the storage unit 125 by the second transfer units 123L-3 and 123R-3. The storage unit 125 is an example of a floating diffusion (FD).

[0060] The reset unit 126 discharges the charge in the storage unit 125 to the power supply wiring to which the power supply voltage VDD is supplied.

[0061] When the photoelectric conversion units 120L and 120R are exposed, a reset control signal φRST that is specific to the pixel block 142 corresponding to the block vertical drive unit 210, i.e., local to the block vertical drive unit 210, is input to the gate terminal of the reset unit 126 from the block vertical drive unit 210.

[0062] 6, a reset control signal φRST is input to the gate terminal of the reset unit 126 immediately before the charges accumulated on the output sides of the photoelectric conversion units 120L and 120R are transferred to the memories 124L and 124R in accordance with exposure timing that is set differently for each pixel block 142. This is synchronized with the input of control signals φTX3L and φTX3R to the second transfer units 123L-3 and 123R-3. As a result, the reset unit 126 and the second transfer units 123L-3 and 123R-3 simultaneously go high, and the charges remaining in the memories 124L and 124R and the accumulation unit 125 are discharged to the power supply wiring, resetting the memories 124L and 124R.

[0063] When pixel signals are read out, a global reset control signal φRST common to the plurality of pixel blocks 142 is input from the vertical drive unit 62 to the gate terminal of the reset unit 126 .

[0064] 6 , the reset control signal φRST is input to the gate terminal of the reset unit 126 twice during pixel signal readout in each frame, in accordance with a readout timing common to the plurality of pixel blocks 142. More specifically, in each frame, the reset control signal φRST is input to the gate terminal of the reset unit 126 once immediately before the control signal φTX3L is input to the gate terminal of the second transfer unit 123L-3, and once immediately before the control signal φTX3R is input to the gate terminal of the second transfer unit 123R-3. This causes the reset unit 126 to go high at each timing, and the charge remaining in the accumulation unit 125 is discharged to the power supply wiring.

[0065] The pixel output unit 127 outputs a signal based on the potential of the storage unit 125 to the signal line 122. The pixel output unit 127 has an amplifier unit 128 and a selection unit 129. The amplifier unit 128 has a gate terminal connected to the storage unit 125, a drain terminal connected to a power supply wiring that supplies a power supply voltage VDD, and a source terminal connected to the drain terminal of the selection unit 129.

[0066] The selection unit 129 controls the electrical connection between the pixel 202 and the signal line 122. When the selection unit 129 electrically connects the pixel 202 and the signal line 122, pixel signals corresponding to the photoelectric conversion units 120L and 120R are sequentially output from the pixel 202 to the signal line 122. A global selection control signal φSEL, which is common to a plurality of pixel blocks 142, is input from the vertical drive unit 62 to a gate terminal of the selection unit 129. A source terminal of the selection unit 129 is connected to the load current source 121.

[0067] 6 , once during readout of pixel signals in each frame, a selection control signal φSEL is input to the gate terminal of the selection unit 129 in accordance with a readout timing common to the plurality of pixel blocks 142. This causes the selection unit 129 to go high, and while the high state is maintained, the charges accumulated in the memories 124L and 124R during exposure of the photoelectric conversion units 120L and 120R are sequentially transferred to the accumulation unit 125, and a pixel signal of an intensity corresponding to the amount of charge in the accumulation unit 125 at each timing at which the charge is transferred is output from the power supply wiring to the signal line 122.

[0068] The load current source 121 supplies a current to the signal line 122. According to the above configuration, it can be said that the exposure time is mainly controlled by the control signals φTX1L, φTX1R, control signals φTX3L, φTX3R, and control signals φTX2L, φTX2R, and the readout timing is mainly controlled by the control signals φTX3L, φTX3R, and selection control signal φSEL.

[0069] In other words, the pixel 202 includes at least one photoelectric conversion unit 120, a pixel output unit 127 as a readout unit that reads out an image signal from the at least one photoelectric conversion unit 120 to a signal line 122, etc. The pixel 202 can also be said to be the smallest unit of a circuit that outputs pixel signals that constitute an image to the signal line 122.

[0070] In this embodiment, the discharge units 123L-2 and 123R-2 have been described as discharging the charges of the photoelectric conversion units 120L and 120R to the power supply wiring to which the power supply voltage VDD is supplied, but they may also be discharged to a power supply wiring to which a power supply voltage different from the power supply voltage VDD is supplied. Also, the discharge units 123L-2 and 123R-2 may be omitted, in which case only the first transfer units 123L-1 and 123R-1 may be connected to the output sides of the photoelectric conversion units 120L and 120R. In this case, instead of the discharge units 123L-2 and 123R-2 discharging the charges of the photoelectric conversion units 120L and 120R to the power supply wiring, the charges accumulated in the photoelectric conversion units 120L and 120R may be discharged to the power supply wiring to which the power supply voltage VDD is supplied by the first transfer units 123L-1 and 123R-1, the second transfer units 123L-3 and 123R-3, and the reset unit 126. At this time, signals specific to the pixel block 142 corresponding to the block vertical drive unit 210, i.e., local signals, are input from the block vertical drive unit 210 to the gate terminals of the first transfer units 123L-1 and 123R-1, the second transfer units 123L-3 and 123R-3, and the reset unit 126 in synchronization with each other.

[0071] As described above, in the present embodiment, the reset control signal φRST is input to the gate terminal of the reset unit 126 twice during readout of pixel signals in each frame, in accordance with a readout timing common to the plurality of pixel blocks 142. Alternatively, the reset control signal φRST may be input to the gate terminal of the reset unit 126 only once during readout of pixel signals in each frame, that is, just before the control signal φTX3L is input to the gate terminal of the second transfer unit 123L-3.

[0072] Fig. 7 is a diagram illustrating an example of an image captured of a subject 55 with depth. Fig. 8 is an example of a processing flow of the image sensor 100 according to the first embodiment.

[0073] 7, subject 55 having depth extends long from the lower right to the upper left of the page, and pixel block 142A located at the lower right of the page captures the portion of subject 55 located closest to image sensor 100. Pixel block 142B located at the upper left of the page captures the portion of subject 55 located closest to image sensor 100. Pixel block 142C located near the center of the page captures the portion of subject 55 between the portion closest to image sensor 142A and the portion closest to image sensor 142B.

[0074] As an example, the image sensor 100 starts the processing flow by bringing a subject 55 with depth as shown in FIG. 7 into the imaging field of view.

[0075] The image sensor 100 pre-images one frame of the subject 55 (step S100). Specifically, the image sensor 100 images one frame of the subject 55 without moving the focus lens. At this time, the image sensor 100 does not control the exposure timing of the multiple pixel blocks 142 to be different, that is, the entire pixel unit 50 is exposed at the same timing.

[0076] The image sensor 100 calculates the defocus amount of the optical system 20 for each pixel block 142 based on the phase difference information output from each pixel block 142 (step S101), and calculates the exposure timing of each pixel block 142 from the defocus amount (step S103).

[0077] The image sensor 100 independently exposes and images each pixel block 142 during the focus scan period in which the focus lens moves from a first position to a second position, for example, from the front to the back (step S105), and outputs image information from each pixel block 142 to the image processing unit 24, etc. via the output unit 70, while also acquiring phase difference information from each pixel block 142 (step S107).

[0078] If there is a next frame (step S109: YES), the image sensor 100 returns to step S101. If there is no next frame (step S109: NO), the image sensor 100 ends the processing flow.

[0079] In this way, the image sensor 100 according to this embodiment differentiates the exposure timing of, for example, three pixel blocks 142A, 142B, and 142C shown in Fig. 7 during the movement period in which the focus lens moves from the first position to the second position. For example, the image sensor 100 sets the exposure timing of each of the three pixel blocks 142A, 142B, and 142C during the movement period based on the defocus amount of the optical system 20 for each pixel block corresponding to the three pixel blocks 142A, 142B, and 142C.

[0080] The defocus amount may be calculated from the phase difference information output in the pre-imaging, as in the processing flow of Figure 8, or may be calculated from the phase difference information output in the imaging of one to several frames before in a plurality of consecutive frames.

[0081] 9 shows an example of a timing chart outlining the operation timing of the image sensor 100 according to the first embodiment. With the time axis extending to the right of the page, Fig. 9 shows the exposure timing and signal readout timing of three pixel blocks 142A, 142B, and 142C for two consecutive frames in the image sensor 100 capturing an image of the deep subject 55 shown in Fig. 7. Fig. 9 also shows the position of the focus lens of the optical system 20 for the two frames.

[0082] The lens driver 23 of the imaging device 10 moves the focus lens from a first position to a second position, for example, from the front to the back, to change the focal position of the optical system 20. The imaging element 100 controls the exposure timing of the three pixel blocks 142A, 142B, and 142C so that they are different during the movement period in which the focus lens moves from the front to the back. For example, as described in the example of Figure 7, the imaging element 100 may control the exposure so that the pixel block 142A, which is located closest to the imaging device 10 and is included in the imaging field of view, is exposed in this order, starting with the pixel block 142C and then the pixel block 142B.

[0083] The drive control unit 60 of the image sensor 100 may set the exposure timing of each of the three pixel blocks 142A, 142B, and 142C in the later movement period based on the defocus amount of the optical system 20 in pixel block units corresponding to each of the three pixel blocks 142A, 142B, and 142C when imaged in the earlier movement period of two consecutive movement periods.

[0084] For example, the image sensor 100 may calculate, from the defocus amount of the optical system 20 for each pixel block corresponding to each of the three pixel blocks 142A, 142B, and 142C that captured the first frame, a focus position at which the optical system 20 focuses on each part of the subject 55 captured by each of the three pixel blocks 142A, 142B, and 142C. The image sensor 100 may calculate the time within one frame at which the focus lens is positioned on the optical axis 22 corresponding to each of the three focus positions calculated for the three pixel blocks 142A, 142B, and 142C, and set each time as the exposure timing for the corresponding pixel block. The image sensor 100 may capture the second frame in accordance with the exposure timing set for each of the three pixel blocks 142A, 142B, and 142C. As a result, in the second frame, the image sensor 100 may capture images in which each of the three pixel blocks 142A, 142B, and 142C is in focus on a different part of the subject 55 that is at a different distance from the image sensor 100.

[0085] In the first embodiment described above using FIGS. 1 to 9 , each of the multiple pixels in the pixel unit 50 is described as a 2PD having two photoelectric conversion units, one on the left and one on the right. Alternatively, at least any pair of pixels among the multiple pixels included in at least any of the multiple pixel blocks in the pixel unit 50 may each have only one photoelectric conversion unit. In this case, the pair of photoelectric conversion units provided in the pair of pixels may be symmetrically half-shielded to provide phase difference information; that is, the pixel block including the pair of pixels may be a discrete half-shielded type. Note that the pixel signals output from the pair of pixels can be used not only as phase difference information but also as image information. Specifically, the pair of pixels are arranged close to each other, and image information can be provided by adding up the pixel signals output separately.

[0086] Fig. 10 shows an example of the circuit configuration of the pixel 202-1, and Fig. 11 shows an example of a timing chart illustrating the operation timing of one pixel block 142 including the pixel 202-1.

[0087] Unlike the pixel 202 shown in FIG. 5, the pixel 202-1 does not include the photoelectric conversion units 120L and 120R, but instead includes only one photoelectric conversion unit 120. Also, unlike the pixel 202 shown in FIG. 5, the pixel 202-1 does not include the first transfer units 123L-1 and 123R-1, the discharge units 123L-2 and 123R-2, the second transfer units 123L-3 and 123R-3, or the memories 124L and 124R. Instead, the pixel 202-1 includes a transfer unit 123-1 and a discharge unit 123-2. As a result, as shown in FIG. 11, the timing chart showing the operation timing of each component of the pixel 202-1 is simpler than the timing chart showing the operation timing of each component of the pixel 202 shown in FIG. 6. The functions and operation timings of these components included in pixel 202-1 and components such as the storage unit 125 that are common to pixel 202 are generally similar to those of pixel 202, and therefore redundant explanations will be omitted. Note that the photoelectric conversion unit 120 is an example of a first photoelectric conversion unit or a second photoelectric conversion unit.

[0088] Fig. 12 shows another example of a timing chart that outlines the operation timing of the image sensor 100 according to the first embodiment. The timing chart shown in Fig. 12 shows, with the time axis extending to the right as one faces the page, the exposure timing and signal readout timing of three consecutive frames of the three pixel blocks 142A, 142B, and 142C in the image sensor 100 that captured the deep subject 55 shown in Fig. 7. Fig. 12 also shows the position of the focus lens of the optical system 20 during these three frames.

[0089] Lens driver 23 of imaging device 10 may move the focus lens from a first position to a second position during a movement period to change the focal position of optical system 20, and immediately after the movement period, may move the focus lens from the second position to the first position to change the focal position of optical system 20. For example, as shown in Fig. 12, lens driver 23 may move the focus lens from the front to the back during three consecutive movement periods, from the back to the front during a first movement period, and from the front to the back during a third movement period.

[0090] In this case, immediately after the first movement period described above, i.e., during the period when the focus lens moves from the second position to the first position, the drive control unit 60 of the image sensor 100 causes the photoelectric conversion units of each pixel block 142 to sequentially start accumulating electric charges in accordance with the inverted exposure timing of the exposure timing set for each pixel block 142 during the first movement period. For example, as shown in Fig. 12, during three consecutive movement periods, the drive control unit 60 may expose the pixel blocks 142A, 142C, and 142B in this order during the first movement period, expose the pixel blocks 142B, 142C, and 142A in this order during the second movement period, and expose the pixel blocks 142A, 142C, and 142B in this order during the third movement period.

[0091] According to the first embodiment described above with reference to Figures 1 to 12, the image sensor 100 controls the timings at which the accumulation of charges converted by the multiple photoelectric conversion units begins to differ from one another during the period until the focus lens moves from the first position to the second position. In addition to the above function, the image sensor 100 may also have any of the multiple functions described below. Note that in the multiple examples described below, for the purpose of clarity, some of the configurations in the above-described embodiments may be omitted from description and illustration.

[0092] Fig. 13 is a diagram illustrating an example of an image obtained by capturing images of a plurality of subjects 55A, 55B, and 55C having different reflectances. Fig. 14 is another example of a processing flow of the image sensor 100 according to the first embodiment.

[0093] 13, a person's legs, which reflect light relatively little, are imaged as subject 55A in pixel block 142A. The exterior surfaces of a vehicle and a house, which reflect light relatively easily, are imaged as subjects 55B and 55C in pixel block 142B and pixel block 142C. The reflectances of subjects 55B and 55C are similar, while the reflectance of subject 55A is relatively low. Furthermore, the distance from image sensor 100 to each subject is closest to subject 55A, followed by subjects 55A, 55C, and 55B, in that order.

[0094] As an example, the imaging device 10 including the imaging element 100 may function as a pair of stereo cameras separated by a base line length, and may perform structured light ranging on the subject 55. In this case, light is irradiated from a projector onto the front of the subject 55, and the imaging device 10 captures the light as a stereo camera.

[0095] In the image sensor 100 according to this embodiment, during the period in which the focus lens moves from the first position to the second position, the exposure timing of the three pixel blocks 142A, 142B, and 142C shown in Fig. 13 may be made different, and the exposure times of the three pixel blocks 142A, 142B, and 142C may also be made different, thereby producing a wide dynamic range.

[0096] As an example, the image sensor 100 starts the processing flow by preparing a pattern for calibration imaging and placing the pattern at each focus position within the imaging field of view of the image sensor 100 in order to obtain calibration data for each focus position.

[0097] Instead of placing an actual object as the pattern, the pattern may be projected from a projector onto a screen placed at each focus position. As an example, the positions at which the pattern is placed may be the positions of the three subjects 55A, 55B, and 55C that are the targets of structured light ranging. That is, the patterns may be placed at three positions that are the same distances from the image sensor 100 as the distances from the image sensor 100 to the three subjects 55A, 55B, and 55C that are within the imaging field of view of the image sensor 100.

[0098] The calibration data may be data for calculating the amount of defocus required for structured light ranging. In structured light ranging, the amount of defocus may be converted into distance using information indicating the positional relationship between image capture device 10 and the projector and the state of optical system 20 of image capture device 10.

[0099] The image sensor 100 focuses on the patterns arranged at each focus position and captures calibration images, thereby acquiring calibration data for each focus position (step S200). The image sensor 100 performs calibration image capture once, for example, by focusing on the patterns arranged at the three positions described above, and repeatedly using the calibration data acquired thereby, making it possible to perform structured light ranging of the three subjects 55A, 55B, and 55C. Note that if the position of the subject 55A or the like changes, it is necessary to change the state of the optical system 20 of the image sensor 10, and therefore calibration image capture may be performed again.

[0100] 13, three subjects 55A, 55B, and 55C, which are different from one another in distance and reflectance to the image sensor 100, are placed within the image sensor field of view, and light is projected from a projector onto the front of each subject, starting structured light ranging. The image sensor 100 calculates the defocus amount of the optical system 20 for each pixel block 142 based on the phase difference information from each pixel block 142 acquired as calibration data (step S201), and calculates the exposure timing for each pixel block 142 from the defocus amount (step S203).

[0101] The image sensor 100 independently exposes and images each pixel block 142 during the focus scan period in which the focus lens moves from a first position to a second position, for example, from the front to the back (step S205), and outputs image information from each pixel block 142 to the image processing unit 24, etc. via the output unit 70, while also acquiring phase difference information from each pixel block 142 (step S207).

[0102] If there is a next frame (step S209: YES), the image sensor 100 returns to step S201. If there is no next frame (step S209: NO), the image sensor 100 ends the processing flow.

[0103] In this way, the image sensor 100 sets different exposure timings for each of the three pixel blocks 142A, 142B, and 142C during the movement period based on the defocus amount of the optical system 20 for each pixel block corresponding to the three pixel blocks 142A, 142B, and 142C. The image sensor 100 may further set different exposure times for the three pixel blocks 142A, 142B, and 142C based on the reflectance of the front surface of each of the subjects 55A, 55B, and 55C that fall within the angle of view of each of the three pixel blocks 142A, 142B, and 142C.

[0104] The defocus amount may be calculated from the phase difference information output in the calibration imaging, as in the processing flow of Figure 14, or may be calculated from the phase difference information output in the imaging of one to several frames before in a plurality of consecutive frames.

[0105] Figure 15 shows another example of a timing chart that outlines the operation timing of the image sensor 100 according to the first embodiment. With the time axis extending to the right of the page, Figure 15 shows the exposure timing and signal readout timing of the three pixel blocks 142A, 142B, and 142C for two consecutive frames in the image sensor 100 that captured images of the three subjects 55A, 55B, and 55C shown in Figure 13. Figure 15 also shows the position of the focus lens of the optical system 20 for the two frames.

[0106] The lens driver 23 of the imaging device 10 moves the focus lens from a first position to a second position, for example, from the front to the back, to change the focal position of the optical system 20. The imaging element 100 controls the exposure timing and exposure time of the three pixel blocks 142A, 142B, and 142C so that they are different during the movement period in which the focus lens moves from the front to the back. For example, as described in the example of Figure 13, the imaging element 100 may control the exposure so that the pixel block 142A, which is located closest to the imaging device 10 and is included in the imaging field of view, is exposed in this order, starting with the pixel block 142C and then the pixel block 142B.

[0107] The drive control unit 60 of the image sensor 100 may set the exposure timing of each of the three pixel blocks 142A, 142B, and 142C in the later movement period based on the defocus amount of each of the three pixel blocks 142A, 142B, and 142C when imaged in the earlier movement period of two consecutive movement periods.

[0108] The drive control unit 60 also sets an exposure time within the movement period separately for each of the plurality of pixel blocks. For example, when measuring the distance to each of the plurality of subjects 55 using reflected light from the plurality of subjects 55, the drive control unit 60 sets a longer exposure time for the subject 55 with lower light reflectance among the plurality of subjects 55.

[0109] For example, the image sensor 100 may calculate, from the defocus amount of the optical system 20 for each pixel block corresponding to each of the three pixel blocks 142A, 142B, and 142C that captured the first frame, focus positions at which the subjects 55A, 55B, and 55C captured by each of the three pixel blocks 142A, 142B, and 142C are in focus. The image sensor 100 may calculate the times within one frame at which the focus lens is positioned on the optical axis 22 corresponding to each of the three focus positions calculated for the three pixel blocks 142A, 142B, and 142C, and set each time as the exposure timing for each of the three pixel blocks 142A, 142B, and 142C. The image sensor 100 may capture the second frame in accordance with the exposure timing set for each of the three pixel blocks 142A, 142B, and 142C. As a result, in the second frame, the image sensor 100 may capture images in which the subjects 55A, 55B, and 55C, which are at different distances from the image sensor 100, are focused on in each of the three pixel blocks 142A, 142B, and 142C.

[0110] As described above, the reflectances of subjects 55B and 55C are approximately the same, while the reflectance of subject 55A is relatively low. Therefore, as shown in Fig. 15, the image sensor 100 may set the exposure time of pixel block 142A capturing an image of subject 55A to be longer than the exposure times of pixel block 142B capturing an image of subject 55B and pixel block 142C capturing an image of subject 55C.

[0111] As another example, the image sensor 100 may fix the aperture of the optical system 20 at its maximum open position depending on the reflectance of the subject 55, and acquire images for each depth of field corresponding to the subject 55. In this case, since the depth of field becomes very shallow, the image sensor 100 may perform calibration more frequently than when the aperture is not fully opened. Note that the number of calibration images required varies depending on the F-number of the optical system 20, i.e., depending on the depth of field; the brighter the F-number, the shallower the depth of field, so it may be necessary to perform calibration image capture many times in detail.

[0112] 13 to 15 does not generate an image of subject 55A or the like, but even in the examples in which an image of subject 55 or the like is generated as described with reference to Figures 1 to 12, the exposure time may be made different for each pixel block 142. That is, even when image sensor 100 outputs image information to image processing unit 24 or the like via output unit 70 in order to generate an image of subject 55 or the like, it may make the exposure timing and exposure time of multiple pixel blocks 142 different within the period of one frame in which the focus lens moves from the first position to the second position.

[0113] In this case, the drive control unit 60 of the image sensor 100 may generate an exposure map that maps the exposure timing and exposure time of the first pixel block and the exposure timing and exposure time of the second pixel block to each position of the first pixel block and the second pixel block within the field of view. The exposure map indicates which pixel has which exposure time and may be required when generating an HDR image. Therefore, the drive control unit 60 may output the generated exposure map, along with image information, to the image processing unit 24 or the like via the output unit 70. Note that the drive control unit 60 may use the generated exposure map to set the exposure times of multiple pixel blocks 142.

[0114] 1 to 12 , in the embodiment for generating an image of a subject 55 or the like, the image processing unit 24 or the like of the imaging device 10 may receive, from the image sensor 100, image information as well as information indicating the exposure timing for each pixel block 142. In this case, the image processing unit 24 or the like may perform breathing correction processing based on this information to generate an image in which the boundaries between adjacent pixel blocks 142 are inconspicuous. Note that, in consideration of performing breathing correction processing during image generation, the image sensor 100 may impose restrictions so that the focus position does not change drastically between adjacent pixel blocks 142, i.e., so that the angle of view does not change drastically.

[0115] 1 to 15 , the drive control unit 60 of the image sensor 100 may further detect the acceleration of the subject 55, which is a moving object, and predict the focus position of the subject 55 for each pixel block 142. When the defocus amount, i.e., the focus position, is calculated for each frame to adjust the focus, if the subject 55 is a moving object, the subject will move slightly in the next frame, causing the focus position to change. Therefore, the image sensor 100 may control the exposure timing in accordance with the focus position calculated based on the defocus amount and the focus position predicted from the acceleration of the subject 55.

[0116] In the embodiment described using FIGS. 1 to 15 , the drive control unit 60 of the image sensor 100 calculates the defocus amount of the optical system 20 for each pixel or pixel block based on phase difference information for each pixel or pixel block. Alternatively, the image sensor 100 may output the phase difference information, along with image information, to the control unit 26 of the imaging device 10 via the output unit 70, causing the control unit 26 to calculate the defocus amount. The image sensor 100 may also cause the control unit 26 to set an exposure time for each pixel or pixel block, and may further generate the above-described exposure map based on the set exposure time. Note that when the image sensor 100 calculates the defocus amount or generates the exposure map, the amount of data output to the outside of the image sensor 100 can be reduced, processing time can be shortened, and sufficient tracking can be achieved, compared to these alternative examples.

[0117] In this case, the image sensor 100 may control the exposure timing to be different for each pixel or each pixel block based on information about the defocus amount acquired from the control unit 26. The image sensor 100 may also control the exposure time to be different for each pixel or each pixel block based on an exposure map acquired from the control unit 26. Note that the image sensor 100 does not necessarily have to set the exposure timing based on the defocus amount, and may control the exposure timing to be different for each pixel or each pixel block in accordance with an instruction signal acquired from the control unit 26.

[0118] 1 to 15 , the image sensor 100 is described as not outputting information about the defocus amount of the optical system 20 for each pixel or each pixel block, calculated based on phase difference information for each pixel or each pixel block, to the outside of the image sensor 100. In this case, the image sensor 100 simply outputs only the image information to the image processing unit 24 of the imaging device 10 via the output unit 70, which can reduce data consumption and increase speed. However, if some processing is performed outside the image sensor 100, for example, in the control unit 26 or the image processing unit 24, using the defocus amount information together with the image information, the defocus amount information may be output to the outside of the image sensor 100.

[0119] 1 to 15, the multiple pixels in the pixel unit 50 are all described as pixels for image plane phase difference. Alternatively, the multiple pixels in the pixel unit 50 may not include pixels for image plane phase difference. In this case, the image sensor 100 may acquire information indicating the exposure timing for each pixel or each block from outside the image sensor 100, and may control the exposure timing for each pixel or each block to be different based on the information.

[0120] In the embodiment described using Figures 1 to 15, the multiple pixels in the pixel unit 50 are all described as pixels for image plane phase difference and pixels for images, but instead, the multiple pixels in the pixel unit 50 may include pixels for determining a subject 55 of interest within the imaging field of view.

[0121] 1 to 15 , the drive control unit 60 of the image sensor 100 may set a longer exposure time for the first pixel block and the second pixel block that include a relatively large portion of the object 55 that satisfies a predetermined condition. The object 55 that satisfies the predetermined condition may be an object 55 of interest, such as an object 55 located at the center of the imaging field of view, an object 55 that moves relatively quickly within the imaging field of view, or an object 55 specified by the user of the image sensor 10 via the operation unit 30. The image sensor 100 may set a longer exposure time for a pixel block 142 that includes a larger portion of the object 55 of interest within the imaging field of view, thereby capturing a clearer image of the object 55 of interest.

[0122] The drive control unit 60 of the image sensor 100 may further determine the first and second positions, which define the movement range of the focus lens within the movement period, according to the position of the subject 55 that satisfies a predetermined condition. As an example, the drive control unit 60 may vary the exposure timing for each pixel or pixel block within the period in which the focus lens moves in one direction, and may also change the movement range of the focus lens according to the position of the subject 55. That is, the drive control unit 60 may adjust the movement range of the focus lens according to the position of the subject 55 that is the target of attention. The drive control unit 60 may change the movement range of the focus lens for each frame.

[0123] For example, when the target subject 55 is determined, the drive control unit 60 may set the movement range of the focus lens to a range in which the target subject 55 and areas before and after it are in focus. The drive control unit 60 may widen or narrow the movement range of the focus lens, or may gradually widen or narrow it.

[0124] Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications and improvements can be made to the above embodiments. It is clear from the claims that such modifications and improvements can also be included within the technical scope of the present invention.

[0125] It should be noted that the order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before," "prior to," etc., and that the processes can be performed in any order unless the output of a previous process is used in a subsequent process. Even if the operational flow in the claims, specifications, and drawings is described using "first," "next," etc. for convenience, this does not mean that the processes must be performed in this order. [Other Possible Items] [Item 1] An image sensor comprising: a first semiconductor substrate having a first photoelectric conversion unit that converts light emitted from an optical system having a focus lens into electric charges, and a second photoelectric conversion unit that converts light emitted from the optical system into electric charges, and a second semiconductor substrate that is stacked together with the first semiconductor substrate and has a drive control unit that controls so that a first timing at which accumulation of electric charges converted by the first photoelectric conversion unit starts and a second timing at which accumulation of electric charges converted by the second photoelectric conversion unit starts are different timings during a period in which the focus lens moves from a first position to a second position. [Item 2] An image sensor according to item 1, wherein the drive control unit starts accumulation of electric charges converted by the first photoelectric conversion unit and accumulation of electric charges converted by the second photoelectric conversion unit based on a defocus amount of the optical system. [Item 3] The image sensor according to Item 2, wherein the drive control unit starts accumulation of the electric charges converted by the first photoelectric conversion unit and accumulation of the electric charges converted by the second photoelectric conversion unit based on information relating to the relationship between the period and a focus position of the optical system. [Item 4] The image sensor according to Item 1, wherein the first semiconductor substrate includes a first pixel block having a plurality of photoelectric conversion units including the first photoelectric conversion unit, and a second pixel block having a plurality of photoelectric conversion units including the second photoelectric conversion unit, and the plurality of photoelectric conversion units in each of the first pixel block and the second pixel block include a pixel for image plane phase difference for calculating a defocus amount of the optical system on a pixel block basis.[Item 5] The image sensor according to Item 4, wherein the drive control unit calculates the defocus amount for each of the first pixel block and the second pixel block from outputs of the pixels for image plane phase difference. [Item 6] The image sensor according to Item 1, wherein the first semiconductor substrate includes a first pixel block having a plurality of photoelectric conversion units including the first photoelectric conversion unit, and a second pixel block having a plurality of photoelectric conversion units including the second photoelectric conversion unit, and the drive control unit sets the first timing and the second timing in a later of two consecutive periods based on defocus amounts of the optical system in pixel block units corresponding to the first pixel block and the second pixel block when imaged in a previous of the two consecutive periods. [Item 7] The image sensor according to Item 1, wherein the first semiconductor substrate includes a first pixel block having a plurality of photoelectric conversion units including the first photoelectric conversion unit, and a second pixel block having a plurality of photoelectric conversion units including the second photoelectric conversion unit, and the drive control unit is provided corresponding to each of the first pixel block and the second pixel block. [Item 8] The image sensor according to Item 7, wherein, in a stacking direction of the first semiconductor substrate and the second semiconductor substrate, the drive control unit corresponding to the first pixel block is arranged in a region of the second semiconductor substrate that overlaps with the first pixel block, and the drive control unit corresponding to the second pixel block is arranged in a region of the second semiconductor substrate that overlaps with the second pixel block. [Item 9] In the image sensor described in Item 1, the first semiconductor substrate includes a first pixel block having a plurality of photoelectric conversion units including the first photoelectric conversion unit, and a second pixel block having a plurality of photoelectric conversion units including the second photoelectric conversion unit, and the drive control unit sets an exposure time within the period separately for each of the first pixel block and the second pixel block.[Item 10] The image sensor according to Item 9, wherein, when measuring the distance to each of the plurality of subjects using reflected light from the plurality of subjects, the drive control unit sets the exposure time longer for a subject with lower light reflectance among the plurality of subjects. [Item 11] The image sensor according to Item 9, wherein the drive control unit sets the exposure time longer for a pixel block, of the first pixel block and the second pixel block, that includes a relatively large number of subjects that satisfy a predetermined condition. [Item 12] The image sensor according to Item 9, wherein the drive control unit generates an exposure map that maps the first timing and the exposure time of the first pixel block and the second timing and the exposure time of the second pixel block to each position of the first pixel block and the second pixel block within a field of view. [Item 13] An imaging device comprising: the image sensor according to item 1; an optical system having the focus lens; and a lens driver that controls a focus position of the optical system, wherein the lens driver moves the focus lens from the first position to the second position within the period to change the focus position. [Item 14] In the imaging device according to item 13, the drive controller causes the first photoelectric conversion unit and the second photoelectric conversion unit to sequentially start accumulating electric charges in accordance with the first timing and the second timing within the period in synchronization with movement of the focus lens from the first position to the second position. [Item 15] In the imaging device described in Item 13, the lens driving unit moves the focus lens from the first position to the second position within the period to change the focal position, and immediately after the period moves the focus lens from the second position to the first position to change the focal position, and immediately after the period, the driving control unit starts accumulating electric charges sequentially in each of the second photoelectric conversion unit and the first photoelectric conversion unit in accordance with the first timing and the second timing set during the period, which are inverted.[Item 16] The imaging device according to Item 13, wherein the drive control unit determines the first position and the second position, which define a movement range of the focus lens within the period, in accordance with the position of a subject that satisfies a predetermined condition. [Item 17] An imaging method for an imaging device including a first photoelectric conversion unit that converts light emitted from an optical system having a focus lens into an electric charge, and a second photoelectric conversion unit that converts light emitted from the optical system into an electric charge, the imaging method comprising: moving the focus lens from a first position to a second position to change a focus position of the optical system; and controlling the focus lens so that a first timing at which accumulation of the electric charge converted by the first photoelectric conversion unit starts and a second timing at which accumulation of the electric charge converted by the second photoelectric conversion unit starts are different during a period from the first position to the second position.

[0126] 10 Imaging device 20 Optical system 22 Optical axis 23 Lens driving unit 24 Image processing unit 26 Control unit 28 Recording unit 30 Operation unit 32 Display unit 34 Power supply unit 36 ​​Bus line 50 Pixel unit 55, 55A, 55B, 55C Object 58 Output unit 60 Drive control unit 62 Vertical driving unit 64 Signal processing unit 68 Horizontal driving unit 100 Imaging element 120, 120L, 120R Photoelectric conversion unit 121 Load current source 122 Signal line 123-1 Transfer unit 123L-1, 123R-1 First transfer unit 123-2, 123L-2, 123R-2 Discharge unit 123L-3, 123R-3 Second transfer unit 124L, 124R Memory 125 Storage section 126 Reset section 127 Pixel output section 128 Amplification section 129 Selection section 140 First semiconductor substrate 142, 142A, 142B, 142C Pixel block 150 Second semiconductor substrate 152 Global vertical drive section 154 Global horizontal drive section 162 Signal processing block 200 Unit block 202, 202-1 Pixel 210 Block vertical drive section 220 Signal processing circuit group 222 Signal processing circuit 224 Conversion section 226 Memory section 500 Peripheral circuit section 502 Horizontal line 70 Output section

Claims

1. An imaging element comprising: a first photoelectric conversion unit that converts light emitted from an optical system having a focus lens into electric charges; a second photoelectric conversion unit that converts light emitted from the optical system into electric charges; and a drive control unit that controls the first timing to start accumulating the electric charges converted by the first photoelectric conversion unit and the second timing to start accumulating the electric charges converted by the second photoelectric conversion unit so that they are different timings during the period when the focus lens moves from a first position to a second position.

2. An imaging device according to claim 1, wherein the drive control unit acquires information relating to the defocus amount of the focus lens.

3. An imaging device according to claim 2, wherein the drive control section calculates information relating to the defocus amount.

4. An imaging device according to claim 2, wherein the drive control section determines the first timing and the second timing based on information relating to the defocus amount.

5. An imaging element according to claim 4, wherein the drive control unit controls the first timing to be the timing when the focus lens is focused on a subject that emits light that is incident on the first photoelectric conversion unit, and the second timing to be the timing when the focus lens is focused on a subject that emits light that is incident on the second photoelectric conversion unit.

6. An imaging element according to claim 1, wherein the drive control unit determines a first accumulation time of the electric charge converted by the first photoelectric conversion unit and a second accumulation time of the electric charge converted by the second photoelectric conversion unit.

7. An imaging device according to claim 6, wherein the drive control section determines the first accumulation time and the second accumulation time based on information relating to the subject.

8. An imaging element according to claim 7, wherein the drive control unit makes the first accumulation time longer than the second accumulation time when the first photoelectric conversion unit converts light from a subject that satisfies predetermined conditions into electric charges.

9. An image sensor according to claim 3, comprising a photoelectric conversion unit that converts light emitted from the optical system into electric charges and a third photoelectric conversion unit that outputs a first signal for focus detection of the optical system, and wherein the drive control unit calculates information relating to the amount of defocus using a signal based on the electric charges converted by the third photoelectric conversion unit during a period prior to the first timing.

10. An image sensor according to claim 9, further comprising a fourth photoelectric conversion unit which is a photoelectric conversion unit that converts light emitted from the optical system into electric charges and outputs a second signal for focus detection of the optical system, and wherein the drive control unit calculates information relating to the amount of defocus using a signal based on the electric charges converted by the fourth photoelectric conversion unit during a period prior to the second timing.

11. An imaging device according to claim 10, wherein the drive control section determines the first timing using the first signal and determines the second timing using the second signal.

12. An imaging element according to claim 10, wherein the drive control section has a first control section that controls the accumulation of electric charge in the first photoelectric conversion section, and a second control section that controls the accumulation of electric charge in the second photoelectric conversion section.

13. An imaging element according to claim 12, wherein the first control unit controls the accumulation of charges in the third photoelectric conversion unit, and the second control unit controls the accumulation of charges in the fourth photoelectric conversion unit.

14. An imaging element according to any one of claims 1 to 12, comprising: a first semiconductor substrate on which the first photoelectric conversion unit and the second photoelectric conversion unit are arranged; and a second semiconductor substrate stacked on the first semiconductor substrate, on which the drive control unit is arranged.

15. An imaging device comprising the imaging element according to claim 14.

16. An imaging element according to claim 13, comprising: a first semiconductor substrate on which the first photoelectric conversion section and the second photoelectric conversion section are arranged; and a second semiconductor substrate stacked on the first semiconductor substrate, on which the first control section and the second control section are arranged.

17. An imaging element as described in claim 16, wherein the first photoelectric conversion unit is arranged at a position opposite the first control unit in the stacking direction in which the first semiconductor substrate and the second semiconductor substrate are stacked, and the second photoelectric conversion unit is arranged at a position opposite the second control unit in the stacking direction.

18. An imaging device comprising the imaging element according to claim 16 or 17.

Images