Global shutter image sensor with time-of-flight sensing capability

Abstract

An apparatus (20) for optical sensing includes a first matrix (36) of optical sensing elements (40) arranged on a semiconductor substrate (41). A second matrix of storage nodes is arranged on the substrate such that respective first and second storage nodes (SN) in the second matrix are disposed proximate each of the sensing elements within the first matrix. Switching circuitry (45) is coupled each of the sensing elements to transfer photocharge to the respective first and second storage nodes. Control circuitry (22) controls the switching circuitry in a depth sensing mode such that, within a series of detection cycles, each of the sensing elements and a first adjacent sensing element are connected together to the respective first storage node during a first detection interval, and each of the sensing elements and a second adjacent sensing element are connected together to the respective second storage node during a second detection interval.

Description

Technical Field

[0001] The present invention relates generally to imaging, and more particularly to optical sensors for both two-dimensional imaging and depth mapping. Background Technology

[0002] Some wearable and portable consumer devices, such as smartphones, augmented reality (AR) devices, virtual reality (VR) devices, and smart glasses, include both two-dimensional (2D) image sensors and optical depth mapping systems. Image sensors typically require high spatial resolution to capture high-quality images. At the same time, the physical size of the sensors is limited by the space requirements of these devices, resulting in small pixel sizes. Optical depth mapping systems can be used, for example, in conjunction with 2D imaging to perform reliable facial recognition.

[0003] Various methods for optical depth mapping are known in the prior art, which generate a three-dimensional (3D) profile of an object's surface by processing optical radiation reflected from the object. This 3D profile is also called a 3D map, depth map, or depth image, and depth mapping is also referred to as 3D mapping. (In the context of this specification and the claims, the terms "optical radiation" and "light" are used interchangeably to refer to any electromagnetic radiation in the visible, infrared, and ultraviolet spectral ranges.)

[0004] Some depth mapping systems operate by measuring the radiative direction and the time-of-flight (TOF) from points in the target scene. In a direct TOF (dTOF) system, a light emitter, such as a laser or an array of lasers, directs short pulses of light toward the scene. A receiver, such as a sensitive high-speed photodiode (e.g., an avalanche photodiode) or an array of such photodiodes, receives the light returning from the scene. Control circuitry measures the time delay between the emitted and received light pulses at each point in the scene (this time delay indicates the distance the beam has traveled, and therefore the depth of the object at that point) and uses the depth data thus extracted to generate a 3D map of the scene.

[0005] On the other hand, indirect TOF (iTOF) systems operate by modulating the amplitude of the emitted radiation beam at a certain carrier frequency and then measuring the phase shift of that carrier (at the modulated carrier frequency) in the radiation reflected back from the target scene. The phase shift can be measured by imaged onto an array of optical sensors and by gating or modulating the integration time of the sensors in the array in sync with the modulation of the emitted beam. The phase shift of the reflected radiation received from each point in the scene indicates the direction of radiation and the distance traveled from that point; however, the measurement may be blurred due to the folding of the carrier phase over the distance. Summary of the Invention

[0006] The embodiments of the present invention described below provide apparatus and methods for optical depth sensing.

[0007] Therefore, according to an embodiment of the present invention, an apparatus for optical sensing is provided, the apparatus comprising a semiconductor substrate and a first matrix of optical sensing elements, the first matrix being arranged in rows and columns on the substrate such that any given sensing element within the first matrix has a first adjacent sensing element and a second adjacent sensing element on different sides of the given sensing element in the first matrix. A second matrix of storage nodes is arranged on the substrate such that corresponding first storage nodes and corresponding second storage nodes in the second matrix are disposed near each of the sensing elements within the first matrix. A switching circuit is configured to couple each of the sensing elements to transfer photocharge to the corresponding first storage node and the corresponding second storage node. A control circuit is configured to control the switching circuit in a depth sensing mode such that, in a series of detection cycles including at least a first detection interval and a second detection interval in each cycle, each of the sensing elements and the first adjacent sensing element are connected together to the corresponding first storage node during the first detection interval, and each of the sensing elements and the second adjacent sensing element are connected together to the corresponding second storage node during the second detection interval. A readout circuit is configured to output the photocharge from the storage nodes.

[0008] In some embodiments, the apparatus includes: an illumination assembly configured to guide optical radiation toward a target scene while modulating the optical radiation using a carrier having a predetermined carrier frequency; and an objective lens optics configured to image the target scene onto a first matrix of the optical sensing elements. The control circuitry is configured to synchronize the detection cycle with the carrier frequency. In some of these embodiments, the control circuitry is configured to compare the photocharge output from the first and second storage nodes by the readout circuitry to calculate the depth coordinates of a point in the target scene.

[0009] In some embodiments, the control circuit is configured to change the corresponding phases of the first detection interval and the second detection interval relative to the carrier within the series of detection cycles, and to use the photocharge output within the changed phase to calculate the depth coordinates. In one embodiment, the first detection interval and the second detection interval have a phase separation of 180° relative to the carrier. In a disclosed specific embodiment, the control circuit is configured to control the switching circuit such that the first detection interval and the second detection interval have phases of 0° and 180° relative to the carrier, and that the storage nodes in at least some rows of the rows in the matrix collect the photocharge in third and fourth detection intervals having phases of 90° and 270° relative to the carrier.

[0010] Alternatively, the control circuit is configured to control the switching circuit such that the first detection interval and the second detection interval have phases of 0° and 180° relative to the carrier, and that the storage nodes in at least some of the rows of the matrix collect the photocharge in a third detection interval, a fourth detection interval, a fifth detection interval, and a sixth detection interval having phases of 60°, 120°, 240°, and 300° relative to the carrier.

[0011] Alternatively, the phase of the first detection interval and the phase of the second detection interval are reversed from row to row in consecutive rows of the second matrix.

[0012] In some embodiments, the control circuitry is further configured to control the switching circuitry in a two-dimensional (2D) sensing mode, such that each of the sensing elements is individually connected to a single corresponding memory node in its vicinity within the memory node. In one embodiment, the switching circuitry includes a shutter gate coupled to apply a global shutter to the sensing elements in the 2D sensing mode.

[0013] Alternatively, the switching circuit may include a first transfer switch and a second transfer switch respectively coupled between each sensing element in the sensing elements and the corresponding first storage node and the corresponding second storage node.

[0014] In some embodiments, the optical sensing elements are arranged such that the first adjacent sensing elements and the second adjacent sensing elements of any given sensing element in any given row of the first matrix are positioned on opposite sides of the given sensing element along the given row. In the disclosed embodiments, the storage nodes are staggered with the sensing elements along the rows of the first matrix.

[0015] According to an embodiment of the invention, a method for optical sensing is also provided, the method comprising arranging optical sensing elements in rows and columns on a semiconductor substrate such that any given sensing element within the first matrix has a first adjacent sensing element and a second adjacent sensing element on different sides of the given sensing element in the first matrix. A second matrix of storage nodes is arranged on the substrate such that corresponding first storage nodes and corresponding second storage nodes in the second matrix are disposed near each of the sensing elements within the first matrix. Each of the sensing elements is coupled to transfer photocharge to the corresponding first storage node and the corresponding second storage node via a switching circuit. The switching circuit is controlled in a depth sensing mode such that, in a series of detection cycles including at least a first detection interval and a second detection interval in each cycle, each of the sensing elements and the first adjacent sensing element are connected together to the corresponding first storage node during the first detection interval, and each of the sensing elements and the second adjacent sensing element are connected together to the corresponding second storage node during the second detection interval. The photocharge is output from the storage node.

[0016] The invention will be more fully understood from the following detailed description of embodiments thereof, taken in conjunction with the accompanying drawings, in which: Attached Figure Description

[0017] Figure 1 This is a block diagram schematically illustrating a combined depth mapping and 2D imaging apparatus according to an embodiment of the present invention;

[0018] Figure 2 This is a schematic circuit diagram of an optical sensing element array and associated circuit according to an embodiment of the present invention;

[0019] Figure 3a and Figure 3b This is a schematic circuit diagram of an optical sensing element array according to an embodiment of the present invention, showing the operation of the switching circuit in the array in iTOF depth sensing mode;

[0020] Figure 4a and Figure 4b This is a schematic front view of an image sensor according to an embodiment of the present invention, showing the phase detected by the storage node in the image sensor in iTOF depth sensing mode;

[0021] Figure 5 This is a schematic front view of an image sensor according to another embodiment of the present invention, showing the phase detected by the storage node in the image sensor in iTOF depth sensing mode;

[0022] Figure 6 This is a schematic front view of an image sensor according to another embodiment of the present invention, showing the phase detected by the storage node in the image sensor in iTOF depth sensing mode;

[0023] Figure 7 This is a schematic front view of an image sensor according to another embodiment of the present invention, showing the phase detected by the storage node in the image sensor in iTOF depth sensing mode;

[0024] Figure 8 This is a schematic circuit diagram of an optical sensing element array according to an embodiment of the present invention, illustrating the operation of the switching circuit in the array in 2D imaging mode; and

[0025] Figure 9 This is a schematic circuit diagram illustrating details of the sensing element and associated circuitry in an image sensor according to an embodiment of the present invention. Detailed Implementation

[0026] Overview

[0027] Optical indirect Time-of-Flight (iTOF) systems known in the art use multiple different acquisition phases in the receiver to measure the phase shift of the carrier in the light reflected from each point in the target scene. For this purpose, many iTOF systems use dedicated image sensing arrays, where each sensing element is individually gated to receive and integrate light during the corresponding phase of the carrier cycle. At least three different gated phases are required to measure the phase shift of the carrier in the received light relative to the transmitted beam. For practical reasons, most systems acquire light during four different gated phases.

[0028] In this typical image sensing array, optical sensing elements are arranged in a group of four sensing elements. Each optical sensing element in a given group integrates photocharge generated by light received within one or more corresponding storage nodes due to light received in one or more different corresponding detection intervals, which are synchronized at different phase angles relative to the carrier frequency, such as 0°, 90°, 180°, and 270°. Processing circuitry combines the corresponding signals (referred to as I0, I1, I2, I3, I4, I5, I6, I7, I8, I9, I10, I10, I11, I2 ... 90 I 180 and I 270 To extract the depth value, which is compared with the function tan -1 [(I 270 -I 90 ) / (I0-I 180The ratio is proportional. The constant of the ratio and the maximum depth range depend on the choice of carrier frequency. Alternatively, other combinations of phase angles can be used for this purpose, such as six phases spaced sixty degrees apart (0°, 60°, 120°, 180°, 240° and 300°), with corresponding adjustments to the TOF calculation.

[0029] Other iTOF systems use smaller sets of sensing elements, such as pairs of sensing elements integrated in the corresponding storage node due to photocharge generated by light received in phases 180° apart, or even arrays of sensing elements all sharing the same one or more detection intervals. In such cases, the synchronization of the detection intervals of the entire sensing element array is offset relative to the carrier of the transmitted beam within consecutive image frames to acquire sufficient information to measure the phase shift of the carrier relative to the transmitted beam in the received light. The processing circuitry then combines the pixel values ​​within two or more consecutive image frames to calculate the depth coordinates of each point in the scene.

[0030] To maximize the signal-to-noise ratio (SNR) of the iTOF image, it is desirable for each optical sensing element in the image sensing array to have two storage nodes. A switching circuit then couples the photocharge from the sensing element to each storage node during a corresponding detection interval in each carrier cycle. Thus, for example, each sensing element will generate and store an I0 intensity component in its first storage node and an I0 intensity component in its second storage node in each carrier cycle. 180 Intensity component (or I) 90 and I 270 (Components). This same image sensing array can also be used to capture 2D images in 2D sensing mode, typically under flood illumination, where photocharge is stored in only one of the storage nodes (or together in two storage nodes, rather than in opposite phases). Alternatively, a 2D image can be obtained by summing the signals obtained from the two storage nodes during 3D mapping. However, the need to accommodate dual storage nodes and switching circuitry in each unit of the image sensor increases the spacing of the sensing elements in the array, and thus limits the spatial resolution of the 2D image relative to images that can be captured by a conventional image sensor with only one storage node per unit.

[0031] The embodiments of the invention described herein address this problem by providing an image sensing array with shared storage nodes between adjacent sensing elements. In iTOF mode, a switching circuit in each cell of the array transfers photocharge to two adjacent storage nodes during different corresponding detection intervals in a corresponding phase of each carrier cycle. The switching circuit is operated such that each storage node receives and stores the photocharge generated by the two adjacent sensing elements during the same detection interval. On the other hand, in 2D imaging mode, each sensing element is individually connected to a single corresponding storage node. Because the array contains the same number of storage nodes as the sensing elements, the spacing between the sensing elements (and therefore the resolution of the 2D image) is comparable to that of a conventional high-resolution image sensor. Simultaneously, the novel mode of sharing adjacent storage nodes enables the array to capture SNR and 3D iTOF data with a resolution comparable to an iTOF sensor with two storage nodes per cell.

[0032] The disclosed embodiments provide an apparatus for optical sensing, wherein a matrix of optical sensing elements and a matrix of charge storage nodes are arranged on a semiconductor substrate such that each storage node is near two adjacent sensing elements. Thus, each sensing element within the matrix is ​​near two adjacent storage nodes on a different side of the sensing element. In the embodiments described below and shown in the accompanying drawings, the storage nodes are staggered along the rows of the matrix of sensing elements. In this context, the term "staggered" is used to mean that any given sensing element within any given row of the matrix shares a first storage node with a first adjacent sensing element on one side of that given sensing element along that row, and shares a second storage node with a second adjacent sensing element on the opposite side of that row (except for sensing elements at the ends of the row).

[0033] However, in alternative embodiments, the sensing elements and memory nodes can be arranged in different relationships. For example, a given sensing element may share at least one of its two adjacent memory nodes with an adjacent sensing element in another row of the matrix (such as a neighboring sensing element along the same column of the matrix). Additionally or alternatively, the matrix of memory nodes may be positioned below the matrix of sensing elements, for example, using chip stacking technology. All such alternative arrangements of sensing elements and adjacent memory nodes are considered to be within the scope of the invention, regardless of the specific topology used.

[0034] A switching circuit connects each sensing element to transfer photocharge to two adjacent storage nodes, allowing each sensing element to share these storage nodes with its two adjacent sensing elements. When the array is operated in iTOF depth sensing mode, the control circuit controls the switching circuit to switch between adjacent storage nodes during corresponding detection intervals within a series of detection cycles (typically synchronized with the carrier frequency of illumination of the target scene). In the first detection interval of each cycle, each sensing element is connected to one of the adjacent storage nodes along with one of its two adjacent sensing elements. Then, in the second detection interval of the same cycle, each sensing element is connected to another adjacent storage node along with its other adjacent sensing element. The readout circuit outputs the photocharge from the storage nodes in the array.

[0035] As previously noted and further described below, the device of the present invention can be used for both depth mapping and high-resolution 2D imaging through proper control of the switching circuitry. The switching circuitry may also include a shutter gate coupled to apply a global shutter to the sensing elements in 2D sensing mode. In some embodiments, the device further includes: an illumination assembly that directs optical radiation toward a target scene; and objective optics that form an image of the target scene onto a matrix of optical sensing elements. Alternatively, however, the novel switching and control circuitry in the sensing array can be used in other operating modes with or without a dedicated illumination assembly.

[0036] In the disclosed embodiment, the device performs depth mapping using an iTOF method. An illumination assembly modulates optical radiation directed toward the target scene using a carrier wave with a predetermined carrier frequency. Under the control of control circuitry, each sensing element samples an image of the target scene in two discrete detection intervals during each cycle of the carrier wave. The storage nodes in each row are numbered (for convenience) as “even” and “odd”, and the control circuitry controls the switching associated with each sensing element in such a manner, synchronously with the carrier wave: during a first detection interval, photocharge generated by its neighboring sensing elements is transferred to even-numbered storage nodes, and during a second detection interval, photocharge generated by its neighboring sensing elements is transferred to odd-numbered storage nodes.

[0037] For example, using the above scheme with four detection intervals, during the first cycle (or cyclic sequence) of the carrier, even-numbered storage nodes in the rows accumulate optical charge from the detection interval at 0° phase relative to the carrier, and odd-numbered storage nodes accumulate optical charge from the detection interval at 180° phase. During the next cycle (or cyclic sequence), the storage nodes accumulate optical charge from the corresponding detection intervals at 90° and 270°. Alternatively, alternating storage nodes in even-numbered rows of the matrix may accumulate optical charge in the detection intervals at 0° and 180°, while those in odd-numbered rows accumulate optical charge in the detection intervals at 90° and 270°.

[0038] After each detection cycle, photocharge is output from the storage node by the readout circuit. Since photocharge from two adjacent sensing elements accumulates in the same storage node, each pixel will include both sensing elements. (The term "pixel" is used in this specification and claims to refer to image data read from the storage node in response to the accumulated charge.) Therefore, for a matrix of sensing elements with N columns and M rows, the spatial resolution of the depth map in pixels will be (N / 2) × M, or (N / 2) × (M / 2) if alternating rows are used to acquire alternating detection phases, as described above.

[0039] In 2D sensing mode, according to the disclosed embodiment, a switch connects each sensing element to a single corresponding storage node. For example, each sensing element may be connected to a shared storage node to the left of that sensing element. Therefore, for the above N×M matrix, the spatial resolution in pixels will be the full N×M resolution of the matrix. Due to the shared storage node among the sensing elements, the spacing between the sensing elements can be as small as 2 μm using current manufacturing techniques. This fine spacing enables high-resolution 2D imaging while providing a sensor for iTOF.

[0040] System Description

[0041] Figure 1 This is a block diagram schematically illustrating a combined depth mapping and 2D imaging device 20 according to an embodiment of the invention. The device 20 includes an illumination assembly 24 and a detection assembly 26 under the control of a control circuit 22. In the illustrated embodiment, the illumination and detection assemblies are aligned and thus share the same optical axis outwards from the device 20, without parallax; however, other optical configurations may also be used.

[0042] The illumination assembly 24 includes a beam source 30, such as a suitable semiconductor emitter, like a semiconductor laser or a high-intensity light-emitting diode (LED), or an array of such emitters, which emits optical radiation toward the target scene 28 (in this case, containing a human subject). Typically, the beam source 30 emits infrared radiation, but alternatively, radiation from other parts of the spectrum may be used. The radiation may be collimated by projection optics 34.

[0043] For depth mapping, the synchronization circuit 44 in control circuit 22 modulates the amplitude of the radiation output from source 30 with a carrier having a specified carrier frequency. For example, the carrier frequency could be 100 MHz, meaning the carrier wavelength (when applied to the radiation output from beam source 30) is approximately 3 m, which also determines the effective range of device 20. (Outside this effective range, i.e., 1.5 m in this example, depth measurements may become blurred due to range folding). Alternatively, higher or lower carrier frequencies can be used, depending particularly on considerations of the desired range, accuracy, and signal-to-noise ratio.

[0044] Detection component 26 receives optical radiation reflected from target scene 28 via objective lens optics 35. The objective lens optics form an image of the target scene on matrix 36 of optical sensing elements 40, including photodiodes or phototransistors, such as those in iTOF image sensor 37. Sensing elements 40, formed on semiconductor substrate 41 (such as a silicon chip), are connected to a corresponding matrix 38 of charge storage and readout circuitry 42 on the semiconductor substrate. As shown in subsequent figures, circuitry 42 includes storage nodes and switching circuitry that gates the detection interval during which the sensing elements integrate focused optical radiation on matrix 36. Circuitry 42 also includes switches (such as those for reading charge from storage nodes to column lines of image sensor 37) for reading charge from storage nodes. Figure 9 (As shown).

[0045] Typically, though not strictly necessary, the image sensor 37 comprises a single integrated circuit device in which the sensing element 40 and circuitry 42 are integrated. Alternatively, the sensing element 40 may be connected to the circuitry 42, for example, using other techniques such as chip stacking, and may comprise silicon or other materials, such as III-V semiconductor materials.

[0046] Synchronization circuit 44 controls charge storage and readout circuit 42 such that photocharge generated by sensing element 40 in response to optical radiation incident on image sensor 37 is stored in different storage nodes during different detection intervals synchronized with the carrier frequency applied to beam source 30. For example, circuit 42 includes charge storage nodes and switches controlled to select different storage nodes during different detection intervals synchronized with the carrier frequency at different corresponding time phase angles, as further illustrated in subsequent figures.

[0047] Objective lens optics 35 forms an image of target scene 28 on matrix 36, such that each point in the target scene is imaged onto the corresponding sensing element 40. To calculate the depth coordinates of each point, control circuit 22 combines signals output from the sensing element, gated by circuit 42, as further described below. Control circuit 22 can then output a depth map 46 composed of these depth coordinates, and may also output a two-dimensional image of the scene.

[0048] For 2D imaging, the amplitude of the optical radiation emitted by the beam source 30 toward the target scene 28 can be modulated by the synchronization circuit 44 or emitted as unmodulated radiation. As will be described in further detail below, for 2D imaging sensing, the circuit 42 does not need to be synchronized with the emitted radiation, but is configured to collect a sufficient amount of radiation to generate a 2D image. In this case, the synchronization circuit 44 can drive the charge storage and readout circuit 42 to apply a global shutter in order to control the exposure level of the entire image sensor.

[0049] The control circuit 22 typically includes a general-purpose or special-purpose microprocessor or digital signal processor, which is programmed in software or firmware to perform the functions described herein. The control circuit also includes suitable digital and analog peripheral circuitry and interfaces, including synchronization circuitry 44, for outputting control signals to and receiving inputs from other components of the device 20. The detailed design of such circuitry will become apparent to those skilled in the art of depth mapping devices after reading this specification.

[0050] Sensor design and operation

[0051] Figure 2 This is a schematic circuit diagram illustrating an array of optical sensing elements and associated circuitry according to an embodiment of the present invention. It is assumed that the sensing elements in this embodiment are photodiodes and are accordingly labeled PD. In this and subsequent figures, for clarity of illustration, the sensing elements are shown interleaved with storage nodes along the rows of the array; however, other topological arrangements, as described above, may be used alternatively.

[0052] Figure 2Three proximity sensing elements (PDs) are shown in row 36 of matrix 36. i-1 PD i and PD i+1 Each sensing element is coupled to a pair of adjacent memory nodes SN via a corresponding transfer switch TX. Each sensing element is also connected to a common drain 66 via a switch called the shutter gate SG. For simplicity, the readout circuitry for reading charge from memory nodes SN is omitted here, but will be discussed below. Figure 9 As shown in the diagram. All switches TX and SG together form a switching circuit 45 controlled by control circuit 22.

[0053] The matrix of storage nodes SNs is interleaved with the sensing elements PDs along rows of matrix 36, such that each sensing element PD within any given row is near two adjacent storage nodes SNs on its opposite side along that row. (The term "within any given row" excludes sensing elements at the ends of that row, depending on the detailed structure of the image sensor). Therefore, each adjacent pair of sensing elements PDs shares a storage node SN between them. For example, sensing element PD... i With sensing element PD i-1 Shared charge storage node SN i-1 And with sensing element PD i+1 Shared charge storage node SN i (The terms "charge storage node" and "storage node" are used interchangeably.)

[0054] Each sensing element PD i-1 PD i and PD i+1 Connect to two switches TX X Y This is used to independently connect each sensing element to each of its two adjacent storage nodes SN. In the symbol TX X Y In the diagram, index X refers to the storage node SN to which the switch is connected, and Y is L or R, corresponding to whether the switch is on the left or right side of the storage node. For convenience, ... Figure 2 Select "L" and "R" within the view. For example, switch TX. i-1 R Sensing element PD i Connect to storage node SN i-1 And switch TX i L Connect the same sensing element to the storage node SN i .

[0055] For example, it is used to transmit sensing elements PD. i-1 PD i and PD i+1Coupled to storage node SN i-1 and SN i TX switch X Y The configuration is shown in Table 1:

[0056] Table 1: Switch TX X Y Configuration

[0057]

[0058] Another switch will sense the PD element. i-1 and PD i+1 Coupled to and in Figure 2 The storage nodes are shared by the sensing elements other than those in the figure, but for simplicity, these switches are not labeled in the figure or shown in Table 1.

[0059] As described above, each sensing element PD is also connected to the common drain 66 via a corresponding shutter gate SG to discharge any remaining photocharge from the sensing element before acquiring a new image. For simplicity, only the shutter gate SG... i (for sensing element PD) i ) and common drain 66 marked in Figure 2 middle.

[0060] Figure 3a and Figure 3b These are schematic circuit diagrams illustrating the operation of the switching circuit 45 in iTOF depth sensing mode according to an embodiment of the invention. These diagrams show the switching TX configured in two different detection intervals synchronized to corresponding 0° and 180° phase angles of the carrier frequency applied to the illumination component 24.

[0061] Figure 3a and Figure 3b Each figure in the diagram shows four sensing elements PD of matrix 36, labeled PD. i-1 PD i PD i+1 and PD i+2 All sensing elements PD, switches TX, and drain switches SG use the same... Figure 2 They are labeled using the same numbering scheme. Impact sensing element PD i-1 PD i PD i+1 and PD i+2 The optical radiation is respectively represented by double wavy arrow 70 i-1 70 i 70 i+1 and 70 i+2 instruct.

[0062] Before each exposure, all shutter gates SG are closed to drain any residual charge from the sensing element PD from the previous exposure to the common drain 66. Then, the shutter gates are opened before the next exposure, as shown in the figure.

[0063] At the detection interval synchronized to 0° phase angle ( Figure 3a The control circuit 22 closes the switch TX. i-1 L TX i-1 R TX i+1 L and TX i+1 R (The other switches remain open). Closing these switches will be controlled by the sensing element PD. i-1 and PD i The generated photocharge is transferred to the storage node SN i-1 As shown by arrow 72 i-1 L and 72 i-1 R As shown, and will be controlled by the sensing element PD i+1 and PD i+2 The generated photocharge is transferred to the storage node SN i+1 As shown by arrow 72 i+1 L and 72 i+1 R As shown. Therefore, the sensing element PD i-1 and PD i During this detection interval, a pixel is defined together, and the sensing element PD i+1 and PD i+2 Limit adjacent pixels.

[0064] At the detection interval synchronized to a 180° phase angle ( Figure 3b The control circuit 22 closes the switch TX. i-2 R TX i L TX i R and TX i+2 L (The other switches remain open). Closing these switches will be controlled by the sensing element PD. i and PD i+1 The generated photocharge is transferred to the storage node SN i As shown by arrow 72 i L and 72 i R As shown. Together with the sensor PD i-2 and PDi+3 (exist Figure 3b The photocharge (not shown) comes from the sensing element PD together with the photocharge. i-1 and PD i+2 The photocharge is transferred to the corresponding storage node SN i-2 and SN i+2 As shown by arrow 72 i-2 R and 72 i+2 L As shown. Figure 3b The only full pixel shown (the pixel that receives photoelectric charge from both sensing elements) is the sensing element PD. i and PD i+1 Limited, and sensing element PD i-1 and PD i+2 Each with PD i-2 and PD i+3 Limited pixels.

[0065] Control circuit 22 controls switching circuit 45 in this manner to perform a series of detection cycles. Figure 3a Configuration and Figure 3b The configurations alternate. Therefore, each sensing element PD in the sensing element PD and one of its neighboring sensing elements (on its left or right) are in... Figure 3a During the detection interval shown, they are connected together to one of the adjacent storage nodes SN; and each sensing element in the sensing element and its other adjacent sensing element are in Figure 3b The detection intervals shown are connected together to another adjacent storage node. After the photocharge for the detection intervals at 0° and 180° has been stored in the corresponding storage node, the readout circuit typically (though not always) reads the photocharge as signals I0 and I0 over several cycles. 180 .

[0066] Compare Figure 3a and Figure 3b As can be seen, in each row, the pixels acquired in the detection interval at 180° are offset by one sensing element PD from the pixels acquired at 0°.

[0067] In some implementations, photocharge in the detection intervals at 90° and 270° is collected in a similar manner in each row during a subsequent series of detection cycles, wherein the phase of the switching circuit 45 is appropriately adjusted. Alternatively, the alternating rows of matrix 36 can be driven with a 90° phase difference, such that one row collects photocharge in the detection intervals at 0° and 180°, while the next row collects photocharge in the detection intervals at 90° and 270°. In either case, refer to Figures 3a to 3bThe photocharge collected at 90° is similar to that collected at 0° and is stored in the storage node SN. i-1 and SN i+1 In this process, the photocharge collected at 270° is similar to that collected at 180° and is stored in the storage node SN. i-2 SN i and SN i+2 middle.

[0068] After storing the corresponding photocharges for the detection intervals at 90° and 270°, the readout circuit reads the photocharges as signal I. 90 and I 270 These signals, along with signals I0 and I... 180 Together, the control circuit 22 is used to calculate the depth value for each pixel, which is related to the function tan θ for that pixel. -1 [(I 270 -I 90 ) / (I0-I 180 Proportional.

[0069] Figure 4a and Figure 4b This is a schematic front view of an image sensor 37 according to an embodiment of the present invention, showing the phase detected by a storage node in iTOF depth sensing mode. The image sensor 37 is schematically represented as a matrix of cells 80, each cell including a corresponding sensing element PD, with connections to adjacent storage nodes as shown in the previous figure. Although the illustrated matrix includes only a few hundred sensing elements, in practice, matrices are typically much larger, for example, including 1000x1000 sensing elements. Figure 4a and Figure 4b The illustration shows an enlarged view of the 8×8 group 82 of unit 80.

[0070] Figure 4a The diagram shows the results with two signal values ​​I0 and I, respectively. 180 The spatial distribution of the associated pixels 84 and 86. As shown above... Figure 3a As shown, each of these signals is generated in a pixel comprising two adjacent sensing elements PD in response to the generated photocharge, wherein there is an element overlap between adjacent pixels at 0° and 180°. Therefore, pixel 84 generates signal I0 and pixel 86 generates signal I. 180 To easily show the overlap, pixels 84 and 86 are drawn in consecutive rows.

[0071] Similarly, Figure 4b This shows the generation of two signal values ​​I. 90 and I 270 The spatial distribution of pixels 88 and 90.

[0072] Figure 4a and Figure 4b The sampling scheme demonstrated shows that a pixel has a width of two units 80 and a height of one unit. Therefore, for an image sensor 37 with sensing elements 40 having N columns and M rows, the spatial resolution in pixels will be (N / 2) × M. The photocharge at 0° and 180° is collected ( Figure 4a After that, it is necessary to collect photocharges at 90° and 270°. Figure 4b The storage node SN is read out beforehand. Therefore, photocharge is collected at all four phases (0°, 90°, 180°, and 270°) within (at least) two image frames. Alternatively, if the alternating rows of matrix 36 are driven with a 90° phase difference, a complete iTOF image (including all four phases) can be collected in a single image frame with a spatial resolution of (N / 2) × (M / 2) pixels.

[0073] Figure 5 This is a schematic front view of an image sensor 37 in iTOF depth sensing mode according to another embodiment of the present invention. Illustrations 90a and 90b show two enlarged views of the 4×4 group 92 of unit 80. Similar to... Figure 4a and Figure 4b This generates signal values ​​I0 and I. 180 The pixels are shown in illustration 92a, and the signal value I is generated. 90 and I 270 The pixels are shown in illustration 92b. The layout of the pixels is... Figure 4a and Figure 4b The difference lies in the polarity (i.e., horizontal order) of the phase of the pixel detection interval, which is reversed from row to row. Control circuit 22 can implement polarization reversal by changing the control signal applied to switching circuit 45 in consecutive rows. Such polarization reversal can be advantageously used to reduce artifacts in the resulting depth map.

[0074] Figure 6 This is a schematic front view of an image sensor 37 in iTOF depth sensing mode according to another embodiment of the present invention. Figures 4a to 4b and Figure 5As in the previous example, image sensor 37 is schematically represented as a matrix of cells 80. Illustrations 96a, 96b, and 96c show three enlarged views of the 4×4 group 96 of cells 80. In this embodiment, the carrier is sampled at six detection intervals at 0°, 60°, 120°, 180°, and 240° with three pairs of 0° / 180°, 60° / 240°, and 120° / 300°, where the two elements of each pair are always 180° apart. The carrier is first sampled at 0° and 180°, as shown in illustration 96a. After the stored photocharge is read from the corresponding storage node SN, the carrier is sampled at 60° and 240°, as shown in illustration 96b, and then the charge is read out again. Finally, the carrier is sampled at 120° and 300°, and then the charge is read out.

[0075] Therefore, three image frames are required to read out all photocharges for all six detection intervals. Although the full cycle of data acquisition is thus longer than, for example, a four-interval scheme, the additional data improves the robustness of the depth mapping while providing the same (N / 2)×M spatial resolution.

[0076] Figure 7 This is a schematic front view of an image sensor 37 in iTOF depth sensing mode according to another embodiment of the present invention. Figure 7 The illustration shows an enlarged view of a group 100 having 6×8 units 80. In this embodiment, the carrier is sampled at six detection intervals at 0°, 60°, 120°, 180°, and 240°, as shown in the figure. Figure 6 The same applies. In this case, all detection intervals are sampled in each of the three rows 102 during a single image frame via appropriate timing of the switching circuit 45, thus... Figure 6 In comparison, this achieves a three-fold increase in sampling speed. However, due to the use of three rows for different detection intervals, the spatial resolution in pixels will be (N / 2) × (M / 3).

[0077] In all Figures 4a to 4b , Figure 5 , Figure 6 and Figure 7 In this embodiment, image sensor 37 can be a physically substantially identical sensor having a temporal and spatial sampling scheme controlled by control circuitry 22. In an alternative embodiment, similar to... Figure 5 The polarization reversal shown in the alternating rows can also be in Figure 6 and Figure 7 The scheme shown is implemented. Alternatively, the switching circuit 45 can be configured and controlled to implement a spatial and temporal sampling scheme, provided that at least three detection intervals are sampled relative to a carrier cycle.

[0078] Figure 8 This is a schematic circuit diagram illustrating the operation of the switching circuit 45 in 2D imaging mode according to an embodiment of the present invention. As shown in the previous figure, Figure 8 Three sensing elements PD are shown in row 36 of matrix 36. i-1 PD i and PD i+1 For 2D imaging, a switch TX of each sensing element PD is closed, such that each sensing element is individually connected to a single, corresponding adjacent storage node SN. Therefore, the photocharge generated by each sensing element in response to incident radiation (indicated by the corresponding double-wavy arrow 110) is collected in the storage node SN to the left of that sensing element. For example, for sensing element PD... i TX switch i-1 R It is closed and is controlled by the sensing element PD. i In response to incident radiation 110 i The generated photocharge is collected in the storage node SN. i-1 In the middle, as shown by arrow 112 i-1 As shown. Similarly, by the sensing element PD i-1 and PD i+1 The generated photocharges are stored in the storage node SN. i-2 and SN i middle.

[0079] Therefore, each storage node SN receives optical charge from only one sensing element PD, and each sensing element PD and its adjacent storage node define corresponding pixels of a 2D image, where the image resolution is N×M. The choice of storing the generated optical charge in the storage node to the left of the sensing element is arbitrary, and the storage node to the right of each sensing element can also be selected.

[0080] Figure 9 This is a schematic circuit diagram illustrating an exemplary specific implementation of the sensing element PD and associated circuitry 120 in an image sensor 37 according to an embodiment of the present invention. In this example, the sensing element PD is implemented as a fully depleted pinned photodiode, thereby enabling appropriate transfer switching TX. i L or TX i R When activated, the selected storage node SN is sent. i Provides full charge transfer. (As previously stated in) Figures 3a to 3b and Figure 8 Select the storage node SN as described in the context. i and transfer switch TX i L or TX i RSimilarly, each storage node has an SN. i To the corresponding floating diffusion FD i Transfer gate TRF i When activated, it is implemented by providing full charge transfer to the floating diffusion. Circuit 120 includes a sensing element PD. i and PD i+1 And switching circuit 45 and readout circuit 122 (for storage node SN) i and SN i+1 These are respectively labeled as readout circuits 122 i and 122 i+1 ). Use and Figure 2 , Figures 3a to 3b and Figure 8 The same symbols and tags.

[0081] Readout circuit 122 i (And similarly, readout circuit 122) i+1 ) including floating diffusion nodes FD i It is achieved through the transfer gate TRF i Connect to storage node SN i Floating Diffusion Node (FD) i Also via the reset gate RST i Connected to reset voltage V pix Source follower SF i The gate. Source follower SF i The source and drain are selected via the row select gate RS i Connected to the baseline voltage V DD And column line 124 i Between. Column line 124 i Connect to control circuit 22 to output stored charge (or digital value of stored charge).

[0082] By sensing element PD i and PD i+1 The generated photocharge is collected into the storage node SN. i and SN i+1 One or both of them, such as Figures 3a to 3b and Figure 8 As shown. By actuating the transfer gate TRF i Photocharge from storage node SN i (And similarly, from storage node SN) i+1 ) was transferred to the floating diffusion node FD i This is in the source follower SF i A voltage is generated at the gate. The collected photocharge is transferred from the storage node SN. i Transfer to the corresponding floating diffusion node FD iPreviously, control circuit 22 actuated the reset gate RST. i In order to reset the floating diffusion node to the reset voltage V pix From storage node SN i After the photocharge is transferred, the floating diffusion FD is subtracted from the voltage generated at the floating diffusion node by the transferred photocharge. i Reset voltage V pix This provides compensation for the KTC noise injected into the floating diffusion node during the reset operation.

[0083] For the entire row of matrix 80 of the sensing element ( Figure 4a Alternatively, a group of sensing elements in that row simultaneously performs the reset of the floating diffusion node FD and the charge transfer from the storage node SN. When the control circuit 22 actuates the row selection gate RS i At that time, current source 126 i Drive current through source follower SF i And from the source follower SF i The gate signal is transferred to column line 124 i For further processing. The entire matrix 80 is read sequentially, row by row.

[0084] The photocharge has been transferred from the sensing element PD i Transferred to storage node SN i Then, by actuating the shutter gate SG i Any residual charge from the sensing element is drained to the common drain 66. The sensing element PD is treated in a similar manner. i+1 Discharge.

[0085] In the disclosed embodiments, the structure of image sensor 37 enables operation in 2D imaging and 3D mapping in global shutter mode, wherein simultaneous acquisition of charge in storage node SN is performed for all (or one or more groups of) sensing elements in matrix 80. It also enables operation of image sensor 37 in rolling shutter mode as well as in a combination of rolling shutter and global shutter modes.

[0086] Compared to known 2D global shutter image sensors in the art, circuit 120 includes one additional transistor for each sensing element PD. However, compared to commonly used iTOF circuits, each sensing element PD in circuit 120 requires one less transistor and one less memory node. This feature enables the fabrication of smaller iTOF sensor elements, which can then be used to provide high-resolution 2D imaging.

[0087] It should be understood that the embodiments described above are cited by way of example, and the invention is not limited to what has been specifically shown and described above. Rather, the scope of the invention includes the various features described above, as well as combinations and sub-combinations of variations and modifications thereof that would occur to those skilled in the art upon reading the above description and not disclosed in the prior art.

Claims

1. A device for optical sensing, comprising: Semiconductor substrate; A first matrix of optical sensing elements, the first matrix being arranged in rows and columns on the substrate such that any given sensing element in any given row of the first matrix has a first adjacent sensing element and a second adjacent sensing element disposed on opposite sides of the given sensing element along the given row in the first matrix; A second matrix of storage nodes, the second matrix being arranged on the substrate and intersecting with the sensing elements along the rows of the first matrix, such that corresponding first storage nodes and corresponding second storage nodes in the second matrix are disposed near each of the sensing elements in the first matrix; A switching circuit is configured to couple each of the sensing elements to transfer photocharge to the respective first storage node and the respective second storage node; A control circuit configured to control the switching circuit in a depth sensing mode, such that in a series of detection cycles including at least a first detection interval and a second detection interval in each cycle, each of the sensing elements and the first adjacent sensing element are connected together to the corresponding first storage node during the first detection interval, and each of the sensing elements and the second adjacent sensing element are connected together to the corresponding second storage node during the second detection interval. as well as A readout circuit configured to output the photocharge from the storage node.

2. The apparatus according to claim 1, wherein the apparatus comprises: An illumination assembly configured to guide optical radiation toward a target scene while modulating the optical radiation using a carrier having a predetermined carrier frequency; as well as An objective lens optics device configured to image the target scene onto a first matrix of optical sensing elements. The control circuit is configured to synchronize the detection cycle with the carrier frequency.

3. The apparatus of claim 2, wherein the control circuit is configured to compare the photocharge output by the readout circuit from the first storage node and the second storage node in order to calculate the depth coordinates of a point in the target scene.

4. The apparatus of claim 3, wherein the control circuit is configured to change the corresponding phases of the first detection interval and the second detection interval relative to the carrier within the series of detection cycles, and to use the photocharge output within the changed phase to calculate the depth coordinates.

5. The apparatus of claim 4, wherein the first detection interval and the second detection interval have a phase separation of 180° relative to the carrier.

6. The apparatus of claim 5, wherein the control circuit is configured to control the switching circuit such that the first detection interval and the second detection interval have phases of 0° and 180° relative to the carrier, and such that the storage nodes in at least some of the rows of the matrix collect the photocharge in third and fourth detection intervals having phases of 90° and 270° relative to the carrier.

7. The apparatus of claim 5, wherein the control circuit is configured to control the switching circuit such that the first detection interval and the second detection interval have phases of 0° and 180° relative to the carrier, and such that the storage nodes in at least some of the rows of the matrix collect the photocharge in a third detection interval, a fourth detection interval, a fifth detection interval, and a sixth detection interval having phases of 60°, 120°, 240°, and 300° relative to the carrier.

8. The apparatus of claim 5, wherein the phase of the first detection interval and the phase of the second detection interval are reversed from row to row in consecutive rows of the second matrix.

9. The apparatus according to any one of claims 1 to 8, wherein the control circuit is further configured to control the switching circuit in a two-dimensional 2D sensing mode, such that each of the sensing elements is individually connected to a single corresponding storage node in the vicinity of the storage node.

10. The apparatus of claim 9, wherein the switching circuit includes a shutter gate coupled to apply a global shutter to the sensing element in the 2D sensing mode.

11. The apparatus according to any one of claims 1 to 8, wherein the switching circuit comprises a first transfer switch and a second transfer switch respectively coupled between each sensing element in the sensing elements and the corresponding first storage node and the corresponding second storage node.

12. A method for optical sensing, comprising: A first matrix of optical sensing elements is arranged in rows and columns on a semiconductor substrate, such that any given sensing element in any given row of the first matrix has a first adjacent sensing element and a second adjacent sensing element disposed on opposite sides of the given sensing element along the given row in the first matrix. A second matrix of storage nodes is arranged on the substrate, the second matrix intersects with the sensing elements along the rows of the first matrix, such that the corresponding first storage node and the corresponding second storage node in the second matrix are arranged near each sensing element in the sensing elements in the first matrix; Each of the sensing elements is coupled to transfer optical charge to the corresponding first storage node and the corresponding second storage node via a switching circuit; In depth sensing mode, the switching circuit is controlled such that in a series of detection cycles including at least a first detection interval and a second detection interval in each cycle, each sensing element and the first adjacent sensing element are connected together to the corresponding first storage node during the first detection interval, and each sensing element and the second adjacent sensing element are connected together to the corresponding second storage node during the second detection interval. as well as The photocharge is output from the storage node.

13. The method of claim 12, wherein the method comprises: The optical radiation is guided toward the target scene, while the optical radiation is modulated using a carrier with a predetermined carrier frequency; The target scene is imaged onto the first matrix of the optical sensing element; as well as The detection cycle is synchronized with the carrier frequency.

14. The method of claim 13, wherein the method includes comparing the photocharges output from the first storage node and the second storage node to calculate the depth coordinates of a point in the target scene.

15. The method of claim 14, wherein synchronizing the detection cycles includes changing the corresponding phases of the first detection interval and the second detection interval relative to the carrier within the series of detection cycles, and comparing the photocharges includes using the photocharges output within the changed phases to calculate the depth coordinates.

16. The method of any one of claims 12 to 15, wherein the method comprises controlling the switching circuit in a two-dimensional 2D sensing mode such that each of the sensing elements is individually connected to a single corresponding storage node in the vicinity of the storage node.

17. The method of claim 16, wherein the control switching circuitry includes applying a global shutter to the sensing element in the 2D sensing mode.

18. The method according to any one of claims 12 to 15, wherein the switching circuit includes a first transfer switch and a second transfer switch respectively coupled between each sensing element in the sensing elements and the corresponding first storage node and the corresponding second storage node.

Images