Time-of-flight systems and methods

Abstract

A multi-sensing pixel array including sensing pixels (PIX_R, PIX_G, PIX_B, PIX_D) and active illumination (PIX_I) in a multi-layer stacked die.

Description

[Technical field]

[0001] The present disclosure relates generally to the field of time-of-flight imaging, and more particularly to multi-sensing pixel arrays and corresponding devices implementing the multi-sensing pixel arrays. [Background technology]

[0002] With the continuing development of autonomous driving, traditional 2D cameras are being complemented by other camera technologies such as stereo cameras, IR cameras, RADAR, LiDAR, and Time-of-Flight (ToF) cameras.

[0003] A time-of-flight (ToF) camera is a range imaging camera system that determines the distance of an object by measuring the time of flight of an optical signal between the camera and the object for each point in an image. Typically, a ToF camera is equipped with an illumination unit (LEDs and / or laser diodes, e.g., VCSEL, Vertical Cavity Surface Emitting Laser, Fabry-Perot Semiconductor Laser, etc.) that illuminates a scene with modulated light. The pixel array of the ToF camera collects the light reflected from the scene and measures the phase shift (iToF, indirect ToF) or the travel time of the light (dToF, direct ToF), which allows the extraction of the distance of objects in the scene.

[0004] Currently, acquisition of optical signals at short distances is typically covered by indirect time-of-flight (iToF) systems, and acquisition at medium to long distances is typically covered by direct time-of-flight (dToF) systems.

[0005] In time-of-flight (iToF), three-dimensional (3D) images of a scene are captured. These images are commonly also called "depth maps" or "depth images," and each pixel in the image is assigned its own depth measurement.

[0006] It is therefore generally desirable to provide techniques that improve multi-sensing techniques. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] U.S. Patent Application Publication No. 2021 / 0043687 Al Summary of the Invention [Means for solving the problem]

[0008] According to a first aspect, the present disclosure provides a multi-sensing pixel array including sensing pixels and active illumination in a multi-layer stacked die.

[0009] Further aspects are set out in the dependent claims, the following description and the drawings.Embodiments will now be described, by way of example only, with reference to the accompanying drawings, in which: FIG. [Brief description of the drawings]

[0010] [Figure 1] 1 illustrates a schematic of the basic operating principle of time-of-flight imaging. [Diagram 2] 1 shows a schematic of three different stacks typically applied in an imaging system: an RGB sensor array, a ToF sensor array, and a ToF active illuminator. [Diagram 3] 1 illustrates a schematic diagram of an embodiment of a multi-sensing pixel array including active illumination in plan view. [Figure 4] 1A and 1B illustrate schematic cross-sectional views of a multi-sensing pixel array of an embodiment. [Diagram 5] 1 shows a schematic diagram of an example of a field (Fol) produced by a ToF illuminator. [Figure 6] 1 illustrates a schematic example of designing an optical channel of a multi-sensing pixel array. [Figure 7] 1 illustrates an example of coupling a laser diode to an optical channel of a multi-sensing pixel array of an embodiment. [Figure 8]1 illustrates generally one embodiment of a device implementing an embodiment of a multi-sensing pixel array. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Before proceeding to a detailed description of the embodiment with reference to FIG. 1, a general description will be given.

[0012] The embodiments described in more detail below disclose a multi-sensing pixel array that includes sensing pixels and active illumination in a multi-layer stacked die.

[0013] In a multi-sensing pixel array embodiment, the sensing pixels and the active illumination may, for example, be located in a single pixel array.

[0014] For example, three different dies may be stacked on top of each other to form a multi-sensing pixel array. Each die may be formed by several IC stacks. The multi-layer stack may include, for example, three main components: an optical system, a multi-sensing sensor array (with all its IC stacks, i.e., pixels, and logic), and an illuminator (with all its IC stacks, i.e., cavity, laser driver, controller, etc.). According to some embodiments, the sensing pixels and the active illuminator share the main optical stack.

[0015] For example, imaging pixels may be removed from a conventional regular pattern of imaging pixels and an active illumination pass-through optical channel and depth-sensing pixels inserted in their place.

[0016] The sensing pixels may include, for example, depth sensing pixels.

[0017] According to the embodiment described below, the depth sensing pixels are ToF pixels.

[0018] The depth sensing pixel may be, for example, a ToF pixel. The ToF pixel may be of different types of technology, for example, an iToF pixel (CAPD, gated ToF, etc.) or a dToF pixel (SPD) or a PC pixel (SPAD) or a dynamic photodiode (DPD), etc. That is, the ToF pixel may be implemented according to the principles of dToF (direct ToF), iToF (indirect ToF), or PC (photon counting).

[0019] In some embodiments, the sensing pixels include imaging pixels and depth-sensing pixels.

[0020] The imaging pixels may for example include RGB pixels and / or monochrome pixels and / or infrared pixels (IR), or a combination thereof. The imaging pixels (also called visual pixels) are for example of the CMOS or CCD type.

[0021] The multi-sensing pixel array may be implemented, for example, using stacked sensing techniques.

[0022] In this way, sensing and lighting technologies are stacked together.

[0023] For example, RGBI information may be detected vertically (eg, by using organic pixels).

[0024] According to an embodiment, the multi-sensing pixel array includes a primary optical stack, an imaging stack, and an illumination stack.

[0025] The primary optical stack may include, for example, a primary lens and, optionally, an optical filter.

[0026] The imaging stack may include, for example, a microlens, a pixel array including imaging pixels (RGB and / or monochrome and / or infrared), and a ToF pixel. The microlens may create a dedicated illumination field (Fol) in conjunction with the main optical lens to optimize the amount of active illumination returning from the scene to the ToF pixel.

[0027] The imaging stack may further include analog circuitry and logic for driving the imaging pixels and the ToF pixels.

[0028] The illumination stack of the multi-sensing pixel array may include an active illuminator.

[0029] The illumination stack of the multi-sensing pixel array may further include a driver for driving the active illuminator.

[0030] The imaging stack of the multi-sensing pixel array may include a respective optical channel disposed relative to a respective illumination source and configured to direct illumination to the exterior. For example, an optical channel (or "optical channel") may be disposed across above the stacked ICs and direct active illumination to the exterior of the package. In this manner, the optical channel may be configured to traverse the ICs on the stack and direct active illumination to the exterior of the stack.

[0031] The light channel may be configured as a fiber optic light guide having a step index profile or a distributed index profile, or as a single mode optical fiber.

[0032] For each active illuminator, a respective microlens may be positioned to create the field of illumination (Fol) of the illuminator. For example, the microlenses may create a dedicated field per spot. In conjunction with the main optical lens, a desired profile may be created. For example, each illuminator may generate an illumination spot, e.g., a ring, to optimize the amount of active illumination returning from the scene to the ToF pixel. Otherwise, using the same microlenses as those used in the imaging pixel would result in active illumination returning from the scene retreating to the illumination pixel instead of the ToF pixel.

[0033] The multi-sensing pixel array may be configured to provide the same field of view (FoV) for the imaging pixels and the depth-sensing pixels.

[0034] Also disclosed in embodiments is pixel and cavity control configured to activate only those ToF pixels that are actually needed.

[0035] The multiple sensing pixel array may be implemented, for example, in a single IC or in multiple stacked ICs.

[0036] The multi-sensing pixel array may be implemented, for example, according to organic vertical stacking technology.

[0037] The coupling of the active illumination and the optical channel may be performed, for example, using ball lens coupling technology.

[0038] Embodiments also disclose a device implementing a multi-sensing pixel array as disclosed herein, which may be, for example, a smartphone, a laptop, or AR glasses (glasses for enabling augmented reality).

[0039] The device may also include further circuitry such as a processor, memory (RAM, ROM, etc.), storage, input means (mouse, keyboard, camera, etc.), output means (display (e.g., liquid crystal, (organic) light emitting diode, etc.), speaker, etc., (wireless) interface, etc., as is commonly known for electronic devices (computers, smartphones, etc.). Furthermore, the device may include sensors for sensing still or video data (image sensors, camera sensors, video sensors, etc.), sensors for sensing fingerprints, sensors for sensing environmental parameters (e.g., radar, humidity, light, temperature), etc.

[0040] How Time-of-Flight Imaging Systems (ToF) Work FIG. 1 shows a schematic of the basic operating principle of an indirect time-of-flight imaging system that can be used for depth sensing. The iToF imaging system 11 includes an iToF camera with an imaging sensor 12 having a matrix of pixels and a processor (CPU) 15. The scene 17 is actively illuminated with amplitude-modulated infrared light LMS at a predefined wavelength, for example with several light pulses of at least one predefined modulation frequency DML generated by a timing generator 16, using an illumination device 19 (e.g., ToF active illuminator 23 in FIG. 2). The amplitude-modulated infrared light LMS is reflected from an object in the scene 17. A lens 13 collects the reflected light 19 and forms an image of the object in the scene 17 on the imaging sensor 12. In the indirect time-of-flight (iToF), the CPU 15 determines for each pixel the phase delay between the modulated signal DML and the reflected light RL. Based on these correlations, a so-called in-phase component value ("I value") and a so-called quadrature component value ("Q value") can be determined for each pixel (more details will be given below).

[0041] 1 is an example of an indirect time-of-flight imaging system to explain the principle of the time-of-flight imaging system. However, the embodiment described below is not limited to the indirect time-of-flight principle. The depth sensing pixel may be, for example, an iToF pixel (CAPD, gated ToF, etc.), a dToF pixel (SPAD), a PC pixel (SPAD), a dynamic photodiode (DPD), etc. That is, the ToF pixel may be implemented according to the principle of dToF (direct ToF), iToF (indirect ToF), or PC (photon counting).

[0042] Multimodal Image Sensor An imaging system or device typically includes stacks of sensors and illumination technologies, such as a stack of imaging sensors, a stack of ToF sensors, and a stack of illuminators, which are usually separated.

[0043] FIG. 2 shows a schematic of three different stacks typically applied in an imaging system: an RGB sensor array 21, a ToF sensor array 22, and a ToF active illuminator 23. The RGB sensor array 21 includes an array of pixels PIX_R, PIX_G, PIX_B, where PIX_R is a pixel configured to capture red light, PIX_G is a pixel configured to capture green light, and PIX_B is a pixel configured to capture blue light. The ToF sensor array 22 includes an array of PIX_D. The ToF active illuminator 23 includes an array of ToF active illuminators PIX_I. The ToF active illuminator PIX_I may be implemented, for example, as a vertical cavity surface emitting laser (VCSEL), which is a semiconductor diode that converts voltage into photons. The photons emitted by the ToF active illuminator 23 are directed towards the scene, reflected from the scene, and then captured by the ToF sensor array 22 to determine a depth map of the scene. To evaluate the information from the RGB sensor array 21 and the ToF sensor array 22, an RGBD fusion technique may be applied.

[0044] In a device such as a smartphone, other sensing and lighting technologies (not shown in FIG. 2), such as flashing LEDs, may also be implemented.

[0045] Using multiple units in one device creates problems regarding the space available in the device. For example, in a mobile phone 0 with several sensors, e.g., RGB sensor, flash LED, ToF sensor, ToF active illuminator, proximity sensor, there will be many holes to mount the sensing devices.

[0046] Furthermore, in smaller devices, such as a pair of glasses using augmented reality (AR), it may not be desirable or feasible to have so many separate sensors and illuminators.

[0047] Another aspect involved in applying multiple sensing and lighting modalities is the power consumption: for example, the power consumption per pixel of ToF is very high: the more sensors applied to a device, the higher the power consumption.

[0048] When there are transceiver (TX) and receiver (RX) implementations with different optics, parallax aspects (caused by the distance between TX and RX) may also play a role. Parallax makes calibration and registration difficult. Parallax may also affect the zoom functionality of the device.

[0049] All-in-one stack sensing The embodiments described in more detail below provide a multi-sensing pixel array incorporating active illumination, where all sensing and illumination technologies are stacked all-in-one. The embodiments described in more detail below provide an all-in-one approach, stacking three different technologies: a stack of RGB sensors, a stack of ToF sensors, and a stack of illuminators.

[0050] 3 shows a schematic diagram of an embodiment of a multi-sensing pixel array with active illumination in a plan view. The multi-sensing pixel array is implemented as an RGB-ToF sensor array 31 with active illumination for generating a sparse depth map. The RGB-ToF sensor array 31 is composed of an array of pixels PIX_R, PIX_G, PIX_B, and PIX_D, where PIX_R is a pixel configured to capture red light, PIX_G is a pixel configured to capture green light, PIX_B is a pixel configured to capture blue light, and PIX_D is a ToF pixel. In each quadrant of the RGB-ToF sensor array 31, a group of eight ToF pixels is arranged, in the center of which is a ToF active illuminator PIX_I. That is, the central active illuminator pixel PIX_I of the eight ToF pixels PIX_D is provided with an opening in the multi-sensing pixel layer through which active light generated in the active illumination stack (e.g., VCSEL) below exits the multi-stack die, as described in more detail in the following embodiments. There are various ways (e.g., simple holes with lateral barriers, fiber optic-like configurations, etc.) to direct the light from the active illumination stack through the die stack, as described in more detail in the following embodiments.

[0051] The ToF pixel PIX_D can be of many different types of technologies, for example an iToF pixel (CAPD, gated ToF, etc.) or a dToF pixel (SPAD) or a PC pixel (SPAD), etc.

[0052] In the design of Figure 3, imaging pixels are thus removed in order to insert active illumination and ToF pixels. Note, however, that the ToF pixel intensity values ​​provided by ToF pixel PIX_D can compensate for the RGB pixels removed from the sensor to accommodate ToF pixel PIX_D.

[0053] In a multi-sensing pixel array including active illumination, such as the RGB-ToF sensor array 31 of FIG. 3, the different pixels and / or illumination spots do not need to be of regular size.

[0054] The multi-sensing pixel array technique as described in FIG. 3 above can also be applied to stack sensing, ie, RGBI information is detected in the vertical direction (eg, by using organic pixels).

[0055] It should be noted that the multi-sensing pixel array technology is not limited to the RGB-ToF type: any configuration is possible, for example a ToF active illumination surrounded by ToF pixels may be surrounded by IR imaging pixels and / or IR pixels.

[0056] There are use cases for the pixel array in Figure 3, such as object identification using RGBD-fusion, SLAM, etc., where only a few ToF pixels (sparse depth map) are required.

[0057] FIG. 4 shows a schematic cross-sectional view of a multi-sensing pixel array of an embodiment. The multi-sensing pixel array includes a primary optical stack 41, an RGB-ToF stack 42, and an illumination stack 43. The primary optical stack 41 includes a primary lens 44 and an optional optical filter 45. The RGB-ToF stack 42 includes a pixel array including a microlens 45 and optional optical filters (not shown), RGB pixels PIX_R, PIX_G, PIX_B, and a ToF pixel PIX_D, and analog circuits and logic for driving these pixels. The illumination stack 43 includes a ToF active illuminator PIX_I as an illumination source that emits light (as indicated by a vertical upward arrow) and a respective driver (not shown) for driving the illumination source 47. An optical channel 48 in the illumination stack 43 arranged for each illumination source PIX_I directs the illumination. An optical channel (or "optical channel") that crosses the stack ICs directs the active illumination to the outside of the package. The light channels can be implemented in different ways, for example as holes with lateral IR barriers or as optical fibers (filling the core with a material with a higher refractive index n than the surroundings, as shown in FIG. 6 below). At each ToF active illuminator PIX_I, a respective microlens 49 is placed, which creates the illumination field (Fol) of the illuminator. The microlenses 49 create a dedicated illumination field for each spot, so that light reflected at the scene returns to the surrounding PIX_D, not to PIX_I. In relation to the main optical lens, the desired profile is created. For example, each illuminator may generate an illumination spot (for example, a ring) as shown in FIG. 5 below. As shown by the dotted pattern, these microlenses 49 that create the illumination field are different from the microlenses 46 that focus the light on the RGB-ToF pixels R, PIX_G, PIX_B, PIX_D.

[0058] The multi-sensing pixel array of Figure 4 thus shares the main optical stack 41 (FoV) for the sensor and active illuminator. A multi-sensing pixel array as described in Figure 4 may thus provide the same FoV for RGB and ToF, thus simplifying calibration and post-processing.

[0059] Note that in the multi-sensing pixel array of FIG. 4, the active light source is located below the die stack, i.e., not in the same die stack as the pixel sensor array.

[0060] Multi-sensing pixel arrays such as those described in Figure 4 above use optical channels that traverse the ICs on the stack to direct active illumination outside the stack, creating a dedicated FoI (field of illumination) for each spot, and a desired profile (e.g., a ring shape as described in Figure 5 below) in relation to the main optical lens.

[0061] The coupling of the laser illuminator PIX_I to the light channel 48 can be done in different ways. For example, ball lens technology (see FIG. 7 and corresponding description) may be used to couple the laser illuminator PIX_I to the light channel 48.

[0062] Note that the example in FIG. 4 shows a three-layer stack for simplicity. However, in other embodiments there may be more layers. The pixel stack may contain multiple sub-stacks (pixel, analog and digital), the illumination stack may contain several sub-stacks (laser cavity, analog and possibly logic), etc. FIG. 5 shows a schematic example of an illumination field (Fol) generated by a ToF illuminator. The figure shows the x-dimension (in degrees) on the horizontal axis and the y-dimension (in degrees) on the vertical axis. The ToF illuminator (PIX_I in FIG. 3 and FIG. 4) generates a ring Fol51 for each spot.

[0063] However, it is noted that the arrangement of the ToF pixels can be changed from spots to other configurations. The pixel sensor technology can be, for example, uniform or non-uniform. Different approaches can be applied to confine the illumination within the optical channel so that it does not affect the surrounding ToF pixels, for example, optical barriers surrounding the optical channel, small optical fiber-like channels (by creating a reflectance step between the channel and the surrounding pixels), etc.

[0064] Additionally, other aspects may be applied to the embodiments, for example pixel and cavity control may be provided, i.e., only those ToF pixels that are actually needed are activated, so that light is emitted only when needed.

[0065] The multi-sensing pixel array of Figures 3 and 4 reduces power consumption since only a sparse array of ToF points is required. Furthermore, the multi-sensing pixel array described in Figures 3 and 4 reduces parallax since the transceiver (TX) and receiver (RX) are implemented on a single stack.

[0066] Moreover, the parallax problem is eliminated with ToF: the active ToF illuminator and the ToF pixels share the same lens, which corrects for optical imperfections, which also allows you to optically zoom in and out with ToF.

[0067] Additionally, multi-sensing pixel arrays such as those described in Figures 3 and 4 do not require cross-technology compatibility. This is beneficial because current lasers and sensors typically use different technologies. In a design such as that described in Figure 4, this is beneficial because different suppliers can provide different stacks.

[0068] Furthermore, multi-sensing pixel arrays such as those described in Figures 3 and 4 can reduce the number of holes in a device from three to one, which is beneficial for small devices and cost reduction. Multi-sensing pixel arrays do not require as many holes to interface the device with the outside world, which is particularly beneficial in small devices such as AR glasses, where having many separate sensors / illuminators is undesirable / not feasible.

[0069] Furthermore, multi-sensing pixel arrays such as those described in Figures 3 and 4 can reduce the amount of space required for sensors within a device. For example, a sensing device such as a mobile phone can have several RGB sensors, flash LEDs, a ToF sensor, and an active illuminator mounted on the device without requiring a lot of space. This is particularly beneficial for small devices, such as glasses that use AR.

[0070] Furthermore, potential reflections (backside) of the active light on the main optical lens of the ToF pixel can be easily eliminated as it can be detected with virtually zero delay.

[0071] Another advantage of a multi-sensing pixel array as described in Figures 3 and 4 is that the sensor array and illuminator are optimally used. For example, it is not necessary to have a full array of sensors to identify the location of a spot. The illumination can be as efficient and powerful as a spot illuminator. Furthermore, there is no need to have extra active pixels to deal with the displacement of the dots due to parallax.

[0072] The multi-sensing pixel array described in FIG. 3 and FIG. 4 may be implemented in a single IC, for example, using known stack illuminator technology (under-cavity driver). The ToF pixel may be implemented according to the dToF (direct ToF), iToF (indirect ToF), or PC (photon counting) principle. Thus, the multi-sensing pixel array described in FIG. 3 and FIG. 4 connects the dToF, PC, and PC-ToF technologies to the stack illuminator technology. Multi-sensing pixel array technology as known from stack illuminator technology may be used, for example, to implement the multi-sensing pixel array according to the embodiment. For example, organic vertical stack pixels as disclosed in Patent Document 1 may be applied. Here, the organic photoelectric conversion layer absorbs only light in the visible light region and generates signal charges corresponding to the light of each color component of R (red), G (green), and B (blue). The light of the IR component in the infrared region is transmitted through the organic photoelectric conversion layer. This allows RGB pixels and IR pixels to be included in one stack.

[0073] Figure 6 shows schematic examples of optical channel (48 in Figure 4) design for a multi-sensing pixel array. Figure 6 shows three examples of optical channel configurations using fiber optic light guides.

[0074] According to a first example a), an optical fiber with a step refractive index profile is used. A step refractive index profile is a refractive index profile characterized by a uniform refractive index in the core and a sudden drop in the refractive index at the core-cladding interface, resulting in a lower refractive index in the cladding. The optical channel in this example is configured in a circular shape with an inner diameter of 200 mm and an outer diameter of 380 mm. The inner material of the optical channel has a refractive index n greater than the wall material. The refractive index has a step-like shape. The input pulse has a pulse profile at the input of the optical channel as shown diagrammatically in the profile diagram, with a maximum in the center of the profile and decreasing towards the ends. When passing through the optical channel, the input pulse is reflected at the transition from the inner material to the wall material. Different parts of the input pulse are reflected at different reflection angles, resulting in an attenuation of the input pulse. The resulting output pulse has a profile as shown diagrammatically in the pulse diagram at the output of the optical channel. The maximum of the profile of the output pulse is less pronounced than the maximum of the profile of the input pulse.

[0075] According to the second example b), a refractive index dispersion fiber is used. A refractive index dispersion is an optical fiber with a core whose refractive index decreases with increasing radial distance from the optical axis of the fiber. The optical channel in this example is configured in a circular shape with an inner diameter of 50-100 mm and an outer diameter of 125 mm. The refractive index is not of a step shape, but increases gradually towards the center of the optical channel and is practically constant in the area of ​​the inner diameter. The input pulse has, at the input of the optical channel, a pulse profile as shown diagrammatically in the profile diagram, with a maximum in the center of the profile and decreasing towards the ends. When passing through the optical channel, the input pulse is gradually reflected by the refractive index profile as shown diagrammatically in the diagram. The resulting output pulse has, at the output of the optical channel, a profile as shown diagrammatically in the pulse diagram. In a refractive index dispersion fiber, the attenuation of the pulse maximum is less pronounced than in the example a) of a step index fiber.

[0076] According to a third example c), a single mode optical fiber (SMF) is used. A single mode optical fiber, also known as fundamental mode fiber or monomode fiber, is an optical fiber designed to propagate only a single mode of light, the transverse mode. The optical channel in this example c) is configured in a circular shape with an inner diameter of less than 10 mm and an outer diameter of 125 mm. The material inside the optical channel has a refractive index n greater than the material of the walls. As in example a), the refractive index has a step-like shape. However, due to the narrow configuration, only a single mode of the laser illuminator passes through the optical channel. The input pulse has a pulse profile at the input of the optical channel as shown diagrammatically in the profile diagram, with a maximum in the center of the profile and decreasing towards the ends. The single mode is propagated by the optical channel substantially without loss. The resulting output pulse has a profile at the output of the optical channel as shown diagrammatically in the pulse diagram. The profile of the output pulse is substantially the same as the profile of the input pulse.

[0077] FIG. 7 shows an example of coupling a laser diode to an optical channel of a multi-sensing pixel array of an embodiment. In this example, ball lens coupling technology is utilized. The multi-sensing pixel array includes an illumination layer, a logic layer, and a pixel layer. The illumination layer includes a VCSEL 71 configured to generate laser light that is directed to the logic layer and the pixel layer. An absorber 72 in the logic layer directs light to a ball lens 73 that focuses the light into an optical fiber 74. The optical fiber 74 is fixed to a sleeve 75 that acts as a mechanical holder. Two ToF pixels 76 are disposed in the pixel layer. The optical fiber 74 passes between the two ToF pixels 76 and directs the laser light of the VCSEL out. The ball lens 73 has several optical properties that make it particularly suitable for coupling laser light into an optical fiber, such as a short focal length and a large aperture. The ball lens 73 is also mechanically symmetric, making it easy to align and center. Ball lens 73 resides inside a sleeve 75 at the end of optical fiber 74 and is self-aligning for ease of alignment.

[0078] The sleeve 75, which acts as a mechanical holder, may be, for example, a dielectric material.

[0079] In the example of Figure 7, a ball lens coupling technique is used to couple the laser light into the optical fiber, however, it should be noted that in alternative embodiments, other coupling techniques such as butt coupling can be applied.

[0080] execution FIG. 8 illustrates a schematic of an embodiment of a device utilizing a multi-sensing pixel array as described in the above embodiments. The electronic device 2100 may further perform all other processing of standard RGB, IR, iToF, dToF, PC, or Spot ToF systems. The electronic device 2100 includes a CPU 2101 as a processor. The electronic device 2100 further includes a multi-sensing pixel array 2106 connected to the processor 2101. The processor 2101 may perform, for example, RGB and depth measurement. The electronic device 2100 further includes a user interface 2107 connected to the processor 2101. The user interface 2107 serves as a man-machine interface and allows an administrator to interact with the electronic system. For example, the administrator may use the user interface 2107 to configure settings for the system. The electronic device 2100 further includes a Bluetooth interface 2104, a WLAN interface 2105, and an Ethernet interface 2108. These units 2104, 2105 serve as I / O interfaces for data communication with external devices. For example, other devices with Ethernet, WLAN or Bluetooth connections may be coupled to the processor 2101 via these interfaces 2104, 2105, 2108. The electronic device 2100 further comprises a data storage 2102 and a data memory 2103 (here a RAM). The data storage 2102 is arranged as a long-term storage, for example to store parameters of one or more use cases, to record sensor data obtained from the multi-sensing pixel array 2106, etc. The data memory 2103 is arranged to temporarily store or cache data or computer instructions for processing by the processor 2101.

[0081] It should be noted that the above description is merely an example configuration: alternative configurations may be implemented using additional or other units, sensors, etc.

[0082] Also, note that the division of the device in FIG. 8 into units is for illustrative purposes, and this disclosure is not limited to any particular division of functionality in any particular unit.

[0083] Furthermore, it should be noted that embodiments are not constrained to a particular detection technology: RGB, IR, iToF, dToF, PC and SPAD technologies are listed as examples only.

[0084] It should also be appreciated that the embodiments describe methods having exemplary orders of method steps, however the specific order of method steps is provided for illustrative purposes only and should not be construed as binding.

[0085] All units and entities described in this specification and recited in the appended claims may, unless otherwise specified, be implemented as integrated circuit logic on a chip, for example, and the functions provided by such units and entities may, unless otherwise specified, be implemented by software.

[0086] To the extent that the embodiments of the present disclosure described above are implemented, at least in part, using software-controlled data processing apparatus, it will be understood that computer programs providing such software control, and the transmission, storage, or other media on which such computer programs are provided, are contemplated as aspects of the present disclosure.

[0087] The present technology can also be configured as described below. (1) A multi-sensing pixel array including sensing pixels (PIX_R, PIX_G, PIX_B, PIX_D) and active illumination (PIX_I) in a multi-layer stacked die. (2) The multi-sensing pixel array described in (1), wherein the sensing pixels (PIX_R, PIX_G, PIX_B, PIX_D) include a depth sensing pixel (PIX_D). (3) A multi-sensing pixel array as described in (1) or (2), wherein the sensing pixels (PIX_R, PIX_G, PIX_B, PIX_D) include an imaging pixel (PIX_R, PIX_G, PIX_B) and a depth sensing pixel (PIX_D). (4) A multi-sensing pixel array according to any one of (1) or (3), wherein the sensing pixels (PIX_R, PIX_G, PIX_B, PIX_D) and the active illumination (PIX_I) are arranged in a single pixel array. (5) A multi-sensing pixel array as described in any one of (1) or (4), wherein the sensing pixels (PIX_R, PIX_G, PIX_B, PIX_D) and the active illuminator (PIX_I) share a main optical stack (41). (6) A multi-sensing pixel array according to any one of (1) to (5), wherein the imaging pixels include RGB pixels (PIX_R, PIX_G, PIX_B) or IR pixels, or a combination thereof. (7) A multi-sensing pixel array according to any one of (1) to (6), wherein the depth sensing pixel includes a ToF pixel (PIX_D). (8) A multisensing pixel array according to any one of (1) to (7), comprising a primary optical stack (41), an imaging stack (42) and an illumination stack (43). (9) The multi-sensing pixel array of (8), wherein the primary optical stack (41) includes a primary lens (44). (10) A multi-sensing pixel array as described in (8) or (9), wherein the imaging stack (42) includes a microlens (45), a pixel array including RGB pixels (PIX_R, PIX_G, PIX_B), and a ToF pixel (PIX_D). (11) A multi-sensing pixel array according to any one of (8) to (10), wherein the illumination stack (43) includes an active illuminator (PIX_I). (12) A multi-sensing pixel array according to any one of (1) to (11), wherein the illumination stack (43) is arranged for each illumination light source (PIX_I) and includes a respective light channel (48) configured to direct illumination. (13) A multi-sensing pixel array according to any one of (1) to (12), wherein the optical channel (48) is configured as a fiber optic light guide having a step refractive index profile or a refractive index distribution profile, or as a single mode optical fiber. (14) The multi-sensing pixel array according to any one of (1) to (13), wherein for each active illuminator (PIX_I), a respective microlens (49) is arranged to create an illumination field (Fol) of the illuminator. (15) A multi-sensing pixel array described in any one of (1) to (14) configured to provide the same field of view (FoV) to the imaging pixels (PIX_R, PIX_G, PIX_B) and the depth sensing pixel (PIX_D). (16) A multi-sensing pixel array according to any one of (1) to (15), wherein pixel and cavity control is provided configured to activate only those ToF pixels actually required. (17) The multi-sensing pixel array according to any one of (1) to (16), wherein the multi-sensing pixel array is implemented in a single IC. (18) The multi-sensing pixel array according to any one of (1) to (17), wherein the multi-sensing pixel array is implemented according to an organic vertical stacking technology. (19) A multi-sensing pixel array described in any one of (1) to (18), wherein the coupling between the active illumination (PIX_I) and the optical channel (48) is performed using ball lens coupling technology.

Claims

1. A multi-sensing pixel array including sensing pixels and active illumination in a multi-layer stacked die.

2. The multi-sensing pixel array of claim 1 , wherein the sensing pixels include depth-sensing pixels.

3. The multi-sensing pixel array of claim 1 , wherein the sensing pixels include imaging pixels and depth-sensing pixels.

4. The multi-sensing pixel array of claim 1 , wherein the sensing pixels and the active illumination are arranged in a single pixel array.

5. The multi-sensing pixel array of claim 1 , wherein the sensing pixels and the active illuminator share a main optical stack.

6. The multi-sensing pixel array of claim 1 , wherein the imaging pixels include RGB pixels or IR pixels, or a combination thereof.

7. The multi-sensing pixel array of claim 1 , wherein the depth-sensing pixels include ToF pixels.

8. The multi-sensing pixel array of claim 1 , comprising a primary optical stack, an imaging stack, and an illumination stack.

9. The multi-sensing pixel array of claim 8 , wherein the primary optical stack includes a primary lens.

10. 10. The multi-sensing pixel array of claim 8, wherein the imaging stack includes a microlens, a pixel array including RGB pixels, and a ToF pixel.

11. The multi-sensing pixel array of claim 8 , wherein the illumination stack comprises an active illuminator.

12. 10. The multi-sensing pixel array of claim 1, wherein the illumination stack comprises a respective light channel disposed relative to a respective illumination source and configured to direct illumination.

13. 10. The multi-sensing pixel array of claim 1, wherein the optical channel is configured as a fiber optic light guide having a step refractive index profile or a distributed refractive index profile, or as a single mode optical fiber.

14. 2. The multi-sensing pixel array of claim 1, wherein for each active illuminator, a respective microlens is positioned to create a field of illumination for that illuminator.

15. 10. The multi-sensing pixel array of claim 1 configured to provide the imaging pixels and the depth sensing pixels with the same field of view.

16. 10. The multi-sensing pixel array of claim 1, wherein pixel and cavity control is provided that is configured to activate only those ToF pixels that are actually required.

17. The multi-sensing pixel array of claim 1 , wherein the multi-sensing pixel array is implemented on a single IC.

18. The multi-sensing pixel array of claim 1 , wherein the multi-sensing pixel array is implemented according to an organic vertical stacking technology.

19. 10. The multi-sensing pixel array of claim 1, wherein the active illumination and optical channel coupling is performed using ball lens coupling technology.

Images