Image sensor and sensing method for acquiring gated imaging information

Abstract

A gated camera system includes a light emitter that illuminates a scene by emitting a pulse and a gated camera for capturing light retroreflected from the scene. The gated camera includes an image sensor synchronized with the light emitter. The image sensor comprises a plurality of gated imaging pixels, each gated imaging pixel comprising: a photodiode (PD) for converting light into a charge; a plurality of N taps, each tap having a storage node for storing an amount of charge corresponding to light retroreflected from a specific delay period of the light pulse emission; and a plurality of N PGs, where each PG is associated with a different one of the N storage nodes, each PG can transfer charge from the PD to its associated storage node, and at least one of the N taps is configured to deploy a gated timing to generate an image formed by light signals retroreflected from a particular scene depth range.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 578,198, filed August 23, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments disclosed herein generally relate to digital cameras, and more particularly to gated digital cameras for two-dimensional (2D) and three-dimensional (3D) imaging applications. Background Technology

[0003] Gated imaging (GI) is a known active imaging technique used to image a scene (“3D slice”) within a specific range using a distance-gated imaging camera. It effectively penetrates obscuring media such as haze, fog, or poor lighting conditions, improving image sharpness. “Active” here refers to the use of a light emitter (or “illuminator”) to emit light for a specific period of time. In gated imaging, the emission of the light emitter is synchronized with the operation of the gated imaging camera. Gated imaging is based on the time-of-flight (ToF) principle. In addition to two-dimensional images, gated imaging can also provide three-dimensional or depth information.

[0004] Figure 1A exemplarily illustrates a known Time-of-Flight (ToF) camera system 100, which operates according to the ToF principle for capturing a 3D scene. The ToF camera system 100 includes an emitter 102 and a camera 104. The emitter 102 illuminates the scene within a specific wavelength range. The camera 104 can operate within said specific wavelength range to acquire three-dimensional image data of the scene. This three-dimensional image data is referred to as a “3D map” or “depth map.” It should be noted that a 3D camera (e.g., camera 104) can also be used to acquire two-dimensional image data of the scene. For example, the ToF camera system 100 can acquire both two-dimensional and three-dimensional image data of an object (e.g., object 106) located within its field-of-view (FOV). The ToF camera system 100 acquires 3D information by measuring the propagation time of the following light: (1) light emitted by the emitter 102; (2) light reflected back from object 106; and (3) light returning to the ToF camera system 100. The time of light propagation is converted (or “translated”) to the distance (D) from the object to the camera, i.e., the depth. Therefore, D and the corresponding time of light propagation (T) are equivalent. In this paper, D and T are used interchangeably. Camera 104 includes a time-of-flight (ToF) image sensor with high temporal resolution, reaching the nanosecond (ns) level or even shorter.

[0005] Figure 1B exemplarily illustrates a known GI system 110, which includes a transmitter 112 and a GI camera 114. In the GI system, the transmitter 112 can repeatedly emit light pulses with a specific pulse length. For the GI system, an image sensor (“imager”) included in the GI camera 114 receives back-reflected light with a specific delay period (or simply “time delay”) relative to the light pulse emission period (referred to as gating timing, or “GT”) within a specific time period. In other words, the GI camera only images a specific 3D slice of the scene. This means that the GI camera can only image scene data within a specific 3D slice. Scene data located at smaller or larger distances cannot be imaged. A specific imaging 3D slice corresponds to a specific gating timing.

[0006] For example, the GI system 110 can be used to distinguish three 3D slices of a scene. The first 3D slice contains a distance (or depth) range "d". GT1 ", corresponding to the first gating time ("GT1"). The distance ranges corresponding to the second and third 3D slices are respectively called "d". GT2 " and "d GT3 Here, there is no overlap between the three 3D slices. In other examples, these three 3D slices may overlap.

[0007] As shown in Figure 1B, at d GT1 There is no (or "location") of any object in d GT2 There is a circular object 116, d GT3 A rectangular object 118 exists in the scene. When the imaging system 110 images the scene, it generates a first input image (“II”) 120, a second input image 122, and a third input image 124. The first input image 120 contains only images from d. GT1 The scene information, the second input image 122 only contains information from d GT2 The scene information, the third input image 124 only contains information from d GT3 Scene information. Therefore, due to d GT1 The input image 120 does not contain any objects, nor does it contain any objects. GT2 The second input image 122 contains the input image 126 containing the circular object 116. G3 Since it includes a rectangular object 118, the third input image 124 includes the input image 128 of the rectangular object 118.

[0008] In some examples, the three input images described above can be used to generate an output image, which can be a 3D map or a 2D image. In some examples, all three input images (120, 122, and 124) can be used to compute a 3D map. In other examples, all three input images can be used to compute a 2D image of the scene. The advantage of a 2D image of the scene is that it has relatively good visibility across the entire imaging range of the GI system 110. “Relatively good visibility across the entire imaging range” means that the visibility of the GI system 110 is better than that of a conventional (non-GI) camera, such as a conventional RGB camera, across the entire imaging range or in certain portions of the imaging range. For example, better visibility can be obtained in fog because GI can suppress backscattering caused by illuminating the scene. For example, better imaging at night because GI can image the scene depth more uniformly compared to a headlight, that is, the signal-to-noise ratio (SNR) difference between distant and near scene areas is smaller. For example, this better imaging effect is beneficial for detecting objects within the imaging range with a higher probability, which is crucial for computer vision.

[0009] By repeatedly emitting light at a specific ground plane (GT) and receiving the back-reflected light, a 3D sliced ​​input image associated with the corresponding GT can be obtained. The signal-to-noise ratio (SNR) of the input image depends on the number of repetitions. Generally, the more repetitions, the higher the SNR.

[0010] Figure 1C illustrates an embodiment of a known GI pixel 130 for performing GI. GI pixel 130 is a 1-tap pixel used in a GI system as shown in 110. Charge is generated in a photodiode (“PD”) and stored in a single storage node (or “tap”) C assigned to a photogate (“PG”). The PG controls the charge flowing to the storage node C, i.e., the transfer of charge to C. The charge in C is used to generate an input image, such as a first input image 120, a second input image 122, and a third input image 124. Here, all the charge stored in C corresponds to the light arriving at GI pixel 130 within a specific GI timing sequence. The charge in C is converted into an output signal (“VOUT”) by a source follower (“S / F”) acting as a buffer. GI pixel 130 may include a drain gate (“DG”), for example, for draining the PD, resetting the PD, etc. “Draining” the PD helps reduce errors in two-dimensional or three-dimensional measurements.

[0011] Image sensors with multiple GI pixels (e.g., GI pixel 130) can image a 3D slice within each frame readout cycle. For example, generating a first input image 120, a second input image 122, and a third input image 124 requires three consecutive readout cycles. Typically, computer vision requires a video stream of output images. This video stream is captured at a certain frame rate, such as 20 or 30 frames per second (fps). As mentioned earlier, an output image is typically generated by using multiple input images. Each frame corresponds to one output image, so 20 or 30 fps corresponds to 60 or 90 input images per second. Generally, high frame rate video streams are more beneficial. It is important to note that, as shown in Figure 1D, excessive readout cycles are undesirable from a frame rate perspective.

[0012] Figure 1D shows a known timing diagram 140 for a 1-tap pixel (e.g., pixel 130) used to perform GI. Timing diagram 140 shows the "output frame time" or "T". OUTF "T" refers to the time (or "period") required to acquire one output image (or one output frame in this document) generated from three input images. The first input image (e.g., 120) is captured at the first image capture time indicated by block 142 ("T"). IM It was captured during the period and read out at time T. R-O During this period, it is read out. Subsequently, the second input image (e.g., 122) is read out in the second T, indicated by block 144. IM Captured during the period, and in T R-O During this period, it is read out. Then, the third input image (e.g., 124), as indicated by block 146, is read out in the third T... IM Captured during the period, and in T R-O It was read out during this period. Overall, T OUTF = 3×TF. To obtain an output video stream of 30 fps, a single tap pixel of gated imaging needs to generate 90 fps (or 90 input images per second). For simplicity, assume the image capture time T for the three captured input images. IM They are all the same.

[0013] Using relatively low-cost image sensors in gated imaging camera systems to provide relatively high frame rates of gated images would have significant advantages. Summary of the Invention

[0014] Various exemplary embodiments provide a gating camera system and related methods for performing gating imaging.

[0015] In some exemplary embodiments, a method includes: providing a light emitter and an image sensor synchronized with the light emitter, the image sensor including a first plurality of M imaging pixels and a second plurality of N storage nodes, each imaging pixel including a photodiode (PD); emitting a light pulse through the light emitter to illuminate a scene; receiving, at the image sensor, light back-reflected from the scene after a specific time delay from the emission of the light pulse, the specific time delay defining a specific scene depth range; storing a first charge corresponding to the first specific scene depth range in the first storage nodes; storing a second charge corresponding to the second specific scene depth range in the second storage nodes; and forming an image of the scene using the first charge and the second charge.

[0016] In some exemplary embodiments, the associated gated camera system includes a light emitter for illuminating a scene and an image sensor synchronized with the light emitter. The image sensor includes a first plurality of M imaging pixels and a second plurality of N storage nodes. Each imaging pixel includes a photodiode (PD). The image sensor is used to receive light reflected back from the scene after a specific time delay from the emission of the light pulse, the specific time delay defining a specific scene depth range. The first storage node is used to store a first charge corresponding to the first specific scene depth range. The second storage node is used to store a second charge corresponding to the second specific scene depth range. The first charge and the second charge are used to form an image of the scene.

[0017] In some examples, the method further includes storing a third charge corresponding to a third specific scene depth range in a third storage node, and using the first charge, the second charge, and the third charge to form an image.

[0018] In some examples, the method further includes storing a fourth charge corresponding to a fourth scene-specific depth range in a fourth storage node, and using the first, second, third, and fourth charges to form an image.

[0019] In some examples, the method also includes: depleting the PD by generating a third charge and storing the third charge in a third storage node.

[0020] In some examples, the method also includes: measuring ambient lighting in the scene by storing a third charge in a third storage node, and using the first, second, and third charges to form an image.

[0021] In some exemplary embodiments, a method includes: providing a light emitter and an image sensor synchronized with the light emitter, the image sensor including a first plurality of M imaging pixels and a second plurality of N storage nodes, each imaging pixel including a photodiode (PD); emitting a light pulse through the light emitter to illuminate a scene; receiving, at the image sensor, light back-reflected from the scene after a specific time delay from the emission of the light pulse, the specific time delay defining a specific scene depth range; storing a first charge corresponding to the first specific scene depth range in the first storage nodes; depleting the photodiodes by generating a second charge and storing the second charge in the second storage nodes; and using the first charge to form an image of the scene.

[0022] In some exemplary embodiments, the associated gated camera system includes a light emitter for illuminating a scene and an image sensor synchronized with the light emitter. The image sensor includes a first plurality of M gated imaging pixels and a second plurality of N storage nodes. Each gated imaging pixel includes a photodiode (PD). The image sensor is used to receive light reflected back from the scene after a specific time delay from the emission of the light pulse, the specific time delay defining a specific scene depth range. The first storage node is used to store a first charge corresponding to the first specific scene depth range. The second storage node is used to store a second charge generated by the depletion of the photodiode. The first charge is used to form an image.

[0023] In some exemplary embodiments, a method includes: a light emitter and an image sensor synchronized with the light emitter, the image sensor including a first plurality of M imaging pixels and a second plurality of N storage nodes, each imaging pixel including a photodiode (PD); emitting a light pulse through the light emitter to illuminate a scene; receiving, at the image sensor, light back-reflected from the scene after a specific time delay from the emission of the light pulse, the specific time delay defining a specific scene depth range; storing a first charge corresponding to the first specific scene depth range in the first storage nodes; measuring ambient illumination in the scene by storing a second charge in the second storage nodes; and forming an image of the scene using the first charge and the second charge.

[0024] In some exemplary embodiments, the associated gated camera system includes a light emitter for illuminating a scene and an image sensor synchronized with the light emitter. The image sensor includes a first plurality of M gated imaging pixels and a second plurality of N storage nodes. Each gated imaging pixel includes a photodiode (PD). The image sensor is used to receive light reflected back from the scene after a specific time delay from the emission of the light pulse, the specific time delay defining a specific scene depth range. The first storage node is used to store a first charge corresponding to the first specific scene depth range. The second storage node is used to store a second charge generated by measuring ambient light in the scene. The first charge and the second charge are used to form an image of the scene.

[0025] In some examples, N = 3-8. In some examples, N = 4.

[0026] In some examples, M ranges from 0.1 megapixels (MP) to 4 MP.

[0027] In some examples, the pixel size of the imaging pixels ranges from 1 µm to 10 µm.

[0028] In some examples, the imaging pixels can be operated to perform pixel merging.

[0029] In some examples, light emitters and image sensors are used to perform adaptive gating imaging.

[0030] In some examples, the image is a 3D image or a 2D image.

[0031] In some examples, the system is contained within a mobile device, and the method is executed on the mobile device. In some examples, the mobile device is a smartphone.

[0032] In some examples, the gating camera system is contained within a vehicle, and the method is executed within the vehicle.

[0033] In some examples, the gated camera system is configured to compute a 3D image / video stream with a frame rate greater than or equal to 30 frames per second (fps). In some examples, the gated camera system is configured to compute a 2D image / video stream with a frame rate greater than or equal to 30 frames per second (fps). In some examples, the frame rate is greater than or equal to 45 fps. In some examples, the frame rate is greater than or equal to 45 fps. In some examples, the frame rate is greater than or equal to 60 fps. In some examples, the frame rate is greater than or equal to 60 fps.

[0034] In some examples, the minimum distance (D) within the depth range is 1 meter to 250 meters. In other examples, D is 15 meters to 200 meters.

[0035] In some examples, the thickness T in the depth ranges from 5 meters to 200 meters. In other examples, T ranges from 10 meters to 150 meters.

[0036] In some examples, N=4, the gated camera system operates in a 1-pulse-1 gate configuration, a 1-pulse-4 gate configuration, a 2-pulse-4 gate configuration, or a 3-pulse-4 gate configuration.

[0037] In some examples, N=3, and the gated camera system operates in a 1-pulse-1 gate configuration, a 1-pulse-3 gate configuration, or a 2-pulse-3 gate configuration.

[0038] In some cases, N=2, and the gated camera system operates in a 1-pulse-1 gate configuration or a 1-pulse-2 gate configuration.

[0039] In some examples, N=4, and in order to generate an image containing light signals reflected back from a specific scene depth range, the gated camera system operates in a 1-pulse-4 gate configuration during the first time period and in a 1-pulse-1 gate configuration during the second time period.

[0040] In some examples, the GI pixel can be manipulated to perform pixel merging. The merging method can be 4-bining.

[0041] In some cases, the camera system can perform adaptive gating imaging.

[0042] In some examples, the gated camera system operates in the near-infrared wavelength range. In other examples, the gated camera system operates in the short-wave infrared wavelength range.

[0043] In some examples, the mobile device includes a gating camera system as described above or below. In some examples, the mobile device is a smartphone. In some examples, the mobile device is an augmented reality headset or a virtual reality headset.

[0044] In some cases, the vehicle may be equipped with a gate camera system as described above or below. Attached Figure Description

[0045] The embodiments disclosed herein are described below with reference to the accompanying drawings, which are not intended to be limiting. Identical structures, elements, or components appearing in the various drawings are generally designated using the same numerals throughout. The drawings and descriptions are intended to illustrate and explain the embodiments disclosed herein and should not be construed as limiting in any way.

[0046] Figure 1A shows a known ToF system; Figure 1B shows a known gated imaging system; Figure 1C shows known gated imaging pixels; Figure 1D shows a known temporal overview associated with the gated imaging pixels in Figure 1C; Figure 2A The four-tap-gated imaging pixels of this disclosure are shown; Figure 2B This disclosure shows that, with Figure 2A Timing overview of pixel-related parameters in mid-gated imaging; Figure 2C This disclosure shows a 2-tap gated imaging pixel; Figure 2D This disclosure shows that, with Figure 2C Timing overview of pixel-related parameters in mid-gated imaging; Figure 2E The pixel components in high-resolution mode of a known image sensor are shown; Figure 2F Showing low-resolution mode Figure 2E Pixel components in; Figure 3A A timing diagram of this disclosure is shown; Figure 3B Another timing diagram of this disclosure is shown; Figure 4A Another timing diagram of this disclosure is shown; Figure 4B Another timing diagram of this disclosure is shown; Figure 4C Another timing diagram of this disclosure is shown; Figure 5 Another timing diagram of this disclosure is shown; Figure 6 A system for implementing gated imaging as disclosed herein is shown; Figure 7 The steps for implementing the gating imaging method of this disclosure are shown.

[0047] definition Table 1 provides the definitions used in this paper.

[0048]

[0049] Table 1 Detailed Implementation Figure 2AAn embodiment of a “4-tap” (or “4-barrel”) pixel, designated 200, is shown. All such 4-tap (or more generally N-tap) pixels disclosed herein can perform the gated imaging (“GI”) disclosed herein. Generally, all N-tap pixels and their applications in gated imaging camera systems described below are “as described herein” (therefore the term will not be repeated). The 4-tap pixel 200 can be used in a gated imaging camera. An image sensor containing multiple 4-tap pixels 200 can be deployed simultaneously with up to four GTs. In other words, a single 4-tap pixel 200 can simultaneously image up to four 3D slices within one frame time, i.e., within each readout cycle (or “cycle”). Thus, the 4-tap pixel 200 can generate up to four input images (“IIs”, referring to up to four “3D slices” here) within each readout cycle. As previously mentioned, a 1-tap pixel 130 can only generate one 3D slice per readout cycle (Figure 1D). Therefore, when using 1 tap pixel 130, each 3D slice is captured at a different time (Figure 1D), while when using 4 tap pixels 200, up to four 3D slices can be captured simultaneously.

[0050] The corresponding input images (3D slices) are typically obtained by repeating gating cycles, as shown in Table 2. The up to four input images may include input images such as 120, 122, and 124, and may even optionally include a fourth input image. For example, generating input images 120, 122, and 124 requires only one readout cycle. These four input images can be combined into a single output image, which can then be used to generate depth information. In other words, a GI camera with 4 taps and 200 pixels can generate 2D and / or 3D image data at relatively high frame rates, such as 30 frames per second (fps), 40 fps, 60 fps, or even higher.

[0051] The 4-tap pixel 200 contains a photodiode (“PD”). The charge generated by the PD is stored in 4 memory nodes C. A -C D In the middle, they are respectively allocated to PGA, PGB, gate C (PGC), and gate D (PGD). The gate control directs the flow to C. A -C D The charge stored in C. A -C D All charges correspond to the light arriving at pixel 200 within the first, second, third, and fourth GI timing sequences GT1-GT4. The light gates operate sequentially (e.g., as...). Figure 3AAs shown). The charge in each storage node is converted into an output signal (V) by a source follower ("source follower, S / F") that acts as a buffer. OUT ).

[0052] In some examples, a 4-tap pixel 200 can be used to perform indirect time-of-flight (i-ToF) measurements and can be configured to perform GI. Similarly, 2-tap or 8-tap i-ToF pixels (or more generally, N-tap i-ToF pixels) can also be used to perform GI. Using such pixels offers advantages in terms of availability, performance, and cost. N-tap pixel technology is available from multiple vendors and offers relatively high performance at a relatively low price. N-tap i-ToF pixels can operate in the near-infrared (NIR) or short-wave infrared (SWIR) wavelength range.

[0053] Figure 2B Timing diagram 210 is shown for a 4-tap pixel (e.g., pixel 200). Timing diagram 210 shows the frame time T. F Timing diagram 210 also shows the time T required to capture three input images and generate one output image. OUTF The image capture time T used in timing diagram 210 IM The time is the same as that used for multiple input images (3D slices) in Figure 1D. Although up to four 3D slices can be captured simultaneously, only three 3D slices are captured simultaneously here. Each 3D slice is generated by charges stored in a different storage node. For example, these three 3D slices are generated using charges stored in C. A C B and C C The charge generated in C. D The charge in the 3D slice can be used for other tasks, such as depleting the PD or measuring ambient lighting in a scene. The capture time for each 3D slice can be three times longer than the time shown in timing diagram 140. Meanwhile, as shown in block 212, in 3×T... IM The first 3D slice is captured during this period (e.g., 120); as shown in block 214, in 3×T IM During this period, a second 3D slice is captured (as shown in 122); as shown in block 216, in 3×T IM A third 3D slice (e.g., 124) is captured during this period. Subsequently, these three 3D slices are read out at time T. R-O Simultaneously read out during this period, which is typically much longer than the image capture time T. IM Even longer than 3×T IM Overall, T OUTF = T FCompared to timing diagram 140, by saving two readout cycles, T OUTF Reduced by 2×T IM This is beneficial for achieving a relatively high frame rate. For simplicity, here we assume that T for all gating is... IM They are all the same. Generally speaking, the T value of each 3D slice is... IM They vary. Specifically, the T of the first 3D slice is usually... IM T of the second 3D slice farther from the gated camera IM Shorter. The T of the second 3D slice IM The T-slice, which is shorter than the third 3D slice that is farther from the door camera, is also shorter. IM And so on. Using variable T IM The implementation scheme helps to achieve relatively high signal-to-noise ratio (SNR) uniformity between different 3D slices.

[0054] Figure 2C An embodiment of the two-tap pixel 210 is shown. The two-tap pixel 210 can be used in a GI camera system. The two-tap pixel 210 is capable of simultaneously capturing two different 3D slices (in a T... F The generated charge is stored in two storage nodes C allocated to the PGA and PGB. A -C B In. Stored in C. A -C B The charge in the image corresponds to the light reaching the 2-tap pixel 210 within the first and second specific GTs. It should be noted that the 2-tap pixel 210 can perform up to two 3D slice captures per readout cycle. That is, the GI pixel 2-tap pixel 210 can generate up to two input images per readout cycle. These up to two input images may include input images such as a first input image 120 and a second input image 122. For example, generating the first input image 120, the second input image 122, and the third input image 124 requires two readout cycles.

[0055] Figure 2D Timing diagram 230 is shown for a 2-tap pixel (e.g., pixel 220). Timing diagram 230 shows the time T required to generate one output image from three input images. OUTF Since the 2-tap pixel has two C's, up to two 3D slices can be captured simultaneously. The image capture time T used in timing diagram 230 is... IM The capture time is the same as that used for the input image (3D slice) in Figure 1D. Specifically, since two 3D slices are captured simultaneously, each T... IM The time is twice that in timing diagram 140. Simultaneously, the first input image (e.g., 120) is processed at 2×T as indicated by block 232. IMDuring the capture, the second input image (e.g., 122) is captured in a 2×T array as indicated by block 234. IM Captured during the period. Then, at readout time T. R-O Two input images are read simultaneously. Then, at T, as indicated by block 236... IM A third input image (e.g., 124) is captured during this period. Overall, T OUTF = 2×T F Compared to timing diagram 140, T OUTF Reduced by one T F This saves time, specifically a read cycle. This is beneficial for achieving a relatively high frame rate.

[0056] For the sake of generalization, it is recommended to use "N-tap" (or "multi-tap") pixels. An N-tap refers to N storage nodes (C) and one photodiode (PD). Each C is used to store a different charge, with different charges corresponding to different 3D slices (or more generally, different input images). The generated charges are stored in the N storage nodes C. A -C N In this context, these storage nodes are allocated to PGA, PGB, ..., PGN, respectively. The storage is located in C... A -C N The charges in the N-tap pixel correspond to the light reaching the N-tap pixel within the first, second, ..., Nth specific ground planes (GTs). This N-tap pixel can deploy up to N different GTs to achieve a certain frequency distribution within a single T-wavelength. F Up to N different 3D slices can be captured simultaneously within one readout cycle. In other words, N tap pixels can be captured in one T... F Up to N input images can be generated. The N tap pixels can generate 2D and / or 3D image data at relatively high frame rates, such as 30 frames per second (fps), 40 fps, 60 fps, or even higher. In some embodiments, one or more of the N Cs are used for other purposes, such as depleting a photodiode (PD) or measuring ambient light in a scene.

[0057] In other N-tap imaging embodiments, the N-tap pixels may employ known pixel technologies, such as current-assisted photodiode (CAPD) or lateral electric field control (LEFC), or any other technology employing N-tap pixels.

[0058] In some embodiments, an image sensor including 4 tap pixels 200 or 2 tap pixels 220 can perform pixel merging operations.

[0059] Figure 2EA pixel assembly 250 of an image sensor is shown, which, in a first configuration, contains four individual pixels numbered 1 to 4. Each of these four individual pixels can be a 4-tap pixel 200 or a 2-tap pixel 220, or more generally, an N-tap pixel. These four individual pixels capture scene information independently of each other. That is, when capturing an image, each of the four individual pixels 1 to 4 provides a different pixel value matrix, which contains values ​​for each of the N storage nodes. In total, 4 × N values ​​are provided. We refer to this configuration as "higher resolution mode" (HRM) or "full resolution mode".

[0060] Figure 2F A second configuration of pixel component 250 is shown. These four pixels are merged (or “binded”) into one pixel and do not independently capture scene information. That is, when capturing the input image, these four merged pixels collectively provide a pixel value matrix containing the values ​​corresponding to N storage nodes. In total, N values ​​are provided. We refer to this type of configuration as “binning mode” or “lower resolution mode” (LRM). Specifically, we call this second configuration of merging 4 pixels into one pixel “4-binning”. In other examples, 9 pixels can be merged into one pixel (“9-binning”).

[0061] It should be noted that the area (or “region”) of a single pixel is reduced in HRM compared to LRM. The extent to which the area of ​​a single smaller pixel in HRM is reduced relative to the area of ​​a larger merged pixel in LRM depends on the number of pixels merged in their respective lowest pixel resolution modes. In 4-binning mode, the area is reduced to 1 / 4 of its original size; in 9-binning mode, the area is reduced to 1 / 9 of its original size, and so on. This area reduction results in a decrease in the amount of light entering (or “captured”) a single smaller pixel, which can be measured, for example, by the number of photons entering a single smaller pixel per unit time. The amount of light reduction in a single smaller pixel is proportional to the area of ​​that pixel. Therefore, timing examples (“TEs”) can be adjusted (or “modified”) to compensate for these changes in the amount of captured light.

[0062] In some examples that employ pixel merging techniques in HRM ( Figure 2EEven so, an entire pixel component (such as 250) can still be treated as a single pixel. For example, pixel 1 in pixel component 250 could use the first timing example (“TE”), pixel 2 in pixel component 250 could use the second TE, and so on. For example, using TE 300, a pixel component for 4-binning (such as pixel component 250) could deploy 16 different gating timings, meaning it could capture 16 different 3D slices of the scene. This helps to obtain a relatively “high depth resolution” or “detailed 3D output image”. In other words, higher depth resolution can still be achieved while using a lower pixel (or “spatial”) resolution.

[0063] Figure 3A An embodiment of TE for a 4-gate camera system, number 300, is shown. The term "4-gate" indicates that four different gating sequences can be used simultaneously, thereby enabling the use of a single TE. F Four 3D slices are captured. The 4-gate camera system, as well as all N-gate camera systems disclosed herein, can use N-tap pixels. When using 4-tap pixels (e.g., 4-tap pixel 200), different 3D slices can be captured simultaneously using all four taps by employing four different gating timings (GT). When using 2-tap pixels (e.g., 2-tap pixel 220), two 3D slices can be captured simultaneously using two taps in the first measurement, with the remaining two 3D slices captured in a subsequent second measurement. TE 300 illustrates the synchronization activity of the components in the 4-gate camera system, specifically the N-tap pixels and the emitter. The vertical time axis is shown in the figure. TE 300 shows the complete 4-tap sequence period 302 of the 4-gate camera system. Furthermore, the starting point of the second 4-tap sequence period 304 is also shown. The values ​​shown in TE 300 are shown in Table 2. In other examples, the values ​​may differ (or “varie”) from those given in Table 2, for example, by ±10%, ±50%, or even ±100%. This also applies to all TEs disclosed herein. A 4-door camera system may include a transmitter (e.g., transmitter 112) and pixels (e.g., pixel 200) – each pixel containing four storage nodes C. A -C D These are assigned to four optical gates (PGA-PGD) and one drain gate (DG). In each sequence period, four taps are deployed sequentially, each with a gated period time T1-T4. The time period during which the transmitter is active (or "on") to transmit pulses is called "T". E ", also known as "transmitter pulse length", such as Figure 3AThe first column is shown and labeled 306.

[0064] Please note that in some examples, the transmitter may operate in constant power mode, i.e., at T... E At any given moment during this period, the transmitter's output power remains constant. In other examples, the transmitter might employ a pulse mode, i.e., during T... E During this period, the transmitter's output power oscillates between an upper and lower limit, meaning the output power varies, with its average (or "effective") output power falling between the upper and lower limits. "T D "" represents the measured delay time from the end of light pulse emission to the activation of the light gate (PG). Figure 3 shows various such delay times. D Marked as "T" D1 "to "T D4 ".T D The minimum distance (“D”) between the 4-door camera system and the 3D slice is defined. Min ”, where D Min = TD / 2×c, where c is the speed of light. Figure 3A In the middle, the corresponding time period T during which PGA, PGB, PGC, and PGD are in an active state. PG They are displayed in the second, third, fourth, and fifth columns (from right to left), and labeled 308, 310, 312, and 314 respectively. E T D and T PG They jointly defined the maximum distance ("D") between the 4-door camera system and the 3D slice. Max ”, where D Max = (T E +T D +T PG ) / 2×c. In other words, D Min This represents the minimum distance contained within a specific 3D slice, while D... Max This represents the maximum distance contained within a specific 3D slice. In Table 2, D Min and D Max These represent the first and second values ​​of the given range in the "range" column, respectively.

[0065] In a 4-gate camera system, the DG prevents parasitic charges that manifest as noise (unwanted image information) in the captured images. "Parasitic charges" refer to irrelevant charges, such as those from ambient light, that are captured by the photodiode (PD) when the no-light gate (PG) is activated. Figure 3AThe sixth column, marked 316, shows the time period during which DG is active. DG is always active when PGA, PGB, PGC, or PGD are all inactive. "#cycles" indicates the number of times the gating cycle is repeated. As shown in the figure, the gating cycle time for the i-th tap (i=1, 2, 3, 4) is T. i = T Ei + T Di + T PGi The time complexity of a single 4-tap sequence is T. SEQ = T1 + T2 + T3 + T4. However, T SEQ At capture time T IM The internal contents may change.

[0066] As shown in the “Range” column of Table 2, in TE 300, 3D slices associated with different gating timings partially overlap at their edge regions. This may be desirable for depth calculations and uniform illumination among multiple 3D slices. In other examples, 3D slices may not overlap with each other.

[0067]

[0068] Table 2 Please note that, under normal circumstances, T SEQ During capture time T IM The internal structure changes. For example, taking the TE 300 as an example, we found that the number of cycles (#cycles) varies for different ground planes (GTs). Specifically, the #cycles of GT 1 are less than those of GT 2, which are less than those of GT 3, and so on. As the distance between the 3D slice and the camera increases, the #cycles increase to compensate for the decrease in the intensity of backscattered light. In some examples, such as... Figure 3A As shown, at the beginning of a frame, taps 1-4 (PGA-PGB) may be activated sequentially until #cycles reaches 500. From this sequence onward, only taps... Figure 3A The sequence shown activates taps 2-4 (PGB-PGD) sequentially until #cycles reaches 1000. From this sequence onward, only taps... Figure 3A The sequence shown activates taps 3-4 (PGC-PGD) sequentially until #cycles reaches 2000. From this sequence onward, only tap 4 (PGD) can be activated sequentially until #cycles reaches 4000, after which all data is collected. This means the aforementioned capture time T... IM Given by the following formula: T IM= #cycles1×T1+ #cycles2×T2+ #cycles3×T3+ #cycles4×T4, where #cycles represents the #cycles value corresponding to each GT (i = 1-4).

[0069] In some examples, after completing the entire four PG activation sequence (e.g., sequence 302), the depth range can be offset (or "moved") in subsequent sequences as shown in the "Range" column of Table 2. This can be achieved, for example, by changing the delay time of one or more ground truths (GTs). For instance, in the first four PG activation sequence, the delay time T given in Table 2 could be used. D1 To T D4 In the second four-PG activation sequence, an updated delay time (“uT”) can be used. D "), where uT D1 = T D1 +ΔT1 to uT D4 = T D4 + ΔT4. ΔT1 to ΔT4 may be the same or different.

[0070] In the TE of other 4-camera systems, the timing shown in Table 3 can be used. All definitions and descriptions in Table 2 also apply to Table 3.

[0071]

[0072] Table 3 The TE 300 operates in a "1 light pulse - 1 tap" configuration. This means that when any one of the four PGAs-PGDs is activated, a light pulse is emitted. In total, the activation of the four different PGAs will emit four light pulses respectively.

[0073] Figure 3B Another embodiment of the TE for a 4-door camera system, designated 320, is shown. Unless otherwise stated, the description of TE 300 also applies to TE 320. TE 320 shows the complete four PG activation sequences 322 and the beginning portion of the second sequence 324. Here, TE is designated 326. Figure 3B In the middle, the corresponding time period T during which PGA, PGB, PGC, and PGD are in an active state. PG They are displayed in the second, third, fourth, and fifth columns, respectively, and labeled as 328, 330, 332, and 334.

[0074] The values ​​shown in TE 320 are listed in Table 4. As shown in the figure, only one has T. E1A single light pulse is emitted. This single light pulse is sampled by all four taps by sequentially activating the four gated cameras (PGs). The TE 320 operates in a "1-pulse-4-tap" configuration. This means that one light pulse is emitted for the activation of all four PGs. In other TEs, the activation of the four PGs may emit two or three light pulses, referred to as the "2-pulse-4-tap" and "3-pulse-4-tap" configurations, respectively. It should be noted that the same 4-gate camera system can be used to switch (or "switch") between performing GI according to the TE 320 and TE 300. More generally, the same 4-gate camera system can be used to switch between performing GI in all TEs disclosed herein. The advantage of the TE 320 over the TE 300 is that a single light pulse can be used for multiple gating cycles, thereby capturing more back-reflected single light pulses.

[0075]

[0076] Table 4 As shown above, the gating cycle time of TE 320 differs from that of TE 300. The gating cycle time for taps i=1, 2, 3, 4 is determined by T. i = T E + T Di + T PGi Given. Where, T SEQ = T4, and is the same for all sequences. The total acquisition time for the four 3D slices is T. IM = #cycles4 × T4.

[0077] Figure 4A TE embodiment 400 for a 3-gate camera system is shown. "3-gate" refers to capturing three distinct 3D slices within the same frame time (i.e., one readout cycle). TE 400 illustrates the synchronization activity of the components in the 3-gate camera system. TE 400 shows the first set of PG activation sequences 402 and the second set of sequences 404. All definitions and interpretations in TE 300 also apply to TE 400.

[0078] The values ​​shown in TE 400 are listed in Table 5. Figure 4A In the context of the time period T during which PGA, PGB, and PGC are active,... PG These are displayed in the second, third, and fourth columns from right to left, labeled 408, 410, and 412 respectively. PGD or GD 414 or 416 ( Figure 4AThe fifth and sixth columns (in the image) can be used to deplete the PD and are marked as DG when activated. In a 3-gate camera system employing 4 tap pixels as shown in Figure 200, PGD 414 or DG 416 can be used to deplete the PD and prevent parasitic charges from appearing as noise in the captured image. This means that in some embodiments, DG is used to deplete the PD. DG requires a relatively fast switching time. In other embodiments, PGD is used to deplete the PD. When PGD is used to deplete the PD, the corresponding charge is stored in a storage node of the N-tap pixel, for example, in C. D In the figure, the tap gating cycle time for i=1,2,3 is determined by T. i =T Ei + T Di + T PGi Given. Where, T SEQ = T1 + T2 + T3. The TE 400 operates with a "1 optical pulse - 1 tap" configuration, that is, it emits three optical pulses for three different tap gate cycles.

[0079]

[0080] Table 5 Figure 4B Another embodiment of the TE for a 3-gate camera system, designated 420, is shown, employing 4-tap pixels. Unless otherwise stated below, the description of TE 400 also applies to TE 420. TE 420 shows the complete activation sequence 422 of the three PGs and the beginning of the second sequence 424. Here, this TE is labeled 426. Figure 4B In the context of PGA, PGB, and PGC, the corresponding T... PG Displayed in the second, third, and fourth columns (from right to left), and labeled 428, 430, and 432 respectively. PGD 434 or DG 436 ( Figure 4B The fifth or sixth column (in the table) can be used to deplete the PD. The values ​​shown in TE 420 are in Table 6. As shown below, only those with T... E1 The first optical pulse is sampled by all three PGs. The TE 420 operates in a "1-pulse-3-tap" configuration. This means that three optical pulses are emitted for gating cycles with three different taps (three different PGs activated). In other TEs, two or three pulses may be emitted for gating cycles with four different taps (four different PGs activated), referred to as "2-pulse-4-tap" or "3-pulse-4-tap" configurations, respectively. It should be noted that the same 4-tap camera system can be used to switch between performing GI according to the TE 420 and TE 400, as well as other TEs disclosed in this specification.

[0081]

[0082] Table 6 As shown above, the gating cycle time of TE 420 is defined similarly to that of TE 320. When i=1, 2, 3, the gating cycle time of the tap is determined by T. i = T E + T Di + T PGi Given. Where, T SEQ = T3.

[0083] Figure 4C An embodiment of another TE for a 3-gate camera system, numbered 450, is shown. TE 450 is identical to TE 400 except that the GI pauses (or “interrupts”) for a short period between each gating cycle. This is achieved by applying a “delay” as shown in the figure after each gating cycle ends and before the start of the next gating cycle. This delay can range from 2 ns to 20 ns. In some cases, this delay is beneficial because it prevents light emitted in a previous (or earlier) gating cycle from interfering with measurements in the current (or consecutive) gating cycle. TE 450 operates in a “1-pulse-1-tap” configuration and uses the parameter values ​​provided for TE 400 in Table 5. Such delays can also be used in other TEs disclosed herein.

[0084] Figure 5 An embodiment of a TE for a 1-gate camera system, numbered 500, is shown. "1-gate" refers to each TE... F One 3D slice is captured (per readout cycle). TE 500 shows the beginning portions of the first sequence 502 and the second sequence 504. All definitions and descriptions regarding TE 300, 320, and TE 400, 420, and 450 also apply to TE 500. The numerical values ​​shown in TE 500 are shown in Table 7. Figure 5 In the PGA and PGB T PGThese are shown in the second and third columns, labeled 508 and 510, respectively. Note that in the TE 500, the PGB is not used for deploying gating cycles (i.e., capturing 3D slices), but rather for measuring the "ambient" illumination of the scene being imaged by the camera. "Ambient" refers not to light emitted by the emitters included in the 1-gate camera system and reflected back from the scene, but to light from other ("ambient") light sources, such as the sun, vehicle headlights, streetlights, etc. To measure ambient illumination, the corresponding charge is stored in one of the storage nodes of the N-tap pixel. Generally, ambient light is undesirable in GI because it reduces the signal-to-noise ratio (SNR) and contrast of the output image. In some GI applications, the ambient illumination of the scene is measured to correct the image data acquired through GI, which may help improve the SNR or enhance the accuracy of the 3D image data. In a 1-gate camera system, the DG 514 or PGC 512 is used to deplete the PD.

[0085] In terms of timing, PGB represents an additional gating cycle without the use of lighting. The gating cycle time for gate 1 is T1 = T E1 + T D1 + T PG1 Given. From T D The time to PGB activation is relatively long. In fact, it usually takes T... D The 3D range may be greater than that of a single-camera system, therefore only a relatively small amount of light emitted by the emitter is measured in environmental measurements. Note that, compared to the known GI pixel 130, the N-tap pixels disclosed herein can simultaneously (in the same T...) F (Internal) It captures a 3D slice, measures ambient light, and optionally measures parasitic charge. The TE 500 operates in a "1 pulse - 1 tap" configuration.

[0086]

[0087] Table 7 Please note that timing sequences such as GT and sequential timing are not always perfectly consistent. Furthermore, it should be noted that the timing of PG activations may be non-uniform (or "irregular"). In some examples, a uniform timing sequence may be used.

[0088] Figure 6An embodiment of a computer vision system 600 is schematically illustrated, configured to implement gated imaging (GI) as described herein. The computer vision system 600 may be included, for example, in a mobile device (e.g., a smartphone) or in a vehicle (e.g., for ADAS). The computer vision system 600 includes a GI camera system 610 comprising a lens 612, an N-tap GI image sensor 614, and a transmitter 616. The N-tap GI image sensor 614 may include a plurality (M) of imaging pixels. M represents the pixel resolution of the image sensor 614, ranging from 0.05 megapixels (MP) to 10 MP, 0.1 MP to 4 MP, or 0.1 MP to 2 MP, such as 0.3 MP, 0.6 MP, or 1.2 MP. Each of the M imaging pixels may include N storage nodes. N may be in the range of 2-6 or 3-8, such as N=4 or N=8. Optionally, the computer vision system 600 may also include a second camera 620. The GI camera system 610 and the second camera 620 may operate, for example, in the visible light wavelength range, the near-infrared (NIR) wavelength range, the short-wave infrared (SWIR) wavelength range, or the long-wave infrared (LWIR) wavelength range. The computer vision system 600 also includes a processor 630 configured to perform the methods described herein. The processor 630 includes (optionally) a scene analyzer 632 (e.g., configured to analyze image data of a scene), a transmitter controller 634 configured to control the transmitter 616, and a GI sensor controller 636 configured to configure the image sensor 614. Furthermore, the processor 630 includes a depth calculator 638, for example for calculating (or “estimating”) a depth map from two or more 3D slices; and an image fusion unit 640, for example for merging two or more 3D slices into a 2D or 3D output image. The computer vision system 600 may also include an application programming interface 650, for example, for providing 2D and / or 3D data to a mobile application or ADAS, and a memory 660, for example, for storing parameters of the GI, or for storing calibration data of the GI camera 610, or for storing calibration data between the GI camera 610 and a second camera 620. The computer vision system 600 may also include several additional sensors to acquire additional information. For example, additional sensors may be radar sensor systems, light detection and ranging sensor systems (“LiDAR”), microphones or even directional microphones, positioning sensors (such as GPS), inertial measurement units (IMUs), etc.

[0089] In some embodiments, image sensor 614 may include components of processor 630, thereby operating as a system-on-chip (SoC). Such components may include scene analyzer 632, emitter controller 634, depth calculator 638, image fusionist 640, and on-chip sensor controller 636. Furthermore, it should be noted that image sensor 614 may be configured to perform method 700 disclosed below, namely adaptive gated imaging (GI).

[0090] Figure 7 The steps of one embodiment of the method for performing GI, numbered 70, are shown. This method 700 can be used in computer vision systems (e.g.,...) Figure 6 This method is executed in the computer vision system 600 shown. The method 700 is referred to as "adaptive gating imaging".

[0091] In step 702, the transmitter (e.g., transmitter 616) illuminates the scene.

[0092] In step 704, an image sensor (e.g., image sensor 614) included in the GI camera system measures the amount of light in the scene. This amount of light may include two distinct contributions: a first contribution representing the (back-reflected) light emitted by the emitter in step 702, and a second contribution representing ambient light. Generally, for GI, it is desirable to maximize the first contribution and minimize the second contribution. In some examples, the second contribution may be used to correct for the effect of ambient light on calculations based on the first contribution data.

[0093] In step 706, a (optional) second camera or similar camera can be used to capture scene images. Typically, the second camera does not perform GI (Geometric Interaction) but instead acquires "regular" images, such as RGB images. Note that regular images typically primarily capture ambient light in the scene. Step 706 is optional.

[0094] In step 708, the scene is analyzed by a scene analyzer (e.g., scene analyzer 632), and the optimized parameters of the GI are calculated. The parameters of the GI may include the emitter's emission intensity (or "optical power"), the number of repetitions for a specific period, and T. IM The pixel-binding mode of the pixel-binding sensor, or any other technique discussed in this paper, was used. As a result, optimized parameters for the transmitter and image sensor were obtained.

[0095] In some examples, optimization parameters may depend on the visibility of the scene. Visibility can be affected by factors such as specific time, weather conditions, and the presence of fog. In some examples, optimization parameters may depend on the region of interest (ROI), a specific 3D slice, or the depth of interest (DOI) within the scene. For example, a computer vision system (such as 600) might want to image a specific ROI with a particularly high signal-to-noise ratio (SNR), contrast, or desired brightness, and adjust the parameters accordingly. In other examples, a computer vision system (such as 600) might want to image a specific DOI with a particularly high signal-to-noise ratio (SNR), contrast, or desired brightness, and adjust the parameters accordingly. Adjustment (or "modification") of parameters may, for example, be used to compensate for the relatively low reflectivity of objects within the ROI or DOI. In some examples, it may only be necessary to image the DOI. In such cases, parameters can be adjusted to image the 3D slice containing the DOI in an advantageous manner, such as in terms of signal-to-noise ratio.

[0096] In step 710, the transmitter parameters are adjusted based on the optimized parameters obtained in step 708.

[0097] In step 712, the parameters of the image sensor are adjusted according to the optimized parameters obtained in step 708.

[0098] In step 714, the input image (II) is acquired using the optimized parameters. In some examples, such an input image may have a particularly high signal-to-noise ratio (SNR). In other examples, it may have a particularly high SNR within a specific region of interest (ROI) or a specific depth region (DOI).

[0099] While this disclosure describes only a limited number of embodiments, it should be understood that many variations, modifications, and other applications can be made to these embodiments. Generally, this disclosure should not be construed as limited to the specific embodiments described herein, but only to the scope of the appended claims.

[0100] All references mentioned in this specification are incorporated herein by way of incorporation in their entirety, and their effect is the same as if each reference were expressly and individually identified and included in this specification. Furthermore, any reference or designation in this application should not be construed as an admission that such reference is prior art.

Claims

1. A method comprising: A light emitter and an image sensor synchronized with the light emitter are provided. The image sensor includes a first plurality of M imaging pixels and a second plurality of N storage nodes, each imaging pixel including a photodiode (PD). The scene is illuminated by emitting light pulses through the light emitter; At the image sensor, light reflected back from the scene after a specific time delay from the emission of the light pulse is received, the specific time delay defining a specific scene depth range; The first charge corresponding to the depth range of the first specific scene is stored in the first storage node; The second charge corresponding to the depth range of the second specific scene is stored in the second storage node; as well as The first charge and the second charge are used to form an image of the scene.

2. The method according to claim 1, further comprising: The third charge corresponding to the depth range of the third specific scene is stored in the third storage node; And the image is formed using the first charge, the second charge, and the third charge.

3. The method according to claim 2, further comprising: The fourth charge corresponding to the depth range of the fourth specific scene is stored in the fourth storage node; And the image is formed using the first charge, the second charge, the third charge, and the fourth charge.

4. The method according to claim 1, further comprising: The PD is depleted by generating a third charge, which is then stored in a third storage node.

5. The method according to claim 1, further comprising: Ambient lighting in the scene is measured by storing a third charge in a third storage node; And the image is formed using the first charge, the second charge, and the third charge.

6. The method according to claim 1, characterized in that, N = 3-8。 7. The method according to claim 1, characterized in that, N = 4。 8. The method according to claim 1, characterized in that, The value of M ranges from 0.1 megapixels (MP) to 4 MP.

9. The method according to claim 1, characterized in that, The pixel size of the imaging pixel is in the range of 1µm to 10µm.

10. The method according to claim 1, characterized in that, The imaging pixel is operable to perform pixel merging.

11. The method according to claim 1, characterized in that, The light emitter and the image sensor are used to perform adaptive gating imaging.

12. The method according to claim 1, characterized in that, The image can be either 3D or 2D.

13. The method according to any one of claims 1-12, characterized in that, This method is executed on mobile devices.

14. The method according to claim 13, characterized in that, The mobile device is a smartphone.

15. The method according to any one of claims 1-12, characterized in that, This method is executed in the vehicle.

16. A camera system comprising: A light emitter used to illuminate the scene; as well as An image sensor synchronized with the light emitter includes a first plurality of M imaging pixels and a second plurality of N storage nodes, each imaging pixel including a photodiode (PD). The image sensor is used to receive light reflected back from the scene after a specific time delay from the emission of the light pulse, where the specific time delay defines a specific scene depth range. The first storage node is used to store the first charge corresponding to the depth range of the first specific scene. The second storage node is used to store the second charge corresponding to the depth range of the second specific scene. The first charge and the second charge are used to form an image of the scene. Furthermore, the camera system in question is a gated imaging camera system.

17. The camera system according to claim 16, characterized in that, The third storage node is used to store a third charge corresponding to a third specific scene depth range, and the first charge, the second charge, and the third charge are used to form the image.

18. The camera system according to claim 17, characterized in that, The fourth storage node is used to store a fourth charge corresponding to a fourth specific scene depth range, wherein the first charge, the second charge, the third charge, and the fourth charge are used to form the image.

19. The camera system according to claim 16, characterized in that, The third storage node is used to store the fourth charge generated due to the depletion of PD.

20. The camera system according to claim 16, characterized in that, The third storage node is used to store a third charge to measure ambient light in the scene, and the first charge, the second charge, and the third charge are used to form the image.

21. The camera system according to claim 16, characterized in that, N = 3-8。 22. The camera system according to claim 16, characterized in that, N = 4。 23. The camera system according to claim 16, characterized in that, The value of M ranges from 0.1 megapixels (MP) to 4 MP.

24. The camera system according to claim 16, characterized in that, The pixel size of the imaging pixel is in the range of 1µm to 10µm.

25. The camera system according to claim 16, characterized in that, The imaging pixel is operable to perform pixel merging.

26. The camera system according to claim 16, characterized in that, The light emitter and the image sensor are used to perform adaptive gating imaging.

27. The camera system according to claim 16, characterized in that, The image can be either 3D or 2D.

28. The camera system according to any one of claims 16-27, characterized in that, The camera system is included in the mobile device.

29. The camera system according to claim 28, characterized in that, The mobile device is a smartphone.

30. The camera system according to any one of claims 16-27, characterized in that, The camera system is included in the vehicle.

31. A method comprising: A light emitter and an image sensor synchronized with the light emitter are provided. The image sensor includes a first plurality of M imaging pixels and a second plurality of N storage nodes, each imaging pixel including a photodiode (PD). The scene is illuminated by emitting light pulses through the light emitter; At the image sensor, light reflected back from the scene after a specific time delay from the emission of the light pulse is received, the specific time delay defining a specific scene depth range; The first charge corresponding to the depth range of the first specific scene is stored in the first storage node; The PD is depleted by generating a second charge, and the second charge is stored in a second memory; as well as The first charge is used to form an image of the scene.

32. The method of claim 31, further comprising: The third and fourth charges, corresponding to the depth ranges of the second and third specific scenes, are stored in the third and fourth storage nodes, respectively. And the image is formed using the first charge, the third charge, and the fourth charge.

33. The method of claim 31, further comprising: Ambient lighting in the scene is measured by storing a third charge in a third storage node; And the first charge and the third charge are used to form the image.

34. The method according to claim 31, characterized in that, N = 3-8。 35. The method according to claim 31, characterized in that, N = 4。 36. The method according to claim 31, characterized in that, The value of M ranges from 0.1 megapixels (MP) to 4 MP.

37. The method according to claim 31, characterized in that, The pixel size of the imaging pixel is in the range of 1 µm to 10 µm.

38. The method according to claim 31, characterized in that, The imaging pixel is operable to perform pixel merging.

39. The method according to claim 31, characterized in that, The light emitter and the image sensor are used to perform adaptive gating imaging.

40. The method according to claim 31, characterized in that, The image can be either 3D or 2D.

41. The method according to any one of claims 31-40, characterized in that, This method is executed on mobile devices.

42. The method according to claim 41, characterized in that, The mobile device is a smartphone.

43. The method according to any one of claims 31-40, characterized in that, This method is executed in the vehicle.

44. A camera system comprising: A light emitter used to illuminate the scene; as well as An image sensor synchronized with the light emitter includes a first plurality of M gated imaging pixels and a second plurality of N storage nodes, each gated imaging pixel including a photodiode (PD). The image sensor is used to receive light reflected back from the scene after a specific time delay from the emission of the light pulse, where the specific time delay defines a specific scene depth range. The first storage node is used to store the first charge corresponding to the first specific scene depth range. The second storage node is used to store the second charge generated by the depletion of the PD. The first charge is used to form an image. Furthermore, the camera system is a gated imaging camera system.

45. The method according to claim 44, characterized in that, The third and fourth storage nodes are used to store the third and fourth charges corresponding to the third and fourth specific scene depth ranges, respectively, and the first, third, and fourth charges are used to form the image.

46. ​​The method according to claim 44, characterized in that, The third storage node is used to store a third charge to measure ambient light in the scene, and the first charge and the third charge are used to form the image.

47. The camera system according to claim 44, characterized in that, N = 3-8。 48. The camera system according to claim 44, characterized in that, N = 4。 49. The camera system according to claim 44, characterized in that, The value of M ranges from 0.1 megapixels (MP) to 4 MP.

50. The camera system according to claim 44, characterized in that, The pixel size of the imaging pixel is in the range of 1 µm to 10 µm.

51. The camera system according to claim 44, characterized in that, The imaging pixel is operable to perform pixel merging.

52. The camera system according to claim 44, characterized in that, The light emitter and the image sensor are used to perform adaptive gating imaging.

53. The camera system according to claim 44, characterized in that, The image can be either 3D or 2D.

54. The camera system according to any one of claims 44-53, characterized in that, The camera system is included in the mobile device.

55. The camera system according to claim 54, characterized in that, The mobile device is a smartphone.

56. The camera system according to any one of claims 44-53, characterized in that, The camera system is included in the vehicle.

57. A method comprising: A light emitter and an image sensor synchronized with the light emitter are provided. The image sensor includes a first plurality of M imaging pixels and a second plurality of N storage nodes, each imaging pixel including a photodiode (PD). The scene is illuminated by emitting light pulses through the light emitter; At the image sensor, light reflected back from the scene after a specific time delay from the emission of the light pulse is received, the specific time delay defining a specific scene depth range; The first charge corresponding to the depth range of the first specific scene is stored in the first storage node; The ambient light in the scene is measured by storing a second charge in a second storage node; and The first charge and the second charge are used to form an image of the scene.

58. The method of claim 57, further comprising: The third and fourth charges, corresponding to the depth ranges of the second and third specific scenes, are stored in the third and fourth storage nodes, respectively. And the image is formed using the first charge, the second charge, the third charge, and the fourth charge.

59. The method according to claim 57, characterized in that, N = 3-8。 60. The method according to claim 57, characterized in that, N = 4。 61. The method according to claim 57, characterized in that, The value of M ranges from 0.1 megapixels (MP) to 4 MP.

62. The method according to claim 57, characterized in that, The pixel size of the imaging pixel is in the range of 1 µm to 10 µm.

63. The method according to claim 57, characterized in that, The imaging pixel is operable to perform pixel merging.

64. The method according to claim 57, characterized in that, The light emitter and the image sensor are used to perform adaptive gating imaging.

65. The method according to claim 57, characterized in that, The image can be either 3D or 2D.

66. The method according to any one of claims 57-65, characterized in that, This method is executed on mobile devices.

67. The method according to claim 66, characterized in that, The mobile device is a smartphone.

68. The method according to any one of claims 57-65, characterized in that, This method is executed in the vehicle.

69. A camera system, characterized in that, include: A light emitter used to illuminate the scene; An image sensor synchronized with the light emitter includes a first plurality of M gated imaging pixels and a second plurality of N storage nodes, each gated imaging pixel including a photodiode (PD). The image sensor is used to receive light reflected back from the scene after a specific time delay from the emission of the light pulse, the specific time delay defining a specific scene depth range; The first storage node is used to store the first charge corresponding to the first specific scene depth range. The second storage node is used to store the second charge generated by measuring the ambient light in the scene. The first charge and the second charge are used to form an image of the scene. Furthermore, the camera system is a gated imaging camera system.

70. The camera system according to claim 69, characterized in that, The third and fourth storage nodes are used to store the third and fourth charges corresponding to the third and fourth specific scene depth ranges, respectively, and wherein the first charge, the second charge, the third charge, and the fourth charge are used to form the image.

71. The camera system according to claim 69, characterized in that, N = 3-8。 72. The camera system according to claim 69, characterized in that, N = 4。 73. The camera system according to claim 69, characterized in that, The value of M ranges from 0.1 megapixels (MP) to 4 MP.

74. The camera system according to claim 69, characterized in that, The pixel size of the imaging pixel is in the range of 1 µm to 10 µm.

75. The camera system according to claim 69, characterized in that, The imaging pixel is operable to perform pixel merging.

76. The camera system according to claim 69, characterized in that, The light emitter and the image sensor are used to perform adaptive gating imaging.

77. The camera system according to claim 69, characterized in that, The image can be either 3D or 2D.

78. The camera system according to any one of claims 69-77, characterized in that, The camera system is included in the mobile device.

79. The camera system according to claim 78, characterized in that, The mobile device is a smartphone.

80. The camera system according to any one of claims 69-77, characterized in that, The camera system is included in the vehicle.

Images