3D sensing system and method of providing images based on hybrid sensing array

Abstract

A 3D sensing system and a method for providing images based on a hybrid sensing array are provided. The 3D sensing system includes: a light source configured to emit light; a hybrid sensing array including a 2D sensing region and a 3D depth sensing region, the 2D sensing region being configured to detect ambient light reflected from an object, and the 3D depth sensing region being configured to detect light emitted by the light source and reflected from the object; a superlens on the hybrid sensing array, the superlens being configured to guide ambient light reflected from the object to the 2D sensing region and guide light emitted by the light source and reflected from the object to the 3D depth sensing region; and processing circuitry configured to combine 2D image information provided by the 2D sensing region and 3D information provided by the 3D depth sensing region to generate a combined 3D image.

Description

[0001] This application claims priority and benefit to U.S. Provisional Application No. 63 / 012,000, filed April 17, 2020, entitled “A hybrid photodiode and SPAD array and sensor system to improve 3D depth information,” and U.S. Application No. 16 / 940,308, filed July 27, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0002] One or more embodiments of this disclosure relate to hybrid sensor systems and methods for 3D imaging. Background Technology

[0003] In recent years, there has been an increasing need for applications that rely on computer vision, such as Advanced Driver Assistance Systems (“ADAS”), autonomous driving applications, augmented reality (“AR”), and virtual reality (“VR”). Various detection systems (such as light detection and ranging (“LiDAR”) and cameras) have been developed to support ADAS, autonomous driving applications, AR, and VR. Such detection systems provide information to help applications understand, modify, and act upon information from the real world.

[0004] However, hardware and software limitations that can adversely affect resolution, detection range, speed, and / or stability may make it difficult to meet the needs of ADAS, autonomous driving applications, AR, and VR. For example, providing high-resolution and high-quality 3D depth information for ADAS and autonomous driving applications can be challenging. As another example, a low form factor can be desirable for AR and VR applications, where a longer detection range (e.g., exceeding 10 meters) and mobility of the device can be useful.

[0005] Typically, multiple separate detection systems are used to provide real-world information. However, converting data from these separate detection systems into high-quality 3D images can be challenging. For example, a separate detection system providing separate 2D image datasets and separate 3D depth datasets can be used to provide 3D images by aligning corresponding points from the separate 2D image datasets and separate 3D depth datasets. This process may involve several factors and / or calibration that could consume significant computational resources. Furthermore, misalignment or the inability to align corresponding points between 2D image data and 3D depth data can undermine the effectiveness of the separate systems used for computer vision applications.

[0006] The information disclosed in this background section is intended to enhance the understanding of the background of this disclosure and may therefore contain information that does not constitute prior art. Summary of the Invention

[0007] One or more example embodiments of this disclosure relate to systems and methods for 3D imaging.

[0008] According to one embodiment of this disclosure, a 3D sensing system is provided. The 3D sensing system includes: a light source configured to emit light; a hybrid sensing array including: a 2D sensing region and a 3D depth sensing region, the 2D sensing region being configured to detect ambient light reflected from an object, and the 3D depth sensing region being configured to detect light emitted by the light source and reflected from the object; a superlens on the hybrid sensing array, the superlens being configured to: guide ambient light reflected from the object to the 2D sensing region and guide light emitted by the light source and reflected from the object to the 3D depth sensing region; and processing circuitry configured to: combine 2D image information provided by the 2D sensing region and 3D information provided by the 3D depth sensing region to generate a combined 3D image.

[0009] The processing circuit is also configured to: determine 2D image information of the object based on the 2D sensing area; determine 3D depth information of the object based on the 3D depth sensing area; and update the 2D image information of the object with the 3D depth information.

[0010] The 3D sensing system may further include: row decoders and column amplifiers, which are stacked below the hybrid sensing array.

[0011] Superlenses can be circular, cylindrical, rectangular, or square in shape, and can be polarization-independent.

[0012] The 2D sensing area may include one or more photodiodes, and the 3D depth sensing area may include one or more APDs or SPADs.

[0013] The superlens can be configured to guide NIR light into the 3D depth sensing region.

[0014] The processing circuit can also be configured to: detect objects based on 2D image information; generate regions of interest based on the coordinates of the objects in the 2D image information; and illuminate the regions of interest using a light source.

[0015] The processing circuit can also be configured to: determine the identity of an object based on 2D image information and 3D depth information; track the object based on 2D image information; and update the 2D image information with 3D depth information in response to critical conditions.

[0016] According to one embodiment of this disclosure, a method for providing an image based on a hybrid sensing array is provided. The method includes: emitting light from a light source; detecting ambient light reflected from an object through a 2D sensing region of the hybrid sensing array; detecting light emitted from the light source and reflected from the object toward the 3D depth sensing region through a 3D depth sensing region of the hybrid sensing array; guiding the ambient light to the 2D sensing region through a superlens on the hybrid sensing array; guiding the light emitted from the light source and reflected from the object to the 3D depth sensing region through the superlens of the hybrid sensing array; and combining 2D image information provided through the 2D sensing region and 3D information provided through the 3D depth sensing region through a processing circuit to generate a combined 3D image.

[0017] The method may further include: determining 2D image information of an object based on a 2D sensing area using a processing circuit; determining 3D depth information of an object based on a 3D depth sensing area using a processing circuit; and updating the 2D image information of the object with the 3D depth information using a processing circuit.

[0018] The hybrid sensing array that detects ambient light and light emitted by a light source can be stacked on top of a row decoder and a column amplifier.

[0019] The superlens that guides ambient light to the 2D sensing area and guides light emitted by the light source and reflected from the object to the 3D depth sensing area can be circular, cylindrical, rectangular, or square in shape, and can be polarization-independent.

[0020] The 2D sensing area may include one or more photodiodes for detecting ambient light, and the 3D depth sensing area may include an APD or SPAD for detecting light emitted by a light source and reflected from an object.

[0021] The step of guiding ambient light to a 2D sensing region via a superlens on the hybrid sensing array may include: guiding visible light to the 2D sensing region, and the step of guiding light emitted by a light source and reflected from an object to a 3D depth sensing region via a superlens on the hybrid sensing array may include: guiding NIR light to the 3D depth sensing region.

[0022] The method may further include: detecting an object based on 2D image information using a processing circuit; generating a region of interest based on the coordinates of the object in the 2D image information using a processing circuit; and illuminating the region of interest with a light source using a processing circuit.

[0023] The method may further include: determining the identity of an object based on 2D image information and 3D depth information by a processing circuit; tracking the object based on 2D image information by a processing circuit; and updating the 2D image information with 3D depth information in response to a critical condition by a processing circuit.

[0024] According to one embodiment of this disclosure, a 3D sensing system is provided. The 3D sensing system includes: a light source configured to emit light; a 2D sensing region for detecting visible light reflected from an object; a 3D depth sensing region for detecting near-infrared light reflected from the object; a superlens covering the 2D sensing region, the superlens configured to: guide visible light reflected from the object to the 2D sensing region, and guide light emitted by the light source and reflected from the object to the 3D depth sensing region; and processing circuitry configured to: determine 2D image information of the object based on the 2D sensing region; determine 3D depth information of the object based on the 3D depth sensing region; and update the 2D image information of the object with the 3D depth information.

[0025] The 3D sensing system may further include: a row decoder and a column amplifier, which are stacked below the 2D sensing area and the 3D depth sensing area.

[0026] Superlenses can be circular, cylindrical, rectangular, or square in shape, and can be polarization-independent.

[0027] The 2D sensing area may include one or more photodiodes, and the 3D depth sensing area may include one or more SPADs or APDs. Attached Figure Description

[0028] The above and other aspects and features of this disclosure will become more apparent to those skilled in the art from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

[0029] Figure 1 This is a block diagram of a 3D sensing system according to one or more embodiments of the present disclosure.

[0030] Figure 2A This is a block diagram including a hybrid sensor according to one or more embodiments of the present disclosure.

[0031] Figure 2B This is a block diagram including a hybrid sensor according to one or more embodiments of the present disclosure.

[0032] Figures 3A to 3I This is a plan view of a portion of a hybrid sensing array according to one or more embodiments of the present disclosure.

[0033] Figures 4A to 4D This is a plan view of a portion of a hybrid sensing array having multiple metalens on a hybrid sensing array, according to one or more embodiments of the present disclosure.

[0034] Figures 5A to 5CThis is a cross-sectional view of a hybrid sensing array having a superlens and a microlens on a hybrid sensing array according to one or more embodiments of the present disclosure.

[0035] Figure 6A This is a block diagram of an exploded view of a superlens and a superlens on a hybrid sensing array, according to one or more embodiments of the present disclosure.

[0036] Figure 6B This is an exploded block diagram of a lens and a superlens on a hybrid sensing array according to one or more embodiments of the present disclosure.

[0037] Figure 7A These are plan views and cross-sectional views of a hybrid sensor according to one or more embodiments of the present disclosure.

[0038] Figure 7B This is a cross-sectional view of a hybrid sensor according to one or more embodiments of the present disclosure.

[0039] Figure 8 This is a flowchart of a method for object detection and tracking according to one or more embodiments of the present disclosure.

[0040] Figure 9 This is a flowchart of a method for providing high-resolution 2D images and high-resolution 3D images according to one or more embodiments of the present disclosure. Detailed Implementation

[0041] In the following description, exemplary embodiments will be illustrated with reference to the accompanying drawings, wherein the same reference numerals throughout denote the same elements. However, this disclosure may be implemented in various different forms and should not be construed as being limited to the embodiments shown herein. Rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the aspects and features of this disclosure to those skilled in the art. Therefore, processes, elements, and techniques that are not essential for a person of ordinary skill in the art to fully understand the aspects and features of this disclosure are not described. Unless otherwise stated, the same reference numerals denote the same elements throughout the drawings and textual description, and therefore their descriptions are not repeated.

[0042] Typically, LiDAR systems (e.g., direct time-of-flight (TOF) LiDAR systems) measure the distance (e.g., depth) between an object and its sensor by emitting a light pulse (e.g., a laser pulse) toward the object and measuring the time it takes for the light pulse to reflect from the object and be detected by the LiDAR system's sensor. Some LiDAR systems may use 3D depth sensors, including photodetectors (e.g., single-photon avalanche diodes (SPADs) and / or avalanche photodiodes (APDs)), to detect photons for 3D depth information. Some LiDAR systems may also use separate 2D image sensors, including photodetectors (e.g., photodiodes (PDs)), to detect light at specific wavelengths (e.g., visible light wavelengths) for 2D information (e.g., 2D images), and 3D depth information can be fused with 2D information by aligning corresponding points between different views from the 3D depth sensor and the 2D sensor. By fusing 3D depth information with 2D information, computer vision applications (e.g., object detection for autonomous driving applications and / or ADAS) can detect and identify objects in the real world. Therefore, the speed and resolution of providing fused 2D images and 3D depth data can be critical for real-time computer vision applications.

[0043] However, using separate 2D image sensors and 3D depth sensors presents several challenges. For example, 2D image and 3D depth sensors may have different fields of view (FOV), different resolutions, and / or large shape factors. Therefore, the sensors do not need to be bulky, accurate calibration settings may be required, and a large amount of computing power can be dedicated to aligning the corresponding points between 3D depth information and 2D image data.

[0044] Depending on the FOV of a separate 2D image sensor and a separate 3D depth sensor, some points may not correspond due to areas where the FOVs do not overlap and / or obstacles on one of the sensors. Furthermore, separate 2D image sensors and separate 3D depth sensors can have a large baseline distance (i.e., the distance between the 2D image sensor and the 3D depth sensor), which can make alignment between 2D and 3D information more difficult.

[0045] Individually, combining data from separate 2D image sensors and separate 3D depth sensors can lead to problems that may need to be addressed during alignment. For example, a separate 3D depth sensor may have noise (such as temporal, spatial, or flying pixels), which can make fusing 2D image information and 3D depth information more difficult.

[0046] Furthermore, separate 2D image sensors and 3D depth sensors scan all available sensing pixels to provide data for alignment. However, this processing can consume significant computational resources and power, and scanning all available sensing pixels may not be desirable for computer vision applications. In the case of autonomous vehicles, the increased power consumption can reduce available power, which can reduce driving range based on the set battery charge level. Additionally, the increased computational demands associated with scanning more sensing pixels can reduce the speed at which computations (such as Time-of-Flight) can be performed.

[0047] Regardless, high-resolution and high-quality 3D depth information with a low shape factor is highly desirable for computer vision applications used to detect objects at greater distances in an environment and provide enhanced mobility. In the case of ADAS and autonomous driving applications, increasing the distance at which objects can be detected reduces the risk posed by those objects. Therefore, it is expected to detect objects as large as at least 100 meters to about 200 meters or more. In the case of VR and AR applications, a low shape factor provides greater mobility, and high-quality 3D depth information is expected to identify distant objects (e.g., objects more than 10 meters away from the sensing system).

[0048] According to one or more embodiments of this disclosure, a compact, monolithic hybrid sensor including a hybrid sensing array can be provided. The hybrid sensing array may include a 2D sensing region and a 3D depth sensing region, the 2D sensing region (e.g., a complementary metal-oxide-semiconductor (“CMOS”) sensing region) for detecting visible light features, and the 3D depth sensing region (e.g., a SPAD sensing region or an APD sensing region) for sensing depth. The 2D sensing region and the 3D depth sensing region of the hybrid sensing array may share the same or substantially the same field of view (FOV). By sharing the FOV, the system's calibration requirements can be reduced, and alignment can be simplified.

[0049] In one or more embodiments, one or more metalens located between the hybrid sensing array and external light can guide (or focus) visible light to the 2D sensing region and guide (or focus) near-infrared (“NIR”) light to the 3D depth sensing region to enhance visible light and NIR detection.

[0050] Furthermore, a first plurality of sensing pixels of the PD including a 2D sensing region and a second plurality of sensing pixels including a 3D depth sensing region can be arranged such that a high-resolution 2D image updated or enhanced with high-quality 3D depth information according to a shared FOV can be provided. In other words, a combined 3D image can be provided. For example, in one or more embodiments, 2D image data from the 2D sensing region can be used by processing circuitry to detect objects in a high-resolution 2D view, thereby allowing object detection at greater distances. The processing circuitry can update or enhance the high-resolution 2D image data with 3D depth information by mapping 3D depth information from the 3D depth sensing region onto the 2D image data. For example, the processing circuitry can combine 2D image information provided by the 2D sensing region and 3D depth information provided by the 3D depth sensing region to generate a combined 3D image. For example, the processing circuitry can determine 2D image information of an object based on the 2D sensing region, determine 3D depth information of the object based on the 3D depth sensing region, and update the 2D image information of the object with the 3D depth information.

[0051] In one or more embodiments of this disclosure, the processing circuitry can save power and reduce computational burden by applying a region of interest (ROI) scanning mechanism to the 3D depth sensing region of a hybrid sensor. For example, the object detector of the processing circuitry can use algorithms to identify and track objects of interest (OOIs) based on 2D images. In one or more embodiments, based on the identified and tracked OOI, the ROI controller of the processing circuitry can communicate with the sensing circuitry such that a portion of the sensing pixels of the 3D depth sensing region (e.g., a portion of the SPAD or APD of the 3D depth sensing region) rather than all the sensing pixels of the 3D depth sensing region provides 3D depth information to the processing circuitry. Furthermore, in one or more embodiments, the ROI controller of the processing circuitry can communicate directly or indirectly with a light source to activate a portion of a vertical-cavity surface-emitting laser (VCSEL) array to illuminate the tracked OOI. For example, in ADAS and autonomous vehicle applications, pedestrians can be moving compared to trees, which can be static. Therefore, the 3D depth information of pedestrians can be provided using ROI scanning, while the 3D depth information of trees may not be collected to save power and / or computing resources.

[0052] The above and other aspects and features of one or more exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings.

[0053] Figure 1This is a block diagram 100 of a 3D sensing system 102 according to one or more embodiments of the present disclosure.

[0054] Reference Figure 1 According to one or more example embodiments of this disclosure, the 3D sensing system 102 can acquire information corresponding to its surroundings in the environment. In one or more embodiments, the 3D sensing system includes a light source 104 (e.g., an array of VCSELs) and a hybrid sensor 106.

[0055] The hybrid sensor 106 may include a 2D sensing region and a 3D depth sensing region. The 2D sensing region is configured to generate 2D information (e.g., 2D image information with or without color) based on ambient light, and the 3D depth sensing region is configured to generate 3D depth information based on light emitted from the light source 104 according to TOF measurements.

[0056] For example, the 2D sensing area of ​​the hybrid sensor 106 can detect ambient light reflected from one or more targets to generate 2D image information, and the 3D depth sensing area of ​​the hybrid sensor 106 can detect light emitted from the light source 104 (e.g., emitted light pulses) and reflected back towards the hybrid sensor 106 (the 3D depth sensing area may include one or more superlenses to guide the reflected light into the 3D depth sensing area). Based on the time elapsed from the emission of the light pulse from the light source 104 to the detection of the reflected light pulse through the 3D depth sensing area (i.e., TOF), the 3D sensing system 102 can determine the distance D (e.g., depth) to the surface of one or more objects 108.

[0057] In one or more embodiments, 2D information can be used independently to detect and track one or more objects 108 within a suitable proximity of the 3D sensing system 102 at high resolution. In one or more embodiments, 3D depth information can be used to update the 2D information of one or more tracked objects 108 to provide a high-resolution 2D image supplemented or enhanced with 3D depth information.

[0058] Therefore, the 3D sensing system 102 can use the hybrid sensor 106 to identify and track one or more objects 108 in the environment in 2D at high resolution, and can use the hybrid sensor 106 to enhance or update the 2D information corresponding to one or more objects with 3D depth information. By detecting one or more objects 108 in 2D at high resolution, the 3D sensing system 102 can increase the object detection distance and improve the depth estimation for the detected objects.

[0059] Figure 2A This is a block diagram including a hybrid sensor 106 according to one or more embodiments of the present disclosure. Figure 2B This is a block diagram including a hybrid sensor 106 according to one or more embodiments of the present disclosure.

[0060] Reference Figures 2A to 2B According to one or more example embodiments of this disclosure, the hybrid sensor 106 includes a sensing circuit 202 for detecting light, and a processing circuit 204 including a processor 206 and a memory 208. The processor 206 may be implemented as a general-purpose processor, an application-specific integrated circuit (ASIC), one or more field-programmable gate arrays (FPGAs), a set of processing components, or other suitable electronic processing components. The memory 208 (e.g., memory, memory cells, and / or storage devices, etc.) may include one or more means (e.g., RAM, ROM, flash memory, and / or hard disk storage, etc.) for storing data and / or computer code used to perform or facilitate the various processes described in this application. The memory 208 may be volatile memory or non-volatile memory, or may include volatile memory or non-volatile memory. The memory 208 may include a database component, an object code component, a script component, or any other type of information structure for supporting the various activities and information structures described in this application. According to an example embodiment, memory 208 may be communicatively connected to processor 206 via processing circuitry 204 and includes computer code for (e.g., via processing circuitry 204 and / or processor 206) performing one or more of the processes described herein.

[0061] like Figures 2A to 2B As shown, processing circuitry 204 may be implemented within hybrid sensor 106 as an internal processing circuitry 204 of hybrid sensor 106. However, this disclosure is not limited thereto. In one or more embodiments, the functionality of processing circuitry 204 may be separated or shared across multiple local processing circuits (e.g., processing circuitry 204 of hybrid sensor 106 and another processing circuitry of 3D sensing system 102 that may be separate from hybrid sensor 106). In one or more embodiments, processing circuitry of 3D sensing system 102 may, for example, control the functionality of 3D sensing system 102 (e.g., control the functionality of a motor vehicle) and communicate with processing circuitry 204 of hybrid sensor 106. Therefore, in one or more embodiments, processing circuitry 204 of hybrid sensor 106 may offload some functions that may be performed by processing circuitry of 3D sensing system 102, thereby freeing up processing circuitry of 3D sensing system 102 to perform other functions.

[0062] Although one or more local processing circuits are described as performing the functions described in this disclosure, this disclosure is not limited thereto. For example, in other embodiments, processing circuit 204 or one or more components of processing circuit 204 (e.g., components that execute instructions in memory 208 to perform the methods described in this disclosure) may be distributed across multiple servers or computers (e.g., may exist in distributed locations).

[0063] In one or more embodiments, the processing circuitry 204 may execute instructions in the memory 208 to function as an image signal processor 214, an object detector 212, and / or a ROI controller 210.

[0064] Image signal processor 214 can use data from sensing circuit 202 to provide or reconstruct 2D and / or 3D images. In one or more embodiments, image signal processor 214 provides or reconstructs 2D images by interpolating missing 2D image sensing data. Image signal processor 214 can provide or reconstruct 3D images by interpolating missing 3D depth sensing data and applying 3D depth information to the provided or reconstructed 2D images according to an algorithm, to provide high-resolution 2D images supplemented, supplemented, or updated with 3D depth information (i.e., 3D images).

[0065] In one or more embodiments, the image signal processor 214 may use 2D images to improve 3D quality. For example, the image signal processor 214 may use high-resolution 2D images to detect one or more Out-of-Interference Areas (OOIs) and may guide 3D depth sensing to regions of interest (ROIs) where the OOIs can be detected. In this case, the 2D images may be used to supplement and / or verify the depth information of the 3D images, such as filling in missing 3D information for OOIs consistent with the 2D information.

[0066] Object detector 212 may use algorithms including a neural network-based object detection system to detect and track objects in 2D image data based on data from a 2D sensing region. In one or more embodiments, object detector 212 may track objects in a 2D image and specifically update the region (e.g., ROI) of the tracked object based on ROI scans using 3D information (e.g., 3D depth information).

[0067] The ROI controller 210 can use the object detector 212 with tracking information to interact with the sensing circuit 202 and / or the light source 104 to achieve ROI-based 3D sensing. For example, the ROI controller 210 can send signals to the sensing circuit 202, causing the row decoder 218 and / or column scanner 220 to perform normal scanning or ROI-based scanning according to row and column control logic. In one or more embodiments, the ROI controller 210 can use tracking information from the object detector 212 to guide the light source to specifically illuminate one or more OOIs. For example, the ROI controller can send instructions to activate only the portion of the array of VCSELs of the light source directed toward one or more tracked OOIs. Therefore, battery power consumption can be reduced.

[0068] Therefore, the image signal processor 214, the object detector 212, and the ROI controller 210 achieve high-resolution, high-quality 3D images by performing object recognition and tracking through scanning of objects tracked by ROI.

[0069] Reference Figure 2A In one or more embodiments, the sensing circuit 202 of the hybrid sensor 106 includes a hybrid sensing array 216, which includes a 2D sensing region and a 3D depth sensing region. The 2D sensing region includes a first plurality of sensing pixels arranged in an array of lines forming columns and rows, and the 3D depth sensing region includes a second plurality of sensing pixels arranged in an array of lines forming columns and rows. Rows may extend in a first direction DR1, and columns may extend in a second direction DR2 intersecting the first direction DR1.

[0070] Each of the first plurality of sensing pixels may include a PD, and each of the second plurality of sensing pixels may include a SPAD or an APD. In one or more embodiments, each of the PDs in the 2D sensing region may detect red, green, or blue light, and each of the SPADs or APDs may detect near-infrared light (i.e., light having wavelengths between 0.8 micrometers and about 1 micrometer (e.g., 904 nanometers or 940 nanometers)). However, this disclosure is not limited thereto. For example, one or more of the PDs in the 2D sensing region may detect light of any suitable color in the visible to NIR spectrum (e.g., about 0.4 to about 0.8 micrometers), and one or more of the sensing pixels in the 3D depth sensing region may detect light of any suitable spectrum (e.g., light with wavelengths greater than 0.8 micrometers (e.g., 904 nanometers or 940 nanometers)) for measuring TOF based on light source 104. In the case of the 2D sensing region, any suitable arrangement may be used; for example, the 2D sensing region may include one or more of red sensing pixels, green sensing pixels, and blue sensing pixels, one or more of red sensing pixels and clear sensing pixels, one or more clear sensing pixels, or one or more monochromatic sensing pixels.

[0071] In one or more embodiments, as referred to below Figures 3A to 3H As shown and described in more detail, the sensing pixels of the 2D sensing region and the sensing pixels of the 3D depth sensing region can be interleaved (e.g., as shown in...). Figures 3A to 3D The lattice arrangement shown in the embodiments is interwoven, or in the form of... Figures 3E to 3HThe stripe arrangement shown in the embodiment is interwoven. In other words, each sensing pixel of the 3D depth sensing region can be directly adjacent to one or more sensing pixels of the 2D sensing region, such that sensing pixels of the 2D sensing region can be arranged in the first direction DR1 or the second direction DR2 between sensing pixels of the 3D depth sensing region, or sensing pixels of the 3D depth sensing region can be arranged in the first direction DR1 or the second direction DR2 between two sensing pixels of the 2D sensing region. However, this disclosure is not limited thereto. For example, as Figure 2B , Figure 3I , Figure 6B and Figures 7A to 7B As shown in the embodiments, the 2D sensing area and the 3D depth sensing area can be arranged side-by-side.

[0072] Reference Figure 2B In one or more embodiments, the sensing circuit 202 of the hybrid sensor 106 includes a hybrid sensing array comprising a 2D sensing region 230 and a 3D depth sensing region 232 arranged side-by-side. The 2D sensing region 230 includes a first plurality of sensing pixels arranged in an array of lines forming columns and rows, and the 3D depth sensing region 232 includes a second plurality of sensing pixels arranged in an array of lines forming columns and rows. Rows may extend in a first direction DR1, and columns may extend in a second direction DR2 intersecting the first direction DR1.

[0073] Each of the first plurality of sensing pixels may include a PD, and each of the second plurality of sensing pixels may include a SPAD or an APD. In one or more embodiments, each of the PDs in the 2D sensing region 230 may detect red, green, or blue light, and each of the SPADs or APDs may detect near-infrared light. However, this disclosure is not limited thereto. For example, one or more of the PDs in the 2D sensing region 230 may detect light of any suitable color in the visible to NIR spectrum (e.g., about 0.4 to about 0.8 micrometers), and one or more of the sensing pixels in the 3D depth sensing region 232 may detect light of any suitable spectrum (e.g., light with wavelengths greater than 0.8 micrometers, such as 904 nanometers or 940 nanometers) for measuring TOF based on light source 104. In the case of the 2D sensing region 230, any suitable arrangement may be used; for example, the 2D sensing region 230 may include one or more of red sensing pixels, green sensing pixels, and blue sensing pixels, one or more of red sensing pixels and transparent sensing pixels, one or more transparent sensing pixels, or one or more monochromatic sensing pixels.

[0074] In one or more embodiments, one or more superlenses may be located at Figures 2A to 2BIn the embodiments, the hybrid sensing array is positioned above or on top of one or more superlenses such that light incident on one or more superlenses can be guided to different sensing regions depending on the wavelength of the incident light. For example, light in the visible spectrum (e.g., red, green, and blue light) can be guided (or focused) to sensing pixels in a 2D sensing region through the nanostructure of one or more superlenses, while light in the NIR spectrum can be guided (or focused) to sensing pixels in a 3D depth sensing region. Therefore, one or more superlenses enhance the ability of the sensing pixels of the hybrid sensing array to detect inbound light.

[0075] Reference Figure 2A In one or more embodiments, row decoder 218 and column scanner 220 can be used to address the sensing pixels of hybrid sensor 106. Row decoder 218 and / or column scanner 220 can receive and output drive signals such that selected sensing pixels can detect incident light and output sensing signals in response to the detected light. Sensing signals from sensing pixels in a pixel row can be stored in column memory 228 before being sequentially read out to, for example, processing circuitry 204. In one or more embodiments, sensing circuitry 202 can provide signals to row decoder 218 and / or column scanner 220 based on signals from ROI controller 210 of processing circuitry 204 to perform normal scanning (e.g., ordinary scanning) or ROI-based 3D scanning.

[0076] In one or more embodiments, the sensing circuit 202 includes a column amplifier 224, a column analog-to-digital converter (“ADC”) 226, and a correlated double sampling (“CDS”) unit 222 for amplifying the signal. The column amplifier amplifies the signal from the sensed pixel to the CDS-enabled ADC to provide a readout signal to the processing circuit 204. In one or more embodiments, the circuitry and / or row and column control logic for providing the readout signal may be shared between the 2D sensing area and the 3D depth sensing area. However, this disclosure is not limited thereto. For example, as Figure 2B As shown, the readout signals and / or row and column control logic can be supplied via two separate circuits connected to the 2D sensing area and the 3D depth sensing area, respectively.

[0077] Reference Figure 2BThe 2D sensing area 230 can be connected to a first circuit, which includes a row decoder 234 and a column circuit system 236. The column circuit system 236 includes a column amplifier with CDS functionality, a column ADC with digital double sampling (“DDS”) functionality, and a line-memory block. The 3D depth sensing area 232 can be connected to a second circuit, which includes a row decoder 238 and a column circuit system 240. The column circuit system 240 includes a column amplifier (if needed), a time-to-digital converter (“TDC”), and a line-memory block. In one or more embodiments, the first and second circuits communicate with or include the phase-locked loop (“PLL”) of the sensing circuit 202, a counter that can function as a grayscale counter or a binary counter, a ramp generator, and digital logic (e.g., digital logic for reconstructing 2D and 3D images) to provide 2D image data and 3D depth information. Thus, as Figure 2B As shown, in one or more embodiments, the first circuit and the second circuit may be separate from each other, such that separate readouts and separate row and column control logic are provided.

[0078] Reference Figures 2A to 2B In one or more embodiments, the hybrid sensing array and other blocks (such as row decoders, CDS circuitry, column amplifiers, column ADCs, column memories, column scanners, row memories, TDCs, PLLs, counters, ramp generators, digital logic, processing circuitry 204 and / or components of processing circuitry 204, etc.) may be implemented in the same package (e.g., the same die) and / or may be part of a stacked structure (e.g., stacked dies).

[0079] In one or more embodiments, the hybrid sensing array and other blocks may be arranged side-by-side in the same package (e.g., the same die). In other words, the hybrid sensing array and other blocks may not overlap each other in a planar view and may be connected using conductive components (e.g., conductive traces, etc.).

[0080] In one or more embodiments, the hybrid sensing array and other blocks may be part of a stacked structure, with one or more of the other blocks located below the hybrid sensing array. The other blocks may be connected to the hybrid sensing array and / or connected to each other using conductive components such as hybrid bonding and / or through-silicon vias (TSVs, also known as through-silicon vias).

[0081] Figures 3A to 3I This is a plan view of a portion of a hybrid sensing array 300a, 300b, 300c, 300d, 300e, 300f, 300g, 300h, 300i according to one or more embodiments of the present disclosure.

[0082] Reference Figures 3A to 3D The 2D sensing region includes a plurality of sensing pixels (e.g., one or more green sensing pixels (G) 301 configured to detect green, one or more red sensing pixels (R) 302 configured to detect red, and one or more blue sensing pixels (B) 303 configured to detect blue). In the illustrated embodiment, the 3D depth sensing region includes one or more sensing pixels 304 configured to detect NIR light. The sensing pixels 301, 302, 303 of the 2D sensing region and the sensing pixels 304 of the 3D depth sensing region can be interleaved in various suitable configurations. In one or more embodiments, the 2D sensing region includes a first row of sensing pixels 307 and a second row of sensing pixels 305, the first row of sensing pixels 307 having alternating green sensing pixels 301 and red sensing pixels 302, and the second row of sensing pixels 305 having alternating blue sensing pixels 303 and green sensing pixels 301. The first sensing pixel row 307 may be directly adjacent to the second sensing pixel row 305 to form a 2×2 square segment comprising two green sensing pixels 301, one red sensing pixel 302, and one blue sensing pixel 303. The 2×2 segment may be repeated in both the row direction (e.g., the first direction DR1) and the column direction (e.g., the second direction DR2 intersecting the first direction DR1) to form a Bayer image. However, this disclosure is not limited thereto. For example, the 2D sensing region may include sensing pixels arranged in any suitable shape with any suitable arrangement of color sensing pixels to suit other imaging schemes (e.g., monochrome image schemes, red and transparent image schemes, or transparent image schemes).

[0083] According to one or more embodiments of this disclosure, the ratio between the sensing pixels of the 3D depth sensing region and the sensing pixels of the 2D sensing region can be appropriately varied. For example, in one or more embodiments, the ratio between the sensing pixels of the 3D depth sensing region and the sensing pixels of the 2D sensing region may be 1:1 (e.g., one sensing pixel of a SPAD compared to one monochrome sensing pixel), and in other embodiments, the ratio between the sensing pixels of the 3D depth sensing region and the sensing pixels of the 2D sensing region may be greater than 1:1 (e.g., as...). Figures 3A to 3D As shown, each group of red, green, and blue sensing pixels 301, 302, and 303 has one sensing pixel 304. Figures 3A to 3H As shown, the ratio of sensing pixels 304 in the 3D depth sensing region to sensing pixels 301, 302, and 303 in the 2D sensing region can affect the resolution of the corresponding data in the interleaved arrangement. For example, increasing the number of sensing pixels 301, 302, and 303 in the 2D sensing region for each sensing pixel 304 in the 3D depth sensing region increases the resolution of the 2D sensing data but decreases the resolution of the 3D depth sensing data. (Refer to...) Figures 3A to 3DDifferent ratios of sensing pixels 304 in the 3D depth sensing region and sensing pixels 301, 302, and 303 in the 2D sensing region can be provided to balance high-resolution 2D images and minimum-resolution 3D depth information, which makes it feasible to detect objects of the smallest size.

[0084] In one or more embodiments of this disclosure, such as Figures 3A to 3D As shown in the embodiments, the ratio of the area occupied by the sensing pixel 304 of the 3D depth sensing region to the area occupied by the sensing pixels 301, 302, and 303 of the 2D sensing region can be 1:4. However, this disclosure is not limited thereto. For example, the ratio of the area occupied by the sensing pixel 304 of the 3D depth sensing region to the area occupied by the sensing pixels 301, 302, and 303 of the 2D sensing region can be appropriately varied (e.g., an area ratio of 1:1 or greater than 1:1).

[0085] like Figure 3A As shown, sensing pixels 301, 302, and 303 of the 2D sensing region can be interrupted (or replaced) by sensing pixels 304 of the 3D depth sensing region, allowing the 2D sensing region and the 3D depth sensing region to be interwoven. In other words, depending on the pattern of the columns and / or rows of sensing pixels in the 2D sensing region, sensing pixels 304 of the 3D depth sensing region may replace one or more sensing pixels 301, 302, and 303 of the 2D sensing region. In one or more embodiments, sensing pixels 304 of the 3D depth sensing region may be repeated with a first interval 306 in the row direction (e.g., a first direction DR1) and with a second interval 308 in the column direction (e.g., a second direction DR2). In this case, the first interval 306 and the second interval 308 may be equal to each other, thus the sensing pixels 304 of the 3D depth sensing region may have a square arrangement. Figure 3A In the illustrated embodiment, the ratio of sensing pixels 304 in the 3D depth sensing region to sensing pixels 301, 302, 303 in the 2D sensing region can be 1:12, and each sensing pixel 304 in the 3D depth sensing region can be separated from another sensing pixel 304 in the 3D depth sensing region by two of the sensing pixels 301, 302, 303 in the 2D sensing region. However, this disclosure is not limited thereto. For example, with the corresponding influence of the ratio of sensing pixels 304 in the 3D depth sensing region to sensing pixels 301, 302, 303 in the 2D sensing region, multiple sensing pixels 304 in the 3D depth sensing region can be separated from each other by any suitable number of sensing pixels 301, 302, 303 in the 2D sensing region.

[0086] like Figure 3BAs shown, in one or more embodiments, the sensing pixels 304 of the 3D depth sensing region may be repeated with a third interval 310 in the row direction (e.g., the first direction DR1) and a fourth interval 312 in the column direction (e.g., the second direction DR2) for both the row direction (e.g., the first direction DR1) and the column direction (e.g., the second direction DR2). In this case, the third interval 310 and the fourth interval 312 may be different from each other, and therefore, the sensing pixels 304 of the 3D depth sensing region may have a rectangular arrangement. Figure 3B In the illustrated embodiment, the ratio of sensing pixels 304 in the 3D depth sensing region to sensing pixels in the 2D sensing region can be 1:20, and each sensing pixel 304 in the 3D depth sensing region can be separated from another sensing pixel 304 in the 3D depth sensing region by two of the sensing pixels 301, 302, 303 in the 2D sensing region in the row direction (e.g., first direction DR1) and by four of the sensing pixels 301, 302, 303 in the 2D sensing region in the column direction (e.g., first direction DR1). However, this disclosure is not limited thereto. For example, with the corresponding effect on the ratio of sensing pixels 304 in the 3D depth sensing region to sensing pixels 301, 302, 303 in the 2D sensing region, the sensing pixels 304 in the 3D depth sensing region can be separated from each other by any suitable number of sensing pixels 301, 302, 303 in the 2D sensing region in the row direction (e.g., first direction DR1) and / or the column direction (e.g., second direction DR2).

[0087] like Figure 3C As shown, in one or more embodiments, the sensing pixels 304 of the odd-numbered rows 315 of the 3D depth sensing region may have a first rectangular shape arrangement (indicated by a fifth interval 314 different from the sixth interval 316), and the sensing pixels 304 of the even-numbered rows 317 of the 3D depth sensing region may have a second rectangular shape arrangement (matching the first rectangular shape arrangement). The first rectangular arrangement may be offset from the second rectangular arrangement in the row direction (e.g., the first direction DR1) and / or the column direction (e.g., the second direction DR2). In other words, the sensing pixels 304 of the 3D depth sensing region may have a zig-zag shape arrangement. Figure 3CIn the illustrated embodiment, each column of sensing pixels in the 2D sensing region may be interrupted (or replaced) by sensing pixels 304 in the 3D depth sensing region. Furthermore, the ratio of sensing pixels 304 in the 3D depth sensing region to sensing pixels 301, 302, and 303 in the 2D sensing region may be 1:12, and each sensing pixel in the 3D depth sensing region may be separated from another sensing pixel in the 3D depth sensing region by two of the sensing pixels 301, 302, and 303 in the row direction (e.g., the first direction DR1) and by six of the sensing pixels 301, 302, and 303 in the column direction (e.g., the second direction DR2). However, this disclosure is not limited thereto. For example, with the corresponding influence of the ratio of sensing pixels 304 in the 3D depth sensing region to sensing pixels 301, 302, 303 in the 2D sensing region, the sensing pixels 304 in the 3D depth sensing region can be separated from each other by any suitable number of sensing pixels 301, 302, 303 in the 2D sensing region in the row direction (e.g., the first direction DR1) and / or column direction (e.g., the second direction DR2).

[0088] like Figure 3D As shown, in one or more embodiments, the sensing pixels 304 of the odd-numbered rows 315 of the 3D depth sensing region may have a first square arrangement (indicated by a seventh interval 318, which is the same as the eighth interval 320), and the sensing pixels 304 of the even-numbered rows 317 of the 3D depth sensing region may have a second square arrangement (matching the first square arrangement). The first square arrangement may be offset from the second square arrangement at the same interval (i.e., the ninth interval 322 equals the tenth interval 324) in the row direction (e.g., the first direction DR1) and / or column direction (e.g., the second direction DR2). In other words, the sensing pixels 304 of the 3D depth sensing region may have a diamond arrangement. Figure 3DIn the illustrated embodiment, some columns of sensing pixels in the 2D sensing region may not be interrupted (or replaced) by sensing pixels 304 in the 3D depth sensing region. Furthermore, the ratio of sensing pixels 304 in the 3D depth sensing region to sensing pixels 301, 302, 303 in the 2D sensing region may be 1:28, and each sensing pixel 304 in the 3D depth sensing region may be separated from another sensing pixel 304 in the 3D depth sensing region by six of the sensing pixels 301, 302, 303 in the row direction (e.g., first direction DR1) and by six of the sensing pixels 301, 302, 303 in the column direction (e.g., second direction DR2). However, this disclosure is not limited thereto. For example, with the corresponding influence of the ratio of sensing pixels 304 in the 3D depth sensing region to sensing pixels 301, 302, 303 in the 2D sensing region, the sensing pixels 304 in the 3D depth sensing region can be separated from each other by any suitable number of sensing pixels 301, 302, 303 in the 2D sensing region in the row direction (e.g., the first direction DR1) and / or column direction (e.g., the second direction DR2).

[0089] Reference Figures 3A to 3D Each of the various arrangements can affect object detection and tracking performed by the object detector 212 through the processing circuit 204. For example, the minimum bounding box size for identifying objects (e.g., people, vehicles, and obstacles) in 2D and 3D images can be affected by the arrangement and ratio of sensing pixels 304 in the 3D depth sensing region to sensing pixels 301, 302, and 303 in the 2D sensing region. Figure 3A It can have a minimum bounding box size of 30×8. Figure 3B It can have a minimum bounding box size of 30×8. Figure 3C It can have a minimum bounding box size of 30×6. Figure 3D It can have a minimum bounding box size of 30×8.

[0090] Reference Figures 3E to 3HThe sensing pixel rows 326, 328, 330, 332, 334, and 336 of the hybrid sensing arrays 300e, 300f, and 300g can be sensing pixel rows 328, 330, 334, and 336 of a 2D sensing region or sensing pixel rows 326 and 332 of a 3D depth sensing region. Similarly, the sensing pixel rows 326, 328, 330, 332, 334, and 336 of the hybrid sensing array 300h can be sensing pixel rows 328, 332, and 336 of a 2D sensing region or sensing pixel rows 326, 330, and 334 of a 3D depth sensing region. In other words, the 2D sensing region and the 3D depth sensing region can be arranged as alternating horizontal bars in a stripe arrangement. However, this disclosure is not limited to this. For example, the sensing pixel rows 328, 330, 334, and 336 of the 2D sensing region and the sensing pixel rows 326 and 332 of the 3D depth sensing region can be arranged as alternating vertical bars in a stripe arrangement, depending on their application. For example, in autonomous driving applications, objects located in front of the vehicle may be more important than objects located to the side of the vehicle. Therefore, in this case, the sensing pixel rows 326, 332 of the 3D depth sensing region can be arranged as alternating horizontal bars in a stripe arrangement.

[0091] like Figure 3E As shown, in one or more embodiments, the sensing pixels 304 of the 3D depth sensing region may be continuously repeated in the row direction (e.g., the first direction DR1), and the sensing pixels 301, 302, and 303 of the 2D sensing region may also be continuously repeated in the row direction (e.g., the first direction DR1). In the illustrated embodiment, the sensing pixels 304 of the 3D depth sensing region have a square shape. However, this disclosure is not limited thereto. For example, the sensing pixels 304 of the 3D depth sensing region may have any suitable shape (such as...). Figures 3F to 3H (The rectangular shape shown in the embodiment).

[0092] Reference Figure 3E In the column direction (e.g., the second direction DR2), the sensing pixel row 326 of the 3D depth sensing region can be separated from another sensing pixel row 332 of the 3D depth sensing region by two sensing pixel rows 328 and 330 of the 2D sensing region. However, this disclosure is not limited thereto. For example, as Figure 3G and Figure 3H As shown, the sensing pixel rows 326 and 332 of the 3D depth sensing region can be separated from each other by any number of sensing pixel rows of the 2D sensing region at regular or irregular intervals.

[0093] Reference Figure 3I The hybrid sensing array 300i may include a 2D sensing region 338 and a 3D depth sensing region 340 arranged side by side and separated from each other.

[0094] like Figure 3I As shown, the 2D sensing region 338 is generally adjacent to the 3D depth sensing region 340. In other words, the sensing pixels 304 of the 3D depth sensing region 340 can be directly adjacent to each other, such that the sensing pixels 301, 302, and 303 of the 2D sensing region 338 are not located between the sensing pixels 304 of the 3D depth sensing region 340 in the first direction DR1 or the second direction DR2. Figures 3A to 3H As shown in the embodiments, compared with the arrangement where the sensing pixels 301, 302, 303 of the 2D sensing region can be intertwined with the sensing pixels 304 of the 3D depth sensing region, the side-by-side arrangement of the 2D sensing region 338 and the 3D depth sensing region 340 can provide enhanced 2D image resolution due to the increased density of the sensing pixels 301, 302, 303 of the 2D sensing region.

[0095] Figures 4A to 4D This is a plan view of a portion of a hybrid sensing array 400a, 400b, 400c, 400d having a plurality of superlenses (e.g., superlens 406) on a hybrid sensing array 400a, 400b, 400c, 400d according to one or more embodiments of the present disclosure.

[0096] Reference Figures 4A to 4D In one or more embodiments, the sensing pixels 404 of the odd-numbered rows 408, 412 of the 3D depth sensing region may have a first rectangular shape arrangement (indicated by eleventh and twelfth intervals 416, which are distinct from each other), and the sensing pixels 404 of the even-numbered rows 410 of the 3D depth sensing region may have a second rectangular shape arrangement. The first rectangular arrangement may be offset from the second rectangular arrangement at different intervals (e.g., thirteenth interval 418 differs from fourteenth interval 420) in the row direction (e.g., first direction DR1) and / or column direction (e.g., second direction DR2). In other words, the sensing pixels of the 3D depth sensing region may have a zigzag arrangement. However, this disclosure is not limited thereto, and any suitable shape arrangement may be used. For example, in one or more embodiments, a rhombus, square, or rectangular shape arrangement may be used.

[0097] exist Figures 4A to 4BIn the illustrated embodiment, some columns of sensing pixels in the 2D sensing region may not be interrupted (or replaced) by sensing pixels 404 in the 3D depth sensing region. Furthermore, each sensing pixel 404 in the 3D depth sensing region may be separated from another sensing pixel 404 in the 3D depth sensing region by nine sensing pixels 401, 402, 403 or 402, 405 in the row direction (e.g., first direction DR1), and by seventeen sensing pixels 401, 402, 403 or 402, 405 in the column direction (e.g., second direction DR2). However, this disclosure is not limited thereto. For example, the sensing pixels 404 in the 3D depth sensing region may be separated from each other by any suitable number of sensing pixels 401, 402, 403 or 402, 405 in the 2D sensing region in the row direction (e.g., first direction DR1) and / or column direction (e.g., second direction DR2). Figures 4C to 4D In the embodiment shown, each sensing pixel 404 of the 3D depth sensing region may be separated from another sensing pixel 404 of the 3D depth sensing region in the row direction (e.g., the first direction DR1) by nine sensing pixels 401, 402, 403 or 402, 405, and in the column direction (e.g., the second direction DR2) by fifteen sensing pixels 401, 402, 403 or 402, 405.

[0098] although Figures 4A to 4B The sensing pixel 404 of the 3D depth sensing region shown has an area of ​​a specific size and shape (e.g., a square area having an area ratio of one sensing pixel 404 of the 3D depth sensing region to the area of ​​nine sensing pixels 401, 402, 403 or 402, 405 of the 2D sensing region; however, the sensing pixel 404 of the 3D depth sensing region can be any suitable size or shape). Figures 4C to 4DAs shown, the sensing pixel 404 of the 3D depth sensing region can have a larger area, and this area can be rectangular (e.g., a rectangular area representing the ratio of the area of ​​one sensing pixel 404 of the 3D depth sensing region to the area of ​​15 sensing pixels 401, 402, 403 or 402, 405 of the 2D sensing region). In the case of a sensing pixel 404 with a rectangular shape in the 3D depth sensing region, the sensing pixel 404 of the 3D depth sensing region is more suitable for detecting narrow objects compared to a sensing pixel 404 of a square shape. For example, a square shape is more suitable for detecting objects with a square aspect ratio, while a rectangular shape is more suitable for detecting narrow objects with a rectangular aspect ratio. Furthermore, the larger area of ​​the sensing pixel 404 of the 3D depth sensing region is more suitable for detecting objects at greater distances (e.g., NIR reflected from the object is more likely to be guided (or focused) onto a portion of the sensing pixel 404 of the 3D depth sensing region). By arranging the larger area sensing pixels 404 of the 3D depth sensing region to be adjacent to each other, the sensing signal provided by each sensing pixel 404 of the 3D depth sensing region can be confirmed by the adjacent sensing pixels 404 of the 3D depth sensing region, thereby avoiding false positive 3D depth information caused by noise or other issues at a single sensing pixel 404 of the 3D depth sensing region. Therefore, depending on the application, sensing pixels of the 3D depth sensing region with appropriate size and shape are desirable.

[0099] In one or more embodiments, such as Figures 4A to 4D As shown, a plurality of superlenses (e.g., superlens 406) may cover each sensing pixel 404 of the 3D depth sensing region and one or more sensing pixels 401, 402, 403 or 402, 405 of the 2D sensing region that may be adjacent to the sensing pixels 404 of the 3D depth sensing region. In this case, light incident on superlens 406 may be guided, depending on the wavelength, to the sensing pixels 401, 402, 403 or 402, 405 of the 2D sensing region or the sensing pixels 404 of the 3D depth sensing region. For example, visible light incident on superlens 406 may be guided (or focused) on the sensing pixels 401, 402, 403 or 402, 405 of the 2D sensing region, and NIR light incident on superlens 406 may be guided (or focused) on the sensing pixels 404 of the 3D depth sensing region that are adjacent (e.g., directly adjacent) to the sensing pixels 401, 402, 403 or 402, 405 of the 2D depth sensing region.

[0100] In one or more embodiments, reference will be made as follows. Figures 5A to 6BIn more detail, the superlens 406 guides (or focuses) light based on a nanostructure, which can be appropriately modified to guide (or focus) incident light with a set wavelength spectrum to various sensing regions of the hybrid sensing array. In this case, the central portion of the superlens 406 can be designed to guide (or focus) light of different wavelengths from the outer portions of the superlens 406 surrounding the central portion.

[0101] Although in the illustrated embodiment, each of the plurality of superlenses 406 covers a single sensing pixel 404 of the 3D depth sensing region and sensing pixels 401, 402, 403 or 402, 405 of the 2D depth sensing region adjacent to the sensing pixel 404 of the 3D depth sensing region, this disclosure is not limited thereto. For example, the superlens 406 may cover only the sensing pixels 401, 402, 403 or 402, 405 of the 2D sensing area, but not the sensing pixels 404 of the 3D depth sensing area. This allows the superlens 406 to direct (or focus) visible light onto the sensing pixels 401, 402, 403 or 402, 405 of the 2D sensing area, and to direct (or focus) NIR light onto the sensing pixels 404 of the 3D depth sensing area. In other embodiments, the superlens 406 may cover only the sensing pixels 404 of the 3D depth sensing area, but not the sensing pixels 401, 402, 403 or 402, 405 of the 2D sensing area. This allows the superlens 406 to direct (or focus) visible light onto the sensing pixels of the 2D sensing area, and to direct (or focus) NIR light onto the SPAD. Therefore, the superlens 406 can be of any shape and, with appropriate adjustments, can cover any number or type of sensing pixels in the hybrid sensing array.

[0102] although Figures 4A to 4D Multiple superlenses are shown, but this disclosure is not limited thereto. For example, any number of superlenses 406 can be used to cover the hybrid sensing array, such as a global lens (i.e., a single superlens) having a nanostructure thereon, which can be used to cover the entire hybrid sensing array with appropriate modifications to the nanostructure of the global lens. Although Figure 4A and Figure 4C The embodiments include red sensing pixels, green sensing pixels, and blue sensing pixels, but this disclosure is not limited thereto. For example, such as Figure 4B and Figure 4D As shown, the sensing pixels may include red sensing pixels and transparent sensing pixels, instead of red sensing pixels, green sensing pixels and blue sensing pixels.

[0103] Figures 5A to 5CThe images are cross-sectional views 500a, 500b, and 500c of a hybrid sensing array having a superlens 502 and a microlens 504 on a hybrid sensing array according to one or more embodiments of the present disclosure.

[0104] Reference Figures 5A to 5C According to one or more exemplary embodiments of this disclosure, microlens 504 and superlens 502 may be on a hybrid sensing array 508. Superlens 502 may include a plurality of nanostructures 512, 514, 518, 522, 524, 526 (e.g., a plurality of thin dielectric nanoantenna blocks or scatterers) disposed on a rigid or flexible transparent substrate adapted to transmit external light of one or more target wavelength spectra. Superlens 502 (e.g., the flexible transparent substrate of superlens 502) may be of any suitable shape (e.g., circular, cylindrical, rectangular, or square shapes). In one or more embodiments, superlens 502 may have polarization-independent properties.

[0105] The shape, size, and position of each of the nanostructures 512, 514, 518, 522, 524, and 526 can affect the properties of light (such as phase, polarization, and focus). Therefore, each of the nanostructures 512, 514, 518, 522, 524, and 526 can be configured on a substrate with different geometries and / or arrangements to guide (or focus) external light of different target wavelength spectra onto different portions of the hybrid sensing array 508. In one or more embodiments, the nanostructures 512, 514, 518, 522, 524, and 526 can be circular, cylindrical, rectangular, square, etc., and the nanostructures 512, 514, 518, 522, 524, and 526 can be arranged such that one or more superlenses (e.g., superlens 502) exhibit polarization-independent properties.

[0106] In one or more embodiments, such as Figure 5A As shown, the superlens 502 includes a first group 510 of nanostructures, a second group 516 of nanostructures, and a third group 520 of nanostructures. The first group 510 of nanostructures (e.g., a first nanostructure 512 and a second nanostructure 514) overlaps with a first sensing pixel 511 of the 2D sensing region (e.g., on a third-direction DR3). The second group 516 of nanostructures (e.g., a third nanostructure 518) overlaps with a second sensing pixel 517 of the 2D sensing region (e.g., on a third-direction DR3). The third group 520 of nanostructures (e.g., a fourth nanostructure 522, a fifth nanostructure 524, and a sixth nanostructure 526) overlaps with a sensing pixel 519 of the 3D depth sensing region (e.g., on a third-direction DR3).

[0107] In one or more embodiments, a first set 510 of nanostructures (e.g., a first nanostructure 512 and a second nanostructure 514) may be configured to direct (or focus) external light of different wavelength spectra to different portions of the hybrid sensing array 508. For example, a first nanostructure 512 overlapping with a first sensing pixel 511 may be configured to direct (or focus) light in the NIR spectrum (e.g., greater than 0.8 micrometers) to a sensing pixel 519 in a 3D depth sensing region directly adjacent to the first sensing pixel 511 in the 2D sensing region, and a second nanostructure 514 overlapping with the first sensing pixel 511 in the 2D sensing region may be configured to direct (or focus) light in the visible to NIR spectrum (e.g., about 0.4 micrometers to about 0.8 micrometers) to the first sensing pixel 511 in the 2D sensing region. Thus, the first set 510 of nanostructures can be used as a beam splitter by directing (or focusing) different wavelength spectra to different sensing regions of the hybrid sensing array.

[0108] In one or more embodiments, each nanostructure of the second group 516 (e.g., the third nanostructure 518) may be configured to direct (or focus) external light of the same wavelength spectrum to the same portion of the hybrid sensing array 508. For example, each nanostructure of the second group 516 (e.g., the third nanostructure 518) may be configured to direct (or focus) light in the visible to NIR spectrum (e.g., about 0.4 micrometers to about 0.8 micrometers) to the second sensing pixel 517 of the 2D sensing region. In other words, none of the nanostructures in the second group 516 that overlap with the second sensing pixel 517 of the 2D sensing region (e.g., overlap on a third-direction DR3) directs (or focuses) light to an adjacent sensing pixel. However, this disclosure is not limited thereto. For example, the second group 516 may have nanostructures that direct (or focus) light to adjacent and / or directly adjacent sensing pixels to serve as beam splitters.

[0109] In one or more embodiments, a third group 520 of nanostructures (e.g., a fourth nanostructure 522, a fifth nanostructure 524, and a sixth nanostructure 526) may be configured to guide (or focus) external light of different wavelength spectra onto different portions of the hybrid sensing array 508. For example, a fourth nanostructure 522 overlapping with a sensing pixel 519 in a 3D depth sensing region can be configured to guide (or focus) light in the NIR spectrum (e.g., greater than 0.8 micrometers) to the sensing pixel 519 in the 3D depth sensing region; a fifth nanostructure 524 overlapping with a sensing pixel 519 in a 3D depth sensing region can be configured to guide (or focus) light in the visible to NIR spectrum (e.g., about 0.4 micrometers to about 0.8 micrometers) to a first sensing pixel 511 in a 2D sensing region directly adjacent to the sensing pixel 519 in the 3D depth sensing region; and a sixth nanostructure 526 overlapping with a sensing pixel 519 in a 3D depth sensing region can be configured to guide (or focus) light in the visible to NIR spectrum (e.g., about 0.4 micrometers to about 0.8 micrometers) to a third sensing pixel 521 in a 2D sensing region directly adjacent to the sensing pixel 519 in the 3D depth sensing region. Therefore, the third group 520 of nanostructures can be used as a beam splitter by guiding (or focusing) light in different wavelength spectra to different sensing regions of the hybrid sensing array. In this configuration, the third group 520 can guide (or focus) light onto three different sensing pixels. However, this disclosure is not limited thereto, and any suitable guidance (or focusing) arrangement of nanostructures can be used.

[0110] Despite the superlens Figure 5A The present disclosure is shown and described as including nanostructures overlapping each sensing pixel, but is not limited thereto. For example, as Figure 5B As shown, some sensing pixels (e.g., Figure 5B In the embodiments, the sensing pixel 519 of the 3D depth sensing region may not have a nanostructure that overlaps with the sensing pixel 519 of the 3D depth sensing region in the third direction.

[0111] Despite the superlens in Figure 5A The present disclosure is shown and described as including nanostructures overlapping with sensing pixels, wherein beam splitting involves directly adjacent sensing pixels, but this disclosure is not limited thereto. For example, as Figure 5B As shown, except Figure 5A In addition to the third nanostructure 518 shown in the embodiments, the second group 516 may also include a seventh nanostructure 528. For example... Figure 5BAs shown, the third nanostructure 518 of the second group 516 can be configured to guide (or focus) light in the visible to NIR spectrum (e.g., about 0.4 micrometers to about 0.8 micrometers) to the second sensing pixel 517 of the 2D sensing region, and the seventh nanostructure 528 of the second group 516 can be configured to guide (or focus) light in the NIR spectrum (e.g., greater than 0.8 micrometers) to the sensing pixel 519 of the 3D depth sensing region adjacent to the second sensing pixel 517 of the 2D sensing region, and the first sensing pixel 511 of the 2D sensing region is located between the second sensing pixel 517 of the 2D sensing region and the sensing pixel 519 of the 3D depth sensing region.

[0112] In one or more embodiments, one or more microlenses (e.g., microlens 504) help focus incident light onto a target portion of the sensing pixel. Figure 5A and Figure 5B As shown in the embodiments, the microlens 504 may be located below the superlens 502, and in other embodiments, such as Figure 5C As shown in the embodiment, the microlens 504 may be located above the superlens 502. In other words, in Figure 5A and Figure 5B In one embodiment, the microlens 504 may be positioned between the superlens 502 and the hybrid sensing array, and in other embodiments, such as Figure 5C As shown in the embodiment, the superlens 502 can be positioned between the microlens 504 and the hybrid sensing array.

[0113] In one or more embodiments, a plurality of color filters 506 can filter light with a set wavelength spectrum. Figures 5A to 5C In the embodiments, each color filter 506 corresponds to the wavelength of light sensed by a sensing pixel that overlaps with the color filter 506 in the thickness direction (e.g., third direction DR3). For example, blue light can be transmitted through a blue color filter among a plurality of color filters 506 that overlap with a blue sensing pixel (e.g., first sensing pixel 511), and NIR light can be transmitted through an NIR color filter among a plurality of color filters 506 that overlap with a sensing pixel 519.

[0114] Therefore, as Figures 5A to 5C As shown, external light incident on a superlens 502, which includes one or more nanostructures (e.g., structures for scattering lobes of different wavelengths of light), can be guided (or focused) to a 2D sensing region and / or a 3D depth sensing region of a hybrid sensing array 508 based on one or more target wavelength spectra.

[0115] Despite the microlens 504 Figures 5A to 5CAs shown, microlens 504 provides dedicated light focusing capabilities to complement the focusing and redirection of the target wavelength spectrum by superlens 502. Therefore, in one or more embodiments, microlens 504 may be absent, where additional light focusing may not be required. However, in… Figures 5A to 5C In some embodiments, a microlens 504 may be present to help address the issue of the principal ray angle (“CRA”) or field of view (FOV) of the 2D sensing area and the 3D depth sensing area.

[0116] Figure 6A This is a block diagram of an exploded view of a portion of a global superlens 602 and that portion of the global superlens 602 on a hybrid sensing array, according to one or more embodiments of the present disclosure.

[0117] In one or more embodiments, external light 600, including NIR and visible light, may be incident on a global superlens 602. The global superlens may include one or more nanostructures 608 that guide (or focus) the external light 600 with different wavelength spectra, such that the external light 600 is split into visible and NIR light. In one or more embodiments, visible light may be guided (or focused) to sensing pixels of a 2D sensing region 604 including a PD, and NIR light may be guided (or focused) to sensing pixels of a 3D depth sensing region 606 including one or more SPADs.

[0118] Therefore, the global superlens 602 can cover the entire hybrid sensing array, and different parts of the global superlens 602 can guide (or focus) visible light to a portion of the 2D sensing area close to that portion of the global superlens 602, and different parts of the global superlens 602 can guide (or focus) NIR light to a portion of the 3D depth sensing area close to that portion of the global superlens 602.

[0119] Figure 6B This is an exploded block diagram of a lens (e.g., a global lens) 610 and a global superlens 612 on a hybrid sensing array 614, according to one or more embodiments of the present disclosure.

[0120] Reference Figure 6BIn one or more embodiments, the hybrid sensing array 614 may include 2D sensing regions 616 and 3D depth sensing regions 618 arranged side-by-side adjacent to each other, the 2D sensing regions 616 and 3D depth sensing regions 618 sharing the same field of view (FOV) due to a master lens. In one or more embodiments, a global superlens 612 may be centered to cover the 2D sensing region 616, such that visible light guided (or focused) onto the red, green, and blue sensing pixels of the 2D sensing region by the global superlens 612 results in high 2D image quality. Furthermore, the global superlens 612 guides (or focuses) sufficient NIR light onto the 3D depth sensing region 618 to provide 3D depth information. Therefore, centering the global superlens 612 to cover the 2D sensing region 616 helps to provide a high-resolution 2D image, while the NIR light redirected (or focused) onto the adjacent 3D depth sensing region 618 provides sufficient NIR light for the sensing pixels of the 3D depth sensing region 618 to detect the NIR light and provide 3D depth information.

[0121] Figure 7A These are plan views and cross-sectional views of a hybrid sensor 106 according to one or more embodiments of the present disclosure. Figure 7B This is a cross-sectional view of a hybrid sensor 106 according to one or more embodiments of the present disclosure.

[0122] Reference Figures 7A to 7B In one or more embodiments, the hybrid sensor 106 may include a hybrid sensing array 702, a main lens 708, a global superlens 710, and a printed circuit board 712 (“PCB”). The hybrid sensing array 702 may be located on the PCB 712 and includes a 2D sensing region 704 and a 3D depth sensing region 706 arranged side by side.

[0123] like Figure 7A As shown, the main lens 708 and the global superlens 710 may cover or overlap with the hybrid sensing array 702, such that the main lens 708 and the global superlens 710 are centered on the 2D sensing region 704. In one or more embodiments, the entire 2D sensing region 704 of the hybrid sensing array 702 and the entire 3D depth sensing region of the hybrid sensing array may overlap with the global superlens 710 and the main lens 708. However, this disclosure is not limited thereto. For example, as Figure 7B As shown, the entire 2D sensing area 704 of the hybrid sensing array 702 can overlap with the global superlens 710 and the main lens 708, while the area smaller than the entire 3D depth sensing area 706 can overlap with the global superlens 710 and the main lens 708.

[0124] In one or more embodiments, a global superlens 710 may be positioned between a main lens 708 and a 2D sensing region 704, such that external light incident on the main lens 708 passes through the global superlens 710 before being detected by the hybrid sensing array 702. The light passing through the global superlens 710 may be guided (or focused) by one or more nanostructures 714 located on a transparent substrate (e.g., a glass substrate) 716. Each of the one or more nanostructures may guide (or focus) visible light to the 2D sensing region 704, or guide (or focus) NIR light to the 3D depth sensing region 706. Because the light monitored by the 2D sensing region 704 and the 3D depth sensing region 706 passes through the shared main lens 708, both the 2D sensing region 704 and the 3D depth sensing region 706 have the same field of view (FOV). Therefore, centering the main lens 708 and the global superlens 710 to cover the 2D sensing region 704, as discussed in more detail below, helps to provide a high-resolution 2D image with enhanced 3D depth information.

[0125] like Figure 7A As shown, the planar areas and distances D2, D3, and D4 between the main lens 708, the global superlens 710, and the hybrid sensing array 702 can be set such that each sensing pixel of the hybrid sensing array 702 can be exposed to guided (or focused) light. The focal point of the main lens 708 can be spaced apart from the main lens 708 by a distance D4, and can be spaced apart from the global superlens 710 by a distance D3. In one or more embodiments, the focal point of the main lens can be between the main lens 708 and the global superlens 710. Furthermore, the hybrid sensing array 702 can be spaced apart from the global superlens 710 by a distance D2, such that each sensing pixel of the hybrid sensing array 702 can detect light guided (or focused) from the main lens 708 and / or the global superlens 710.

[0126] Therefore, the compact, integrated hybrid sensor 106 provides a main lens 708 and a global superlens 710 covering or overlapping the hybrid sensing array 702 to provide 2D image data and 3D depth information from a shared field of view (FOV). Furthermore, because the sensing pixels of the 3D depth sensing region 706 do not interrupt (or replace) the sensing pixels of the 2D sensing region 704, thus avoiding the "loss" of sensing pixels in the 2D sensing region 704, the side-by-side arrangement of the 2D sensing region 704 and the 3D depth sensing region 706 helps to provide high-resolution 2D images.

[0127] Figure 8 This is a flowchart of a method 800 for object detection and tracking according to one or more embodiments of the present disclosure.

[0128] Reference Figure 8The object detector 212 of the processing circuit 204 can use a neural network-based object detection system to detect and track objects in 2D image data based on data from the 2D sensing area.

[0129] In one or more embodiments, object detector 212 may use 2D image data to detect objects (802). Object detector 212 may determine whether the detected object is a new object (804). For example, object detector 212 may determine whether the object is a new object based on whether the object detected in the 2D image data has already been stored in object table 810.

[0130] If the object is a previously detected object, object detector 212 continues to detect other objects in the image as part of the object detection loop. If the object is not a new object previously detected in the current 2D image data, object detector 212 works with ROI controller 210 of processing circuitry 204 to activate a 3D depth sensing region. For example, object detector 212 can use 2D image data to detect objects (e.g., pedestrians) (802) and identify the coordinates of objects (e.g., pedestrians) in the 2D image data. Based on the coordinates of objects in the 2D image data, ROI controller 210 can activate a portion of light source 104 (e.g., a portion of the VCSEL array) to illuminate real-world objects (e.g., pedestrians). In one or more embodiments, ROI controller 210 can send a signal to sensing circuitry 202 to select specific sensing pixels (e.g., one or more sensing pixels including SPADs) of the 3D depth sensing region based on the coordinates of objects identified from the 2D image. In one or more embodiments, the sensing pixels of the 3D depth sensing region driven by the sensing circuit 202 may be adjacent to or close to the sensing pixels of the 2D sensing region that provides a sensing signal corresponding to 2D image data of an object (e.g., a pedestrian).

[0131] Therefore, 3D depth sensing information can be collected via hybrid sensor 106 (i.e., 3D depth sensing is enabled (806)). By selectively enabling 3D depth sensing in response to new objects, battery power consumption can be reduced and less computational resources can be used. Based on the combination of 2D image data and 3D depth information, object detector 212 can estimate information about new objects (such as shape, identity, and / or 2D size information) based on 2D sensing information (808) and estimate depth based on 3D depth sensing information, and store the estimated information in object table 810 before returning to detecting other objects (802), thereby completing the detection loop. In one or more embodiments, object table 810 may indicate whether an object is an OOI that should be tracked.

[0132] Therefore, in one or more embodiments, object detector 212 may track objects identified through an object table based on a detection loop. For example, in ADAS or autonomous vehicle applications, trees and pedestrians may be identified through a detection loop. In this case, object detector 212 may determine that trees in the background should not be tracked. On the other hand, object detector 212 may determine that pedestrians should be tracked (e.g., because pedestrians are mobile). Thus, pedestrians identified in 2D image data may be tracked as OOIs using ROI sensing. Although examples of trees and pedestrians are provided, this disclosure is not limited thereto. Depending on the application, any object may be designated for tracking based on several factors (e.g., location, importance, distance, and mobility).

[0133] In one or more embodiments, object detector 212 may continuously track objects in 2D image data (e.g., track the position of objects) (812). If object detector 212 loses tracking of a tracked object in the 2D image data (e.g., the tracked object exits the FOV or an error occurs) (814), object tracking ends until the detection loop detects another object or the same object again for tracking (i.e., object detector 212 exits the tracking loop and returns to the detection loop). Therefore, in the detection loop, there is a possibility that a lost tracked object will be re-identified and tracked again to correct any possible errors.

[0134] If object detector 212 is tracking an object (e.g., OOI) and a critical condition (e.g., in ADAS applications, OOI is in front) is encountered (816), object detector 212 may work with ROI controller 210 of processing circuitry 204 to activate a 3D depth sensing region. ROI controller 210 may send signals to sensing circuitry 202 to select specific sensing pixels of the 3D depth sensing region and / or activate portions of light source 104 (e.g., portions of an array of VCSELs) to illuminate the detected object in the real world. Thus, 3D depth sensing information can be collected by hybrid sensor 106 (i.e., 3D depth sensing is enabled (818)). By selectively enabling 3D depth sensing in response to a critical condition (818), battery power consumption can be reduced and less computational resources can be used. Based on the new information from 3D depth sensing, object detector may enhance or update the 3D depth information of the tracked object (820). In one or more embodiments, the combination of 2D image data and 3D depth information may also enable the object detector to re-evaluate or update object information (such as shape, identity and / or 2D size information) based on 2D sensing information in the object table (820) and to re-evaluate or update depth based on 3D depth sensing information in the object table before returning to the tracked object, until the tracked object is lost (814) or another critical condition is met (816).

[0135] Although the description refers to the detection and tracking loops for a single object Figure 8 However, this disclosure is not limited thereto. For example, multiple objects may be detected and tracked simultaneously as needed. Furthermore, since objects may be designated for tracking by object detector 212 (e.g., designated as OOI), the detection loop may always be active while the tracking loop is active. In one or more embodiments, all objects detected by object detector 212 may be tracked, and in other embodiments, fewer objects than all objects detected by object detector 212 may be tracked as needed.

[0136] Figure 9 This is a flowchart 900 of a method for providing high-resolution 2D images and high-resolution 3D images according to one or more embodiments of the present disclosure.

[0137] Reference Figure 9 According to one or more example embodiments of this disclosure, the image signal processor 214 may use data from the sensing circuit 202 to provide or reconstruct 2D and / or 3D images.

[0138] In one or more embodiments, the image signal processor 214 may provide or reconstruct a 2D image based on the arrangement of sensing pixels in a 2D sensing region and sensing pixels in a 3D depth sensing region. For example, when sensing pixels in a 2D sensing region are interleaved with sensing pixels in a 3D depth sensing region, the 2D image data may have missing patches because the sensing pixels in the 3D depth sensing region occupy areas of the hybrid sensing array that interrupt (or replace) rows and / or columns of sensing pixels in the 2D sensing region. Therefore, the 2D image data generated by the 2D sensing region may have missing pixels (e.g., one or more missing green pixels, one or more missing red pixels, and / or one or more missing blue pixels). However, this disclosure is not limited thereto. For example, in one or more embodiments, any sensing pixel interrupted (or replaced) by sensing pixels in the 3D depth sensing region may result in a “missing” pixel; therefore, the missing pixel can be of any color and / or type, depending on the sensing pixels in the 2D sensing region that may be interrupted (or replaced) by sensing pixels in the 3D depth sensing region.

[0139] In the illustrated embodiment, in response to 2D image data from the 2D sensing area, the image signal processor 214 may interpolate missing green pixels (902), and missing red and blue pixels (904). In response to interpolating the missing pixels, the image signal processor 214 may complete a Bayer image (906). In response to completing the Bayer image, the image signal processor 214 may use a demosaicing algorithm to reconstruct a panchromatic image (908). Thus, the panchromatic image can be reconstructed for use in various applications (e.g., by the object detector 212 of the processing circuitry 204) (910).

[0140] Although the reconstruction of a 2D image is described with reference to interpolation (902, 904) of missing pixels, according to embodiments of this disclosure, the interpolation (902, 904) of missing pixels can be skipped. For example, sensing pixels in the 2D sensing region and sensing pixels in the 3D depth sensing region are in the following positions: Figure 2B , Figure 3I , Figure 6B and Figures 7A to 7B In the side-by-side arrangement shown in the embodiment, there may be no missing pixels to be interpolated, so interpolation of missing pixels can be skipped. In other words, Figure 2B , Figure 3I , Figure 6B and Figures 7A to 7B An embodiment can begin from step 910.

[0141] In one or more embodiments, in response to reconstructing a panchromatic image, the image signal processor 214 can interpolate the missing depth pixels using a weighted average (912). In other words, the 3D depth information from the 3D depth sensing area may not have the same resolution as the 2D image data; therefore, the missing 3D depth information can be determined based on interpolation using the sensed 3D depth information and a weighted average. Thus, a high-resolution 3D image can be provided that includes both a high-resolution 2D image as a base or foundation and 3D depth information that enhances or updates the high-resolution 2D image.

[0142] Therefore, as disclosed herein, embodiments of this disclosure provide high-resolution 2D and 3D images based on a hybrid sensing array comprising a 2D sensing region and a 3D depth sensing region having a shared field of view (FOV). Furthermore, systems and methods of one or more embodiments of this disclosure provide ROI sensing capabilities based on object detection and tracking algorithms to save power and computational resources.

[0143] In the accompanying drawings, the relative dimensions of elements, layers, and regions may be exaggerated and / or simplified for clarity.

[0144] It will be understood that although the terms “first,” “second,” “third,” etc., may be used herein to describe various elements, components, regions, layers, and / or parts, these elements, components, regions, layers, and / or parts should not be limited by these terms. These terms are used to distinguish one element, component, region, layer, or part from another element, component, region, layer, or part. Therefore, without departing from the spirit and scope of this disclosure, the first element, first component, first region, first layer, or first part described above can be referred to as a second element, second component, second region, second layer, or second part.

[0145] It will be understood that when an element or layer is referred to as being "on," "connected to," or "bonded to" another element or layer, the element or layer may be directly on, directly connected to, or directly bonded to the other element or layer, or one or more intermediate elements or layers may exist. Furthermore, it will be understood that when an element or layer is referred to as being "between" two elements or layers, the element or layer may be the only element or layer between the two elements or layers, or one or more intermediate elements or layers may exist.

[0146] The terminology used herein is for the purpose of describing particular embodiments and is not intended to limit this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. It will also be understood that the terms “comprising,” “including,” and “having” as used in this specification indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items. Expressions such as “at least one of…” when following a column of elements modify the entire column, not individual elements within the column.

[0147] Furthermore, the use of the word "may" in describing embodiments of this disclosure means "one or more embodiments of this disclosure". As used herein, the terms "use", "in use" and "being used" may be considered synonymous with the terms "utilize", "being exploited" and "being exploited", respectively.

[0148] As used herein, the terms “substantially,” “about,” and similar terms are used as approximate terms rather than terms of degree, and are intended to explain the inherent biases of measurements or calculations that will be recognized by one of ordinary skill in the art.

[0149] Any numerical range listed herein is intended to include all subranges of the same numerical precision falling within the listed range. For example, the range “1.0 to 10.0” is intended to include all subranges between the listed minimum value 1.0 and the listed maximum value 10.0 (and includes both the listed minimum value 1.0 and the listed maximum value 10.0) (i.e., a minimum value equal to or greater than 1.0 and a maximum value equal to or less than 10.0 (e.g., 2.4 to 7.6)). Any maximum numerical limit listed herein is intended to include all lower numerical limits falling within it, and any minimum numerical limit listed in this specification is intended to include all higher numerical limits falling within it. Therefore, the applicant reserves the right to amend this specification (including the claims) to expressly list any subranges within the range expressly listed herein.

[0150] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. It will also be understood that, unless expressly defined herein, terms (such as those defined in a general dictionary) should be interpreted as having the same meaning as they have in the context of the relevant field and / or in this specification, and should not be interpreted in an idealized or overly formalistic manner.

[0151] Although some exemplary embodiments have been described, those skilled in the art will readily understand that various modifications can be made to the exemplary embodiments without departing from the spirit and scope of this disclosure. It will be understood that, unless otherwise stated, the description of features or aspects within each embodiment should generally be considered applicable to other similar features or aspects in other embodiments. Therefore, it should be understood that the foregoing is a description of various exemplary embodiments and should not be construed as limiting the specific exemplary embodiments disclosed herein, and that various modifications to the disclosed exemplary embodiments, as well as other exemplary embodiments, are intended to be included within the spirit and scope of this disclosure as defined in the appended claims and their equivalents.

Claims

1. A three-dimensional sensing system, comprising: The light source is configured to emit light; A hybrid sensing array includes a two-dimensional sensing region and a three-dimensional depth sensing region, wherein the two-dimensional sensing region is configured to detect ambient light reflected from an object, and the three-dimensional depth sensing region is configured to detect light emitted by a light source and reflected from the object; A superlens, on a hybrid sensing array, is configured to: guide ambient light reflected from an object to a two-dimensional sensing region, and guide light emitted by a light source and reflected from the object to a three-dimensional depth sensing region; and The processing circuit is configured to combine two-dimensional image information provided by the two-dimensional sensing area and three-dimensional depth information provided by the three-dimensional depth sensing area to generate a combined three-dimensional image. The superlens comprises multiple nanostructures configured on a substrate with different geometries and / or arrangements to guide external light of different target wavelength spectra to different parts of the hybrid sensing array.

2. The three-dimensional sensing system according to claim 1, wherein, The processing circuitry is also configured as follows: Determine the two-dimensional image information of the object based on the two-dimensional sensing area; The three-dimensional depth information of an object is determined based on the three-dimensional depth sensing area; and Update the object's two-dimensional image information with three-dimensional depth information.

3. The three-dimensional sensing system according to claim 1 further includes: The row decoder and column amplifier are stacked below the hybrid sensing array.

4. The three-dimensional sensing system according to claim 1, wherein, Superlenses are circular, cylindrical, rectangular, or square in shape, and are polarization-independent.

5. The three-dimensional sensing system according to claim 1, wherein, The two-dimensional sensing area includes one or more photodiodes, and the three-dimensional depth sensing area includes one or more avalanche photodiodes or single-photon avalanche diodes.

6. The three-dimensional sensing system according to claim 1, wherein, The superlens is configured to guide near-infrared light to the three-dimensional depth sensing region and visible light to the two-dimensional sensing region.

7. The three-dimensional sensing system according to any one of claims 1 to 6, wherein, The processing circuitry is also configured as follows: Detecting objects based on two-dimensional image information; Generating regions of interest based on the coordinates of objects in two-dimensional image information; and Illuminate the region of interest with a light source.

8. The three-dimensional sensing system according to claim 1, wherein, The processing circuitry is also configured as follows: The identity of an object is determined based on two-dimensional image information and three-dimensional depth information; Tracking objects based on two-dimensional image information; and In response to critical conditions, the two-dimensional image information is updated with three-dimensional depth information.

9. A method for providing an image based on a hybrid sensing array, the method comprising: Emitting light from a light source; Ambient light reflected from the object is detected through the two-dimensional sensing area of ​​the hybrid sensing array; The three-dimensional depth sensing area of ​​the hybrid sensing array detects light emitted by the light source and reflected from the object toward the three-dimensional depth sensing area; Ambient light is guided to the two-dimensional sensing area by the superlens on the hybrid sensing array; The superlens of the hybrid sensing array guides light emitted from the light source and reflected from the object into the three-dimensional depth sensing area; and The processing circuit combines the two-dimensional image information provided by the two-dimensional sensing area and the three-dimensional depth information provided by the three-dimensional depth sensing area to generate a combined three-dimensional image. The superlens comprises multiple nanostructures configured on a substrate with different geometries and / or arrangements to guide external light of different target wavelength spectra to different parts of the hybrid sensing array.

10. The method according to claim 9, further comprising: The two-dimensional image information of the object is determined by the processing circuit based on the two-dimensional sensing area; The three-dimensional depth information of an object is determined by processing circuitry based on the three-dimensional depth sensing area. and The two-dimensional image information of the object is updated using three-dimensional depth information through processing circuitry.

11. The method according to claim 9, wherein, The hybrid sensing array that detects ambient light and light emitted by the light source is stacked on top of the row decoder and column amplifier.

12. The method according to claim 9, wherein, The superlens that guide ambient light to the two-dimensional sensing area and guide light emitted by the light source and reflected from the object to the three-dimensional depth sensing area are circular, cylindrical, rectangular, or square in shape and are polarization-independent.

13. The method according to claim 9, wherein, The two-dimensional sensing region includes one or more photodiodes for detecting ambient light, and the three-dimensional depth sensing region includes an avalanche photodiode or a single-photon avalanche diode for detecting light emitted by a light source and reflected from an object.

14. The method according to claim 9, wherein, The step of guiding ambient light to a two-dimensional sensing region via a superlens on the hybrid sensing array includes: guiding visible light to the two-dimensional sensing region, and The step of guiding light emitted by a light source and reflected from an object to a three-dimensional depth sensing region through the superlens of the hybrid sensing array includes: guiding near-infrared light to the three-dimensional depth sensing region.

15. The method according to any one of claims 9 to 14, wherein the method further comprises: Objects are detected based on two-dimensional image information through processing circuitry. The region of interest is generated by processing circuitry based on the coordinates of objects in two-dimensional image information; and The processing circuit uses a light source to illuminate the region of interest.

16. The method according to claim 9, further comprising: The object's identity is determined by processing circuitry based on two-dimensional image information and three-dimensional depth information; Tracking objects based on two-dimensional image information through processing circuitry; and By processing the circuit in response to critical conditions, the two-dimensional image information is updated with three-dimensional depth information.

17. A three-dimensional sensing system, comprising: The light source is configured to emit light; A two-dimensional sensing area is used to detect visible light reflected from an object; A three-dimensional depth sensing area is used to detect near-infrared light reflected from an object; A superlens covering a two-dimensional sensing area is configured to guide visible light reflected from an object into the two-dimensional sensing area and to guide light emitted by a light source and reflected from the object into a three-dimensional depth sensing area. and The processing circuit is configured as follows: Determine the two-dimensional image information of the object based on the two-dimensional sensing area; The three-dimensional depth information of an object is determined based on the three-dimensional depth sensing area; and Update the object's two-dimensional image information with three-dimensional depth information. The superlens comprises multiple nanostructures configured on a substrate with different geometries and / or arrangements to guide external light of different target wavelength spectra to different parts of the hybrid sensing array.

18. The three-dimensional sensing system according to claim 17, further comprising: The row decoder and column amplifier are stacked below the two-dimensional sensing area and the three-dimensional depth sensing area.

19. The three-dimensional sensing system according to claim 17, wherein, Superlenses are circular, cylindrical, rectangular, or square in shape, and are polarization-independent.

20. The three-dimensional sensing system according to claim 17, wherein, The two-dimensional sensing area includes one or more photodiodes, and the three-dimensional depth sensing area includes one or more single-photon avalanche diodes or avalanche photodiodes.

Images