Image sensors and sensing devices for Extended Reality (XR) applications

Abstract

To provide a sensor module. [Solution] The sensor module comprises a circuit configured to provide a dual sensing layer (400). The dual sensing layer (400) provides a self-aligning RGB imaging function and a SPAD sensing function. The self-alignment is achieved by stacking the sensing layers.

Description

Technical Field

[0001] The present disclosure generally relates to the technical field of extended reality (XR) devices, methods, and systems.

Background Art

[0002] Extended Reality (XR) represents a perception with enhanced sensory, mainly visual, impressions in a multi-dimensional environment where real-world elements and computer-generated virtual elements are combined. It is typically utilized via mobile devices such as smart glasses, smart watches, smartphones, and tablet computers. The term Extended Reality is a common concept for Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

Summary of the Invention

Problems to be Solved by the Invention

[0003] Mobile devices such as smart glasses, smart watches, and smartphones need to highly recognize the surrounding environment so that users can operate extended reality applications. This means using an appropriate number of imaging and sensing devices that can collect different aspects of a scene, such as images suitable for machine vision (i.e., deep neural network processing, automated tasks), images suitable for video streaming and photography, depth maps for scene reconstruction, and other inputs such as acceleration, temperature, and ambient light sensing from inertial units. State-of-the-art commercial devices use multiple sensors (e.g., the world-facing camera of a smartphone) and complex processing to achieve alignment and image fusion.

[0004] Although there are technologies for image sensors for XR devices, generally, improvements in these technologies are desired.

Means for Solving the Problems

[0005] In one embodiment, the disclosure provides a sensor module comprising a circuit configured to provide a dual sensing layer. The dual sensing layer provides a self-aligning RGB imaging function and a single-photon avalanche diode (SPAD) sensing function. The self-alignment is achieved by stacking the sensing layers.

[0006] In another embodiment, the present disclosure provides an electronic device comprising the sensor module and an application processor communicatively connected to the sensor module.

[0007] Other embodiments are described in the dependent claims, the following description and drawings.

[0008] An example embodiment will be described with reference to the attached drawings. [Brief explanation of the drawing]

[0009] [Figure 1] This disclosure shows a smart glasses device equipped with an all-in-one sensor module according to one embodiment of this disclosure. [Figure 2] This provides a more detailed exemplary architecture of an XR system with an all-in-one sensor module for smart glasses devices. [Figure 3] A schematic cross-sectional view shows an example of a self-aligning, single-aperture RGB+SPAD sensor (110 in Figure 2) in an all-in-one sensor module. [Figure 4] A schematic exploded view of a self-aligning, single-aperture RGB+SPAD sensor is shown. [Figure 5] A more detailed cross-sectional view shows an example of a self-aligning, single-aperture RGB+SPAD sensor as an all-in-one sensor module. [Figure 6]A schematic diagram illustrating the operation of the all-in-one sensor module for direct time-of-flight operation mode is shown. [Figure 7a] This shows the ultra-low power always-on (internal loop) operation mode of the all-in-one sensor module. [Figure 7b] This shows the low-power single-sensor XR (external loop) operating mode of the all-in-one sensor module. [Figure 8] This section shows a typical system operation example of the all-in-one sensor module over time, and also shows the power consumption of the all-in-one sensor module over time. [Figure 9] Figure 9a schematically shows the "fully mapped" imaging / sensing state of the all-in-one sensor module, Figure 9b schematically shows the "anchoring" imaging / sensing state of the all-in-one sensor module, and Figure 9c schematically shows the "smart update" imaging / sensing state of the all-in-one sensor module. [Modes for carrying out the invention]

[0010] Referring to Figure 1, a general explanation will be given before a detailed description of the embodiments.

[0011] The embodiment provides a sensor module comprising a circuit configured to provide a dual sensing layer. The dual sensing layer provides a self-aligning RGB imaging function and a SPAD sensing function. The self-alignment is achieved by stacking the sensing layers. Here, RGB sensing refers to imaging the saturation (of colors) using a photodiode, thereby obtaining a measurement of the light intensity at each pixel over a certain period of time. In contrast, SPAD sensing refers to single-photon imaging, thereby obtaining information related to the arrival time and arrival rate of a single photon at each pixel.

[0012] The dual sensing layer described above may be configured to provide self-aligning, single-aperture RGB+SPAD sensing.

[0013] Self-alignment may mean, for example, that the stacked mounting is coaxial and spatially coherent. Thus, a single aperture sensor may be provided by stacking the sensing layers. Self-alignment may provide, for example, a coaxial image plane. The term self-aligning may refer to both spatial and temporal alignment between RGB and SPAD.

[0014] The above circuit may be configured, for example, based on triple-wafer stacking technology. The third wafer houses, for example, readout and processing electronics that support the RGB and SPAD wafers located on the third wafer. However, the embodiments are not limited to triple-wafer stacking. Front-end electronics housed on the RGB and SPAD wafers may also be used.

[0015] SPAD sensing may offer multiple operating modes, for example, from photon counting (PC) to time-of-flight (ToF) distance imaging. The sensor module may enable multiple RGB & EVS operating modes, for example. Furthermore, it may provide fully programmable macropixels of the SPAD array. The macropixels may be configured as small SPAD arrays (e.g., 2x2, 3x3, 4x4 pixels) connected to dedicated processing electronic equipment, for example.

[0016] The dual sensing layer may include, for example, a photodiode layer. In this way, the sensor module may provide, for example, RGB imaging / sensing.

[0017] The dual sensing layer may further include a SPAD layer. The SPAD layer may be, for example, silicon (VIS-NIR) or germanium (SWIR) based. SWIR detection can also be implemented using, for example, InGaAs / InP. In some embodiments, visible light may also be detected by the SPAD layer. The SPAD layer may provide a SPAD distance imaging function. The SPAD layer may enable applications such as, for example, iToF / dToF and / or photon counting (PC).

[0018] As an example of implementation, for example, an RGB wafer may be placed on top to target visible light (spectrum), and a SPAD may be implemented below for exclusive NIR or SWIR spectrum.

[0019] The dual sensing layer may further include a visible light cut filter for removing residual visible light. The visible light cut may, for example, pass either one or both of NIR or SWIR.

[0020] The circuit may further include a dual bandpass filter disposed on top of the dual sensing layer.

[0021] The circuit may further include a readout layer disposed below the dual sensing layer. The readout layer 407 may be realized, for example, as a CMOS readout and processing layer. The connection part of the SPAD may be realized, for example, as a high-density wafer-to-wafer connection (e.g., by hybrid bonding).

[0022] The circuit of the sensor module may further be configured to provide an EVS sensing function. For example, in a self-aligned single aperture RGB+SPAD sensor, an event-based vision sensor (EVS) function may be added at the RGB wafer level or the readout wafer level. EVS sensing may, for example, provide an event-based mode for low-latency and low-power operation.

[0023] The above-described circuit of the sensor module may further provide all-in-one imaging sensing for mobile extended reality applications.

[0024] The above circuit may further provide a programmable laser transmitter. With flexibly programmable illumination, the sensor module can meet all the typical needs of Extended Reality (XR) applications, such as capturing variable-resolution images or reconstructible depth maps. The programmable laser transmitter may have, for example, a laser driver and a VCSEL array. The programmable laser transmitter may be used, for example, for dynamic arbitrary pattern generation.

[0025] The above-described circuit of the sensor module may further provide programmable on-chip control for local autonomous closed-loop control. The on-chip control may be provided, for example, by a programmable on-chip control unit, such as a CPU or NPU capable of selectively optimizing module operation. The on-chip control unit may have, for example, one or more logic, memory, and processors for on-chip closed-loop operation. The sensor module may provide, for example, an all-in-one image sensor architecture capable of acquiring self-consistent, synchronized, and self-aligned images with on-chip processing capabilities for internal closed-loop operation.

[0026] The on-chip control described above may, for example, provide an ultra-low power mode with an internal closed loop.

[0027] The on-chip control described above may be configured, in particular, to control a programmable laser transmitter.

[0028] According to one embodiment, the circuit may provide a single-aperture photodetector with on-chip control and a self-aligning single-aperture RGB+SPAD sensor.

[0029] The above sensor module may further include a light-receiving optical system associated with the above single aperture light detector.

[0030] The embodiment also discloses an electronic device comprising the sensor module described in claim 1 and an application processor communicatively connected to the all-in-one sensor module. The application processor may be, for example, the main processing unit of a smart device such as a smart glasses device or a smartphone.

[0031] The on-chip control of the sensor module may provide a first operating mode in which the application processor is in a sleep state. This first operating mode may be an ultra-low power always-on operating mode. In this ultra-low power always-on operating mode, the on-chip control unit may provide local autonomous closed-loop control, which may provide low-power sensing on RGB and / or low-latency sensing on EVS and / or low-power sensing on SPAD with an optional fully programmable pulsed laser transmitter (TX).

[0032] The on-chip control of the above sensor module may further provide a second operating mode in which the application processor is active. This second operating mode may be, for example, a low-power single-sensor XR (external loop) operating mode. In this second operating mode, the external loop via the application processor may be configured with CPU / NPU control. In this low-power single-sensor XR (external loop) operating mode, fully customizable imaging / sensing on RGB / EVS and / or fully customizable sensing on SPAD with optionally a fully programmable pulsed laser transmitter (TX).

[0033] The embodiments will be described in detail below with reference to the drawings.

[0034] Figure 1 shows a smart glasses device equipped with an all-in-one sensor module. The smart glasses device 200 is configured as a wearable computer worn on the eyes or head to provide an Extended Reality (XR) experience to the user. The smart glasses device 200 has a display and / or projector that overlays information on what the wearer is looking at. The smart glasses device 200 has an all-in-one XR sensor module 101 configured to collect various aspects of the scene in front of the user's eyes, such as images suitable for machine vision (i.e., deep neural network processing, automation tasks), images suitable for video streaming and photography, and depth maps for scene reconstruction. The smart glasses device 200 may also have additional sensors (not shown in Figure 1) that provide other information such as acceleration, temperature, and ambient light detection from an inertial measurement unit (IMU). Based on the information acquired by these sensors, the smart glasses device 200 can function as an augmented reality device that performs pose tracking.

[0035] Figure 2 shows in more detail an exemplary architecture of an XR system for a smart glasses device. The XR system has an all-in-one sensor module. The XR system has a main processing unit (AP: Application processor) 100 that is communicatively connected to the all-in-one sensor module 101. The all-in-one sensor module 101 is configured to capture a scene 102. The self-locating single-aperture RGB+SPAD sensor 110 of the all-in-one sensor module 101 provides self-locating (coaxial and spatially coherent) single-aperture RGB imaging, EVS sensing, and SPAD sensing capabilities. RGB imaging may provide everything from high-quality imaging to low-power, low-resolution sensing. EVS sensing provides an event-based mode for low-latency, low-power operation. SPAD sensing provides multiple modes, from photon counting (PC) to time-of-flight (ToF) distance imaging. The all-in-one sensor module 101 further includes a fully programmable pulsed laser transmitter (TX) 103 equipped with a laser driver 105 and a VCSEL array 106 for dynamic arbitrary pattern generation. A transmitting optical system 111 is positioned in front of the transmitter (TX) 103. The strictly controlled, programmable VCSEL array 106 is driven by a laser diode driver (LDD) 105. The programmable pulsed laser transmitter (TX) 103 is controlled by on-chip control units 107, 108, and 109 for local autonomous closed-loop control. The on-chip control units include logic 107, memory (MEM) 108, and a processor (CPU / NPU) 109 for on-chip closed-loop operation. The self-aligning single-aperture RGB+SPAD sensor 110, along with the CPU / NPU 109, memory (MEM) 108, and logic (107), forms a single-aperture receiver (RX) of the all-in-one sensor module 101, which is comprised of a light-receiving optical system 113.The all-in-one image sensor module 101 enables multiple RGB & EVS operating modes and provides fully programmable macro pixels arranged in a SPAD array.

[0036] The all-in-one image sensor architecture module shown in Figure 2 is capable of acquiring self-aligned, synchronized, and self-situated images with on-chip processing capabilities for internal closed-loop operation (see Figure 7a for details). The sensing unit meets all typical XR needs, namely variable-resolution images and reconstructible depth maps, using flexibly programmable illumination. Self-situation is achieved by stacked mounting of sensing layers for a fully coaxial image plane (see Figure 3), and the ultra-low-power internal closed-loop is achieved by a programmable control unit and either a CPU or NPU, which can selectively optimize module operation.

[0037] Figure 3 shows a schematic cross-sectional view of an example of a self-aligning single-aperture RGB+SPAD sensor (110 in Figure 2) in an all-in-one sensor module. The sensor is constructed based on triple-wafer stacking technology with a dual-sensing layer 400. A dual-band-pass filter (DBPF) 401 for wavelength of interest selection is located on top of the dual-sensing layer 400. The dual-sensing layer 400 has a thin RGB-sensitive layer 404 (photodiode layer, PD) with little to no NIR / SWIR absorption. The RGB-sensitive layer 404 may have, for example, a silicon photodiode or an organic photodiode. The dual-sensing layer 400 further has a visible light cut filter 405 for removing residual visible light. The dual-sensing layer 400 further has a silicon (NIR) or germanium (SWIR) based SPAD layer 406 for iToF / dToF / PC. The SPAD layer 406 may have, for example, a germanium on silicon or a germanium-based single-photon avalanche diode. A readout layer 407 using CMOS technology is located below the dual sensing layer 400. The readout layer 407 may be implemented, for example, as a CMOS readout and processing layer. The SPAD connections may be implemented, for example, as high-density wafer-to-wafer connections (e.g., by hybrid bonding).

[0038] In the self-aligning, single-aperture RGB+SPAD sensor shown in Figure 3, event-based vision sensor (EVS) functionality may be added at the RGB wafer level or the readout wafer level.

[0039] Figure 4 schematically shows an exploded view of a self-aligning, single-aperture RGB+SPAD sensor. An RGB-sensitive layer 404, consisting of red, green, and blue pixels, is stacked on top of a SPAD layer 406. A readout layer 407 is located below the RGB-sensitive layer 404 and the SPAD layer 406.

[0040] An RGB+SPAD sensor may offer the advantage that RGB, EVS, and SPAD data / images are self-aligned and consistent. That is, a one-to-one spatial and temporal correspondence is possible in the intensity, color, and distance data images. Furthermore, the image data may be temporally aligned. For example, the sensor module may acquire both RGB / EVS imaging and SPAD imaging simultaneously and in complete parallel.

[0041] The all-in-one image sensor of this embodiment is particularly suitable for application in mobile devices that require extremely tight power budgets and a thin form factor, and may eliminate the need for numerous separate image sensors or sensors.

[0042] This all-in-one image sensor configuration makes it possible to meet the extremely stringent area (size) and power requirements of sensing devices in XR applications. In particular, the size challenge is solved by providing a substantial number of imaging and sensing devices that can collect different aspects of a scene, such as images suitable for machine vision (i.e., deep neural network processing, automation tasks), images suitable for video streaming and photography, depth maps for scene reconstruction, and other inputs such as acceleration, temperature, and ambient light detection from inertial units.

[0043] The single-module approach of an all-in-one sensor module requires only a slightly larger area than a single sensor, potentially offering significant area and volume advantages. Furthermore, the single-module approach of an all-in-one sensor module requires only one electrical interface for the different sensor types integrated into the module.

[0044] Furthermore, the processing power required to integrate these diverse information sources is amplified by their heterogeneity, different field of view, different controls, and different data formats. Every decision and data must be complete and often redundant, because every decision is made only after being transmitted through power-hungry application / host processing units, ultimately potentially undermining the possibility of achieving effective low-power always-on operation. An all-in-one image sensor meets stringent power requirements without limiting the operation of the most power-hungry sensors (e.g., ToF sensors) and hindering continuous acquisition, nor limiting the streaming of all sensors due to the power consumption of interfaces and data transfer.

[0045] Furthermore, the all-in-one image sensor of this embodiment avoids the additional complexity (mechanical assembly, data processing, additional interfaces) required for calibrating separate sensors (often from different vendors), aligning them spatially and temporally (i.e., data alignment in image space and time), and maintaining this calibration during operation (e.g., camera baseline in smart glasses with a non-rigid frame).

[0046] All-in-one sensor modules can also offer the advantage of increasing the amount of information available to the sensor. This allows for local integrated processing of dToF, RGB, and EVS data, which may enable additional functions such as mapping, situational awareness, SLAM, and interaction with virtual surfaces. In particular, it may enable an approach optimized for machine vision. The sensor can be designed for information quality rather than image quality, which helps to optimize power consumption. All-in-one sensor modules enable devices that perform dynamic and adaptive laser pattern assignment and acquisition without the assistance of other external sensors, for example.

[0047] The all-in-one sensor module can be used in applications such as virtual reality (VR), extended reality (XR), and mixed reality (MR), as well as in lightweight, battery-powered mobile devices like smart glasses, mobile phones, and smartwatches. Applications for which the all-in-one sensor module can be used may include games, navigation, and virtual interfaces.

[0048] Figure 5 shows a more detailed cross-sectional view of an example of a self-aligning single-aperture RGB+SPAD sensor as an all-in-one sensor module. The RGB layer 404 has a plurality of photodiodes (PDs) 307 arranged in a pixel array (313a and / or 313b). The first pixel array 313a is a possible arrangement example of the sensor 304 having an RGB peripheral connection. The second pixel array 313b is a possible arrangement example of the sensor 305 having a high-density pixel-unit or pixel group-unit connection. The SPAD layer 406 has a plurality of single-photon avalanche diodes (SPADs) 308a, 308b, 308c and 308d. In the first sensor example 304, the pixel array 313a is connected to the CMOS readout layer 407 via an RGB peripheral connection 312 (bottom-top connection), providing pixel-unit SPAD connections to the CMOS readout layer 407 for SPADs 308a and 308b. In the second sensor example 305, the pixel array 313b is connected to the CMOS readout layer 407 via a pixel-level RGB connection, and pixel-level SPAD connections to the CMOS readout layer 407 are provided for SPADs 308c and 308d. The CMOS readout layer 407 has a readout circuit 309 arranged to read data obtained from the photodiodes 307 of the pixel arrays 313a and / or 313b and from SPADs 308a, 308b, 308c and 308d.

[0049] The embodiment in Figure 5 has two alternative configurations 304 and 305 for the sensor module, which are typically applied as alternatives and are not necessarily implemented in the same sensor. According to a further embodiment, the sensor module has only configuration 304. According to another embodiment, the sensor module has only configuration 305.

[0050] The I / O pad 306 is located on the RGB layer 404 and connects the sensor's internal signals to, for example, external pins on the chip package. The conductive through-chip connector 310 is located on the SPAD layer 404, and metal connectors are located on the RGB layer 404, SPAD layer 406, and CMOS layer 407 to transmit signals from the CMOS readout layer circuit 309 to the I / O pad 306. The vertical connections (of the RGB layer's photodiodes, I / O pads, etc.) may be implemented, for example, by through-silicon via technology (TSV) or thermo-compression bonding (TCB) technology.

[0051] Figure 6 schematically shows an exemplary timing diagram of direct time-of-flight operation for depth sensing of an all-in-one sensor module in direct time-of-flight operation mode. The VCSEL array of the all-in-one sensor module is selectively driven by the control unit, and only the selected VCSELs are activated. In time interval A, a single VCSEL in the array (specifically the upper left VCSEL of the array) is activated and driven periodically at a high light intensity. In time interval A, there are three strong peaks in energy consumption associated with this VCSEL drive. In the following time interval B, another single VCSEL in the array is activated and driven periodically at a low light intensity. In time interval B, there are three strong peaks in energy consumption associated with this VCSEL drive. Since the pixel is driven at a moderate light intensity, the energy consumption is less than in time interval A, and the peak intensity is also weaker than in time interval A. In the following time interval C, none of the VCSELs in the array are activated. Therefore, there are no peaks in energy consumption in time interval C. During the subsequent time interval D, the other two VCSELs in the array are activated and periodically driven at a moderate light intensity. During time interval D, there are six peaks in the energy consumption associated with the periodic driving of these two VCSELs. The SPAD of the self-positioning single-aperture RGB+SPAD sensor 110 detects the light emitted from the VCSELs during time intervals A, B, C, and D.

[0052] Please note that the sequence in Figure 6 is merely an example. The sequence and energy can be optimized in various ways, for example, to acquire the necessary information with the minimum power consumption, or to meet specific accuracy requirements with the minimum necessary power.

[0053] Figures 7a and 7b show two alternative examples of the sensor-enabled closed-loop operating modes of the all-in-one sensor module.

[0054] Figure 7a shows the ultra-low power always-on (internal loop) operating mode of the all-in-one sensor module. In this ultra-low power always-on (internal loop) operating mode, the application processor 100 (the main processor of the wearable device) is in a sleep state. The on-chip control unit provides local autonomous closed-loop control. The internal control loop of this self-locating single-aperture RGB+SPAD sensor 110 is configured via the all-in-one sensor module's local CPU / NPU 109. In this ultra-low power always-on (internal loop) operating mode, low-power sensing in RGB and / or low-latency sensing in EVS and / or low-power sensing in SPAD with the optional use of a fully programmable pulsed laser transmitter (TX) may be provided. This operating mode enables, for example, high-speed and low-cost proximity processing (e.g., presence detection, gesture recognition, etc.). The sensor's all-in-one sensor module functions as a smart trigger, performing decisions and responses (e.g., AP activation, higher-resolution image acquisition, etc.). No data transfer to the application processor 100 occurs.

[0055] The internal control loop of the self-aligning single-aperture RGB+SPAD sensor 110 may use memory 108 and logic 107 in addition to the CPU / NPU 109 (resulting in a system-on-chip configuration that does not require external memory or logic). This enables operation without accessing external, power-hungry memory or logic. Memory 108 can store either CPU / NPU programs or algorithms and data.

[0056] The advantage of the ultra-low power always-on (internal loop) operation mode is that no data stream output to the application processor is required while the loop is closed internally. This enables ultra-low power operation that is not achievable with isolated sensors.

[0057] Local processing and feedback within the all-in-one sensor module enable intelligent decision-making aimed at reducing power consumption by dynamically allocating laser power spatially and temporally without the need for data stream output.

[0058] Figure 7b shows the low-power single-sensor XR (external loop) operating mode of the all-in-one sensor module. In this low-power single-sensor XR (external loop) operating mode, the application processor 100 (the main processor of the wearable device) is activated. An external loop is configured via the application processor 100 with CPU / NPU control. This low-power single-sensor XR (external loop) operating mode enables fully customizable imaging / sensing on RGB / EVS and / or fully customizable sensing on SPAD with the optional use of a fully programmable pulsed laser transmitter (TX). This operating mode enables complex external processing (e.g., semantic extraction, scene segmentation, etc.) and sensor programming (e.g., high-resolution imaging, region of interest ToF sampling, etc.).

[0059] With respect to Figures 7a and 7b, programmability and flexibility may be obtained by providing different operating modes as described above. For example, an inner loop and an outer loop can be combined to minimize power consumption and maximize performance.

[0060] Figure 8 shows a typical system operation example of the all-in-one sensor module over time, and the power consumption of the all-in-one sensor module over time. Operation starts at time t0 in ultra-low power always-on operation mode 800. This operation mode corresponds to the operation mode described in Figure 7a above. In this ultra-low power always-on mode, the internal control loop of the self-aligning single-aperture RGB+SPAD sensor 110 is configured via the all-in-one sensor module's local CPU / NPU 109. From time t0 to time t1, the all-in-one sensor module operates at low resolution (Low Res) and functions as a smart trigger. At time t1, the internal control loop aligns the event (e.g., by hand recognition), and this event triggers the acquisition and streaming of a higher-resolution image (e.g., medium-resolution grayscale or RGB). From time t1 onward, tasks such as mapping, situational awareness, SLAM, and interaction with virtual surfaces are performed by the application processor (AP) 100 (the main processor of the wearable device) in different imaging / sensing states (RGB sensing / imaging and / or SPAD: distance imaging). Between t1 and t7, the all-in-one sensor can continue to operate in the mixed state shown in Figures 7a and 7b. In this state, the internal loop continues to optimize power and performance, while the external loop enables frame acquisition by the AP and more complex application-based reconfiguration. From time t1 to time t2, the sensor is in the "full mapping" imaging / sensing state (see also Figure 9a and its corresponding description), in which medium-resolution images are acquired with all dots (generated by the VCSEL array 106) active, and higher-resolution images are acquired very rarely. This results in high power consumption. At time t2, the sensor transitions to the "anchoring" imaging / sensing state (see also Figure 9b and its corresponding description), in which only a few dots are active very frequently. During anchoring, the power consumption of the all-in-one sensor module is significantly reduced.At time t3, the all-in-one sensor module transitions to the "smart update" imaging / sensing state (see also Figure 9c and its corresponding description). In this state, some dots are occasionally active. At time t4, the all-in-one sensor module returns to the "anchoring" state. In this state, only a few dots are very frequently active. The power consumption of the all-in-one sensor module is again significantly reduced. At time t5, the all-in-one sensor module transitions to the "full mapping" state. In this state, higher-resolution images are acquired very rarely at a medium resolution where all dots are active. This significantly increases power consumption. At time t6, the all-in-one sensor module returns to the "anchoring" state. In this state, only a few dots are very frequently active. The power consumption of the all-in-one sensor module is again significantly reduced. At time t7, the application processor (AP) 100 may decide to acquire high-resolution (High res) images based on the results and application requirements. From time t7 to time t8, the all-in-one sensor module operates in the low-power single-sensor XR (external loop) mode described in Figure 7b above. An external loop is configured via the application processor 100 with CPU / NPU control. For complex functions such as streaming, semantic analysis, and photo processing (e.g., person recognition), external processing by the application processor is invoked. When the application processor is actively operating for a high-load task, power consumption increases significantly (not shown in Figure 8).

[0061] The above example states that "full mapping" occurs extremely rarely. "Full mapping" can only occur, for example, during internal / external loop switching, or when the user points to a completely new part of the environment that has not been mapped at all by the application processor AP (such as time t5 in Figure 8). This may occur in a few percent of cases, perhaps 1-2 percent of the time. However, it should be noted that the embodiment is not limited to the extremely rare occurrence of "full mapping." This is merely an example.

[0062] Figure 9a schematically shows the "fully mapped" imaging / sensing state of the all-in-one sensor module. In this "fully mapped" state, images are acquired very rarely at a medium resolution where all dots are active. That is, all VCSELs in the VCSEL array 106 are active. This results in high power consumption.

[0063] Figure 9b schematically shows the “anchoring” imaging / sensing state of the all-in-one sensor module. In this “anchoring” state, only a small number of dots are active at most. During anchoring, the power consumption of the all-in-one sensor module is significantly reduced. During anchoring, only relevant VCSELs (e.g., for illuminating and tracking relevant areas for SLAM (edges, corners, surface textures, etc.)) are active. Dots follow relevant features. For example, image analysis may be performed using images from the RGB sensor to recognize which areas are particularly relevant to a face recognition task. The programmable pulsed laser transmitter (TX) 103 triggers only the VCSELs that have been determined to be relevant by the image analysis. In addition, only the pixels of the SPAD where incident light from the triggered VCSELs is expected is activated. This saves power.

[0064] Figure 9c schematically shows the “smart update” imaging / sensing state of the all-in-one sensor module. In “smart update” state, some dots are activated from time to time. This is particularly useful when the scene moves (for example, when the camera moves). Newly entering areas in the image are assumed to be areas of particular interest and are covered with dots.

[0065] Please note that the above description is merely an example configuration. Alternative configurations using additional units, sensors, etc., are also possible. Furthermore, please note that the system's unit division is for illustrative purposes only, and this disclosure is not limited to specific functional divisions within specific units.

[0066] All units and entities described herein and required by the appended claims can be implemented, for example, as integrated circuit logic in an FPGA on a chip, unless otherwise specified, and the functionality provided by such units and entities can be implemented by software, unless otherwise specified.

[0067] Naturally, the embodiments of the above disclosure are envisioned as embodiments of the present disclosure, insofar as they are implemented at least in part using software-controlled data processing equipment, computer programs that provide such software control and transmission media, storage media, and other media that provide such computer programs.

[0068] Furthermore, this technology can also be configured as described below.

[0069] (1) comprising a circuit configured to provide a dual sensing layer (400), The above dual sensing layer (400) provides self-aligning RGB imaging and SPAD sensing functions. The above self-positioning is achieved by stacking the sensing layer. Sensor module. (2) The sensor module described in (1), The dual sensing layer (400) has a photodiode layer (404). Sensor module. (3) The sensor module described in (2), The above photodiode layer (404) is configured to detect visible light (RGB). Sensor module. (4) A sensor module as described in any of (1) to (3), The dual sensing layer (400) has a SPAD layer (406) for detecting NIR or SWIR spectrum. Sensor module. (5) The sensor module described in (4), The above SPAD layer (406) is configured to detect NIR or SWIR spectra. Sensor module. (6) A sensor module as described in any of (1) to (5), The dual sensing layer (400) further includes a visible light cut filter (405) for removing residual visible light. Sensor module. (7) A sensor module as described in any of (1) to (6), The above circuit has a dual bandpass filter (401) placed on top of the dual sensing layer (400). Sensor module. (8) A sensor module as described in any of (1) to (7), The above circuit has a readout layer (407) located below the dual sensing layer (400). Sensor module. (9) A sensor module as described in any of (1) to (8), The above circuit is configured to further provide EVS sensing capabilities. Sensor module. (10) The sensor module described in (9), The above EVS sensing provides an event-based mode for low-latency, low-power operation. Sensor module. (11) A sensor module as described in any of (1) to (10), The above circuit provides all-in-one imaging and sensing for mobile extended reality applications. Sensor module. (12) A sensor module as described in any of (1) to (11), The above circuit further provides a programmable laser transmitter (103). Sensor module. (13) A sensor module as described in any of (1) to (12), The above circuit further provides programmable on-chip control (107, 108, 109) for local autonomous closed-loop control. Sensor module. (14) The sensor module described in (13), The above on-chip controls (107, 108, 109) provide an ultra-low power mode with an internal closed loop. Sensor module. (15) A sensor module as described in (13) or (14), The above on-chip controls (107, 108, 109) are configured to control a programmable laser transmitter (103). Sensor module. (16) A sensor module as described in any of (1) to (15), The above circuit provides a single aperture photodetector (RX) with on-chip control (107, 108, 109) and a self-aligning single aperture RGB+SPAD sensor (110). Sensor module. (17) The sensor module described in (16), The self-aligning type mentioned above refers to both spatial and temporal alignment between RGB and SPAD. Sensor module. (18) A sensor module as described in any of (1) to (17), An application processor (100) is connected to the above sensor module (101) in a way that allows communication between them. Equipped with electronic equipment. (19) An electronic device as described in (18), The on-chip control (107, 108, 109) of the above sensor module provides a first operating mode in which the above application processor (100) is in a sleep state. electronic equipment. (20) An electronic device as described in (18) or (19), The on-chip control (107, 108, 109) of the above sensor module provides a second operating mode in which the above application processor (100) is active. electronic equipment. [Explanation of Symbols]

[0070] 101...Sensor module 103... Programmable laser transmitter 107, 108, 109…On-chip control 110...Self-aligning single-aperture RGB+SPAD sensor 400... Dual sensing layer 401…Dual Bandpass Filter 404...Photodiode layer 405…Visible light cut filter 406…SPAD layer 407...Read layer

Claims

1. It comprises a circuit configured to provide a dual sensing layer, The dual sensing layer provides a self-localizing RGB imaging function and a SPAD sensing function. The aforementioned self-alignment is achieved by stacking and mounting the sensing layer. Sensor module.

2. A sensor module according to claim 1, The dual sensing layer has a photodiode layer. Sensor module.

3. A sensor module according to claim 2, The photodiode layer is configured to detect visible light. Sensor module.

4. A sensor module according to claim 1, The dual sensing layer has a SPAD layer for detecting NIR or SWIR spectrum. Sensor module.

5. A sensor module according to claim 4, The SPAD layer is configured to detect NIR or SWIR spectra. Sensor module.

6. A sensor module according to claim 1, The dual sensing layer further includes a visible light cut filter for removing residual visible light. Sensor module.

7. A sensor module according to claim 1, The circuit has a dual bandpass filter positioned on the dual sensing layer. Sensor module.

8. A sensor module according to claim 1, The circuit has a readout layer located below the dual sensing layer. Sensor module.

9. A sensor module according to claim 1, The circuit is configured to further provide EVS sensing functionality. Sensor module.

10. A sensor module according to claim 9, The aforementioned EVS sensing provides an event-based mode for low-latency, low-power operation. Sensor module.

11. A sensor module according to claim 1, The circuit provides all-in-one imaging and sensing for mobile extended reality applications. Sensor module.

12. A sensor module according to claim 1, The circuit further provides a programmable laser transmitter. Sensor module.

13. A sensor module according to claim 1, The circuit further provides programmable on-chip control for local autonomous closed-loop control. Sensor module.

14. A sensor module according to claim 13, The on-chip control provides an ultra-low power mode with an internal closed loop. Sensor module.

15. A sensor module according to claim 13, The on-chip control is configured to control a programmable laser transmitter. Sensor module.

16. A sensor module according to claim 1, The circuit provides a single-aperture photodetector with on-chip control and a self-aligning single-aperture RGB+SPAD sensor. Sensor module.

17. A sensor module according to claim 16, The aforementioned self-aligning type refers to both spatial and temporal alignment between RGB and SPAD. Sensor module.

18. The sensor module according to claim 1, The application processor is connected to the aforementioned sensor module in a communicative manner. Equipped with electronic equipment.

19. The electronic device according to claim 18, The on-chip control of the sensor module provides a first operating mode in which the application processor is in a sleep state. electronic equipment.

20. The electronic device according to claim 18, The on-chip control of the sensor module provides a second operating mode in which the application processor is active. electronic equipment.

Images