High-resolution time-of-flight depth imaging

Abstract

Techniques and systems for high-resolution time-of-flight (ToF) depth imaging are provided. In some examples, the apparatus includes a projection system including one or more light emitting devices, each light emitting device configured to illuminate at least a portion of an entire field of view (FOV) of the projection system. The entire FOV includes a plurality of FOV portions. The apparatus also includes a receiving system including a sensor configured to sequentially capture a plurality of images based on a plurality of illumination reflections corresponding to light emitted by the one or more light emitting devices. Each image of the plurality of images corresponds to one of the plurality of FOV portions. An image resolution associated with each image corresponds to a full resolution of the sensor. The apparatus further includes a processor configured to generate a high-resolution depth map associated with the entire FOV using the plurality of images.

Description

[Technical field]

[0001] The present disclosure relates generally to time-of-flight (ToF) sensing. In some examples, aspects of the present disclosure relate to increasing the image resolution of a depth map generated based on ToF data by continuously scanning portions of a field of view (FOV) of an imaging system. [Background technology]

[0002] Image sensors are commonly integrated into a wide range of electronic devices, such as cameras, mobile phones, autonomous systems (e.g., autonomous drones, cars, robots, etc.), smart wearables, extended reality (e.g., augmented reality, virtual reality, mixed reality) devices, and many other devices. Image sensors allow users to capture videos and images from any electronic device equipped with an image sensor. Videos and images may be captured for entertainment use, professional filming, surveillance, and automation, among other applications. Videos and images captured by image sensors can be manipulated in various ways to enhance the quality of the video or image and to create several artistic effects.

[0003] In some cases, the optical signals and image data captured by the image sensor may be analyzed to identify certain characteristics about the image data and / or the scene captured by the image data, which can then be used to modify the captured image data or perform various tasks. For example, the optical signals and / or image data may be analyzed to estimate the distance of the scene captured by the image data. Estimating distance information may be useful for various applications such as three-dimensional (3D) photography, extended reality experiences, object scanning, autonomous vehicle navigation, earth topography, computer vision systems, facial recognition systems, robotics, games, and for creating various artistic effects such as blurring and blurring effects (e.g., defocus effects). However, estimating distance information with sufficient resolution and / or accuracy may be prohibitively power and computationally intensive. Summary of the Invention [Means for solving the problem]

[0004] Systems and techniques that can be implemented to perform high-resolution time-of-flight (ToF) depth imaging are described herein. According to at least one example, an apparatus for high-resolution ToF depth imaging is provided. The exemplary apparatus can include a projection system including one or more light-emitting devices, each of the one or more light-emitting devices configured to illuminate at least a portion of an entire field of view (FOV) of the projection system. The entire FOV can include multiple FOV portions. The apparatus can also include a receiving system including a sensor configured to sequentially capture multiple images based on multiple illumination reflections corresponding to light emitted by the one or more light-emitting devices. Each image of the multiple images corresponds to one of the multiple FOV portions. Furthermore, an image resolution associated with each image of the multiple images corresponds to a full resolution of the sensor. The apparatus can also include a memory (or multiple memories) and a processor or multiple processors (e.g., implemented in a circuit configuration) coupled to the memory (or multiple memories). The processor (or multiple processors) can be configured to generate a high-resolution depth map associated with the entire FOV using the multiple images.

[0005] In another example, an apparatus may include means for illuminating a plurality of FOV portions of an entire field of view (FOV) of the projection system, where the means for illuminating is configured to sequentially illuminate at least one portion of the entire FOV; means for sequentially capturing a plurality of images based on a plurality of illumination reflections corresponding to light emitted by the means for illuminating, where each image of the plurality of images corresponds to one of the plurality of FOV portions and an image resolution associated with each image of the plurality of images corresponds to a full resolution of the means for receiving; and means for generating a high-resolution depth map associated with the entire FOV using the plurality of images.

[0006] In another example, a method for high-resolution ToF depth imaging is provided. The exemplary method may include illuminating a plurality of FOV portions of an entire FOV of the projection system using one or more light-emitting devices of the projection system, each light-emitting device of the one or more light-emitting devices being configured to illuminate at least one portion of the entire FOV. The method may also include sequentially capturing a plurality of images by a sensor of the receiving system based on a plurality of illumination reflections corresponding to light emitted by the one or more light-emitting devices. Each image of the plurality of images corresponds to one of the plurality of FOV portions. Furthermore, an image resolution associated with each image of the plurality of images corresponds to a full resolution of the sensor. The method may also include generating a high-resolution depth map associated with the entire FOV using the plurality of images.

[0007] In another example, a non-transitory computer-readable medium for high-resolution ToF depth imaging is provided. The exemplary non-transitory computer-readable medium can store instructions that, when executed by one or more processors, cause the one or more processors to: illuminate a plurality of FOV portions of an entire field of view (FOV) of the projection system using one or more light-emitting devices of a projection system, where each light-emitting device of the one or more light-emitting devices is configured to illuminate at least one portion of the entire FOV; sequentially capture a plurality of images using a sensor of a receiving system based on a plurality of illumination reflections corresponding to light emitted by the one or more light-emitting devices, where each image of the plurality of images corresponds to one of the plurality of FOV portions, and an image resolution associated with each image of the plurality of images corresponds to a full resolution of the sensor; and generate a high-resolution depth map associated with the entire FOV using the plurality of images.

[0008] In some aspects, the above-described method, apparatus, and computer-readable medium may include illuminating each of the plurality of FOV portions in a sequential illumination order, the sequential illumination order including illuminating a single FOV portion at a time, receiving each illumination reflection of the plurality of illumination reflections in a sequential reception order corresponding to the sequential illumination order, and generating each image of the plurality of images based on each illumination reflection of the plurality of illumination reflections. In some examples, illuminating each of the plurality of FOV portions in a sequential illumination order may include illuminating a first FOV portion of the plurality of FOV portions, receiving a first illumination reflection corresponding to the first FOV portion, and after receiving the first illumination reflection, illuminating a second FOV portion of the plurality of FOV portions, and receiving a second illumination reflection corresponding to the second FOV portion.

[0009] In some embodiments, the one or more light emitting devices may include a plurality of light emitting devices. In some cases, the projection system may include a plurality of projection lenses, each projection lens of the plurality of projection lenses configured to project light emitted by one of the plurality of light emitting devices toward a different FOV portion of the plurality of FOV portions. In some cases, each projection lens of the plurality of projection lenses may be located above the plurality of light emitting devices. In some examples, the projection system includes one or more diffusers located relative to the plurality of projection lenses. For example, the one or more diffusers can be configured to diffuse the light emitted by the plurality of light emitting devices. In some examples, the projection system includes a segmented prism array. For example, the segmented prism array can be configured to direct light to each FOV portion of the plurality of FOV portions.

[0010] In some embodiments, the one or more light emitting devices may include a single light emitting device. In some cases, the projection system may include a scanning mirror configured to project the light emitted by the single light emitting device toward different FOV portions of the multiple FOV portions when oriented in different orientations. In one example, the scanning mirror may be a microelectromechanical system (MEMS) mirror. In some embodiments, the scanning mirror may be located above the single light emitting device. In some cases, each of the different orientations of the scanning mirror may correspond to a different orientation angle between the scanning mirror and a surface of the single light emitting device. In some examples, the projection system includes one or more diffusers located relative to the scanning mirror. For example, the one or more diffusers may be configured to diffuse the light emitted by the single light emitting device.

[0011] In some aspects, the receiving system includes an array of imaging lenses, each imaging lens of the array of imaging lenses can be configured to project, onto the sensor, light associated with a different portion of the scene corresponding to a respective FOV portion of the multiple FOV portions.

[0012] In some aspects, the methods, apparatus, and computer-readable media described above may include a filter located above the sensor configured to transmit light having a frequency corresponding to a frequency of light emitted by the one or more light emitting devices.

[0013] In some aspects, the above-described methods, apparatus, and computer-readable media may include synchronizing the projection system and the receiving system based at least in part on sending a first control signal to the projection system instructing the projection system to illuminate a particular FOV portion of the plurality of FOV portions and sending a second control signal to the receiving system instructing the receiving system to associate the illumination reflection received by the sensor with the particular FOV portion, wherein the first control signal and the second control signal are time synchronized.

[0014] In some aspects, a first FOV portion of the plurality of FOV portions may overlap at least a second FOV portion of the plurality of FOV portions.

[0015] In some aspects, generating the enhanced resolution depth map associated with the entire FOV may include generating a plurality of partial distance measurements, each of the plurality of partial distance measurements corresponding to one of the plurality of illumination reflections, and combining the plurality of partial distance measurements. In some examples, an image resolution of the enhanced resolution depth map may correspond to a maximum resolution of the sensor multiplied by the number of the plurality of FOV portions.

[0016] In some aspects, each of the above-mentioned devices is or includes a camera, a mobile device (e.g., a mobile phone, i.e., a so-called "smartphone" or other mobile device), a smart wearable device, an extended reality device (e.g., a virtual reality (VR), an augmented reality (AR), or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, a self-driving car, or other device. In some aspects, the device includes a camera or multiple cameras for capturing one or more videos and / or images. In some aspects, the device further includes a display for displaying one or more videos and / or images. In some aspects, the above-mentioned devices include one or more sensors.

[0017] This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter, which should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0018] The above, together with other features and examples, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0019] Illustrative examples of the present application are described in detail below with reference to the following drawings: [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a simplified block diagram illustrating an example image processing system for time-of-flight (ToF) signal processing, in accordance with some examples of the present disclosure. [Figure 2A] FIG. 1 is a simplified block diagram illustrating an example of a direct ToF sensing procedure, according to some examples of the present disclosure. [Figure 2B] FIG. 1 is a simplified block diagram illustrating an example of an indirect ToF sensing procedure, according to some examples of the present disclosure. [Diagram 3] 1A-1C illustrate example techniques for measuring the phase angle of an optical signal, in accordance with some examples of the present disclosure. [Figure 4] FIG. 1 illustrates a continuous wave method for ToF sensing, according to some examples of the present disclosure. [Diagram 5] FIG. 1 is a simplified block diagram illustrating an example image processing system for ToF signal processing, in accordance with some examples of the present disclosure. [Figure 6A] A diagram showing an example field of view (FOV) of an image processing system for ToF signal processing, according to some examples of the present disclosure. [Figure 6B] 1 is a diagram of an example portion of the FOV of an image processing system, in accordance with some examples of the present disclosure. [Figure 7] 1 is a diagram of an exemplary device for illuminating a portion of the FOV of an image processing system, in accordance with some examples of the present disclosure. [Figure 8] 1 is a diagram of another exemplary device for illuminating a portion of the FOV of an image processing system, in accordance with some examples of the present disclosure. [Figure 9A] 1 is a diagram of an exemplary device for receiving illumination reflections of a portion of the FOV of an image processing system, in accordance with some examples of the present disclosure. [Figure 9B] FIG. 2 illustrates an exemplary continuous ToF image capture process, according to some examples of the present disclosure. [Figure 9C] FIG. 2 illustrates an exemplary continuous ToF image capture process, according to some examples of the present disclosure. [Figure 10A] FIG. 1 illustrates an example system for synchronized ToF sensing, in accordance with some examples of the present disclosure. [Figure 10B] FIG. 1 illustrates an example system for synchronized ToF sensing, in accordance with some examples of the present disclosure. [Figure 11] 1 is a flow diagram illustrating an example process for high-resolution ToF depth imaging, according to some examples. [Figure 12] FIG. 1 illustrates an example of a system for implementing certain aspects described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Some aspects and examples of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and examples may be applied independently, and some of them may be applied in combination. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the subject matter of the present application. However, it will be apparent that various examples may be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0022] The following description provides illustrative examples only and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description is intended to provide those skilled in the art with an enabling description for implementing the illustrative examples. It should be understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0023] Image sensors are commonly integrated into a wide variety of electronic devices, such as cameras, mobile phones, autonomous systems (e.g., autonomous drones, cars, robots, etc.), extended reality (e.g., augmented reality, virtual reality, mixed reality) devices, Internet of Things (IoT) devices, smart wearables, and many other devices. Video and image recording capabilities are becoming more prevalent as more and more electronic devices are equipped with such image sensors. Furthermore, the image processing capabilities of electronic devices are constantly improving, allowing the image data to be manipulated to produce higher quality videos and images, produce a variety of artistic effects, and implement the image data in various applications. For example, the optical signals and image data captured by the sensor can be analyzed to identify certain characteristics about the image data and / or the scene captured by the image data. This information can then be used to modify the captured image data or perform various tasks. For example, the optical signals and / or image data can be analyzed to estimate the distance of an object in the scene captured by the image data.

[0024] A time-of-flight (ToF) sensor system can use light (e.g., infrared (IR) light, near infrared (NIR) light, and / or other light) to determine depth and / or distance information about a target (e.g., a surrounding / nearby scene, one or more surrounding / nearby objects, etc.). The ToF sensor system can include a light emitter configured to emit a light signal toward the target, which can strike the target and reflect back to the ToF sensor system. The ToF sensor system can also include a sensor configured to detect and / or measure the returning / reflected light, which can then be used to determine depth and / or distance information about the target. The distance of the target relative to the ToF sensor system can be used to perform depth mapping. The distance of the target can be calculated through direct ToF or indirect ToF.

[0025] The resolution (e.g., image resolution) of some ToF sensor systems may correspond to the number of reflected light signals that the ToF sensor system can receive and process at one time. For example, the image resolution of a ToF sensor system may correspond to the number of pixels (e.g., photosensitive elements) of the ToF sensor. In many cases, ToF sensor technologies use complex hardware and / or extensive processing power. As a result, increasing the resolution (e.g., number of pixels) of a ToF sensor may be impractical and / or impossible (e.g., cost-prohibitive). For example, some existing ToF systems are limited to a resolution of 320×240 pixels or less. Such low resolution may be insufficient for various types of imaging and sensing applications, such as long-range imaging, wide field-of-view (FOV) imaging, autonomous vehicle navigation, machine vision systems, face recognition systems, digital measurement systems, gaming, robotics, and the like, among others. Therefore, it would be beneficial to develop improved ToF systems and techniques for efficiently capturing high-resolution image depth information.

[0026] This disclosure describes systems, apparatus, methods, and computer-readable media (collectively referred to as "systems and techniques") for providing high-resolution ToF depth imaging. The systems and techniques described herein enable a ToF system to generate a depth map (e.g., a depth image map) at a resolution that exceeds the full (e.g., maximum) resolution of the ToF sensor of the ToF system. Furthermore, the ToF system can generate a high-resolution depth map without incurring significant delays in processing time. To generate a high-resolution depth map, the ToF system can capture depth information associated with several portions of the ToF system's overall field of view (FOV). In some cases, the ToF system can sequentially capture depth information associated with individual portions (referred to as FOV portions) of the overall FOV. For example, the ToF system can "scan" through the FOV portions. The ToF system can use a single ToF sensor (e.g., the same ToF sensor) to scan each FOV portion. The individual FOV portions can correspond (or approximately correspond) to the FOV of the single ToF sensor. The ToF system can generate a depth map corresponding to the entire FOV by combining the depth information obtained from each FOV portion. Thus, the resolution (e.g., image resolution) of the depth map of the entire FOV corresponds to the resolution of a single ToF sensor multiplied by the number of FOV portions scanned. Furthermore, because each FOV portion is smaller than the entire FOV, the light projected by the ToF device to each FOV portion is highly condensed (e.g., compared to the condensation of light projected by a ToF system that immediately illuminates the entire FOV). A corresponding result of this increased illumination is a reduction in exposure time (e.g., the amount of time it takes to obtain an accurate distance measurement), thereby enabling the ToF system to maintain a high frame rate while generating or updating the depth map. For example, if the ToF system divides the FOV into x sub-FOVs, the ToF system can generate a depth map with x times higher resolution with the same latency as a ToF system that does not continuously scan the FOV portions.

[0027] In some examples, the ToF system may include a projector system configured to sequentially illuminate individual FOV portions. The projector system may include one or more light emitting devices. In one example, the light emitting device may include a plurality of vertical cavity surface emitting lasers (VCSELs) arranged in a VCSEL array. In some cases, each VCSEL in the VCSEL array may emit light in a narrowband near infrared (NIR) light. The projector system may also include one or more light directing mechanisms (e.g., one or more projection lenses, a prism, a combination of a projection lens and one or more prisms, and / or a scanning mirror) that appropriately direct the light emitted by the light emitting device.

[0028] In one example, the projector system may include multiple light emitting devices (e.g., multiple VCSEL arrays), each configured to illuminate a particular FOV portion of the overall FOV. For example, the number of light emitting devices may be equal to the number of FOV portions into which the FOV is divided. Each light emitting device may be selectively and / or individually activated (e.g., such that at any one time, only a single light emitting device is emitting light toward a respective FOV portion). In such an example, the one or more light directing mechanisms may include a lens array of multiple projection lenses and / or a prism (or multiple prisms in some cases). For example, each projection lens of the lens array may correspond to a respective light emitting device. The multiple projection lenses of the lens array may be positioned (e.g., held at a particular angle) to ensure that light emitted by each light emitting device illuminates the appropriate FOV portion. Using multiple light emitting devices in combination with a lens array and / or a prism (or multiple prisms) may ensure that no mechanical movement is required to capture distance measurements for the entire FOV.

[0029] In another example, the projector system may include a single light emitting device (e.g., a single VCSEL array) configured to selectively illuminate individual FOV portions. In such an example, the one or more light directing mechanisms may include a scanning mirror (e.g., a microelectromechanical system (MEMS) mirror). The orientation of the scanning mirror may be periodically adjusted (e.g., moved to a particular angle) to sequentially reflect the light emitted by the light emitting device to illuminate each FOV portion. Although using the scanning mirror may involve mechanical movement of the ToF system, mechanical fatigue of the scanning mirror may be negligible. Thus, the mechanical movement of the scanning mirror may not degrade the performance of the projector system over the lifetime of the scanning mirror. Using the scanning mirror may enable only a single light emitting device to be used by the ToF system.

[0030] The ToF system can generate a complete depth map of the entire FOV by combining distance measurements corresponding to individual FOV portions of the entire FOV. For example, the ToF system can "stitch" frames corresponding to individual FOV portions together. The frames can be combined as the frames are captured or after all of the frames have been captured. In some cases, frames corresponding to adjacent FOV portions may include a predetermined amount of overlap (e.g., a predetermined number of overlapping pixels). The overlap may enable the ToF system to efficiently and / or seamlessly stitch together the individual frames into a cohesive depth map of the entire FOV.

[0031] In some cases, the ToF system may synchronize the projector system and the ToF sensor according to a synchronization scheme. By implementing synchronization, the ToF system may ensure that the ToF sensor determines a distance measurement associated with a particular FOV portion when that FOV portion is illuminated. In one example, the ToF system may include a controller that sends a synchronization signal to the projector system. If the projector system includes multiple light emitting devices (e.g., multiple VCSEL arrays), the synchronization signal may indicate which light emitting device should currently be turned on. If the projector system includes a scanning mirror, the synchronization signal may indicate a particular angle at which the scanning mirror should be oriented. Additionally, the controller may send a synchronization signal to the ToF sensor. This synchronization signal may indicate to the sensor which FOV portion is currently illuminated by the projector system. Based on the synchronization signal, the ToF sensor may accurately label distance measurements associated with individual FOV portions when the FOV portions are illuminated consecutively.

[0032] Further details regarding the high-resolution ToF system are provided herein with respect to various figures. FIG. 1 illustrates an exemplary image processing system 100 for time-of-flight (ToF) signal processing. In this illustrative example, the image processing system 100 may include a ToF system 102, an image sensor 104, storage 106, and an application processor 110. In some examples, the image processing system 100 may optionally include other computing components 108, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and / or an image signal processor (ISP), which the image processing system 100 may use to perform one or more of the operations / functionality described herein with respect to the application processor 110. In some cases, the application processor 110 and / or other computing components 108 may implement a ToF engine 130, an image processing engine 134, and / or a rendering engine 136.

[0033] It should be noted that in some examples, the application processor 110 and / or other computational components 108 may also implement one or more computational engines not shown in FIG. 1. The ToF engine 130, the image processing engine 134, and the rendering engine 136 are provided herein for illustrative and descriptive purposes, and other possible computational engines are not provided for simplicity. Also, for illustrative and descriptive purposes, the ToF engine 130, the image processing engine 134, the rendering engine 136, and their various disclosed operations are described herein as being implemented by the application processor 110. However, one skilled in the art will recognize that in other examples, the ToF engine 130, the image processing engine 134, the rendering engine 136, and / or their various disclosed operations may be implemented by other computational components 108.

[0034] The image processing system 100 can be part of or implemented by a computing device or multiple computing devices. In some examples, the image processing system 100 can be part of an electronic device (or multiple devices), such as a camera system (e.g., digital camera, IP camera, video camera, security camera, etc.), a telephone system (e.g., smartphone, cellular phone, conferencing system, etc.), a laptop or notebook computer, a tablet computer, a set-top box, a television, a display device, a digital media player, a gaming console, a video streaming device, a head-mounted display (HMD), an extended reality (XR) device, a drone, an in-car computer, an IoT (Internet of Things) device, a smart wearable device, or any other suitable electronic device. In some implementations, the ToF system 102, the image sensor 104, the storage 106, the other computational components 108, the application processor 110, the ToF engine 130, the image processing engine 134, and the rendering engine 136 can be part of the same computing device.

[0035] For example, in some cases, the ToF system 102, image sensor 104, storage 106, other computational components 108, application processor 110, ToF engine 130, image processing engine 134, and rendering engine 136 may be integrated into a camera, a smartphone, a laptop, a tablet computer, a smart wearable device, an HMD, an XR device, an IoT device, a gaming system, and / or any other computing device. However, in some implementations, one or more of the ToF system 102, image sensor 104, storage 106, other computational components 108, application processor 110, ToF engine 130, image processing engine 134, and / or rendering engine 136 may be part of or implemented by two or more separate computing devices.

[0036] The ToF system 102 can use light, such as near infrared light (NIR), to determine depth and / or distance information about a target (e.g., a surrounding / nearby scene, one or more surrounding / nearby objects, etc.). In some examples, the ToF system 102 can measure both the distance and intensity of each pixel in a target, such as a scene. The ToF system 102 can include a light emitter for emitting a light signal toward the target (e.g., a scene, an object, etc.), which can strike the target and return / reflect back to the ToF system 102. The ToF system 102 can include a sensor for detecting and / or measuring the return / reflected light, which can then be used to determine depth and / or distance information about the target. The distance of the target relative to the ToF system 102 can be used to perform depth mapping. The distance of the target can be calculated through direct ToF or indirect ToF.

[0037] In direct ToF, the distance can be calculated based on the travel time of the emitted light pulse signal and the returned / reflected light pulse signal (e.g., the time from when the light signal is emitted to when the returned / reflected light signal is received). For example, the round trip distance of the emitted light pulse signal and the returned / reflected light pulse signal can be calculated by multiplying the travel time of the emitted light pulse signal and the returned / reflected light pulse signal by the speed of light, commonly denoted as c. The calculated round trip distance can then be divided by two to determine the distance from the ToF system 102 to the target.

[0038] In indirect ToF, distance can be calculated by sending modulated light towards a target and measuring the phase of the returned / reflected light. Knowing the frequency (f) of the emitted light, the phase shift of the returned / reflected light, and the speed of light, the distance to the target can be calculated. For example, run-time differences between the paths of the emitted light and the returned / reflected light will result in a phase shift of the returned / reflected light. The phase difference between the emitted light and the returned / reflected light and the modulation frequency (f) of the light can be used to calculate the distance between the ToF system 102 and the target. For example, the formula for the distance between the ToF system 102 and the target can be (c / 2f)×phase shift / 2π. As this shows, a higher frequency of light can provide a higher measurement accuracy, but results in a shorter maximum distance that can be measured.

[0039] Thus, in some examples, dual frequencies may be used to improve measurement accuracy and / or distance, as described further herein. For example, a 60 MHz optical signal may be used to measure a target 2.5 meters away, and a 100 MHz optical signal may be used to measure a target 1.5 meters away. In a dual frequency scenario, both the 60 MHz optical signal and the 100 MHz optical signal may be used to calculate the distance to the target.

[0040] The image sensor 104 may include any image and / or video sensor or capture device, such as a digital camera sensor, a video camera sensor, a smartphone camera sensor, an image / video capture device on an electronic device such as a television or computer, a camera, etc. In some cases, the image sensor 104 may be part of a camera or computing device, such as a digital camera, a video camera, an IP camera, a smartphone, a smart television, a gaming system, etc. In some examples, the image sensor 104 may include multiple image sensors, such as a rear and a front sensor device, and may be part of a dual camera or other multi-camera assembly (e.g., including two cameras, three cameras, four cameras, or other number of cameras). The image sensor 104 may capture images and / or video frames (e.g., raw image and / or video data), which may then be processed by the application processor 110, the ToF engine 130, the image processing engine 134, and / or the rendering engine 136, as described further herein.

[0041] Storage 106 can be any storage device for storing data. Additionally, storage 106 can store data from any of the components of image processing system 100. For example, storage 106 may store data from ToF system 102 (e.g., ToF sensor data or measurements), image sensor 104 (e.g., frames, videos, etc.), data from and / or used by other computational components 108 and / or application processor 110 (e.g., processing parameters, image data, ToF measurements, depth maps, tuning parameters, processing output, software, files, settings, etc.), ToF engine 130 (e.g., one or more neural networks, image data, tuning parameters, auxiliary metadata, ToF sensor data, ToF measurements, depth maps, training data sets, etc.), data from and / or used by image processing engine 134 (e.g., image processing data and / or parameters, etc.), rendering engine 136 (e.g., output frames), the operating system of image processing system 100, software of image processing system 100, and / or any other type of data.

[0042] The application processor 110 may include, for example, but not limited to, a CPU 112, a GPU 114, a DSP 116, and / or an ISP 118, which the application processor 110 may use to perform various computational operations, such as image / video processing, ToF signal processing, graphics rendering, machine learning, data processing, calculations, and / or any other operations. In the example shown in FIG. 1, the application processor 110 implements a ToF engine 130, an image processing engine 134, and a rendering engine 136. In other examples, the application processor 110 may also implement one or more other processing engines. Moreover, in some cases, the ToF engine 130 may implement one or more machine learning algorithms (e.g., one or more neural networks) configured to perform ToF signal processing and / or generate a depth map.

[0043] In some cases, the application processor 110 may also include memory 122 (e.g., random access memory (RAM), dynamic RAM, etc.) and cache 120. The memory 122 may include one or more memory devices and may include any type of memory, such as, for example, volatile memory (e.g., RAM, DRAM, SDRAM, DDR, static RAM, etc.), flash memory, flash-based memory (e.g., solid-state drive), etc. In some examples, the memory 122 may include one or more DDR (e.g., DDR, DDR2, DDR3, DDR4, etc.) memory modules. In other examples, the memory 122 may include other types of memory modules. The memory 122 may be used to store data, such as, for example, image data, ToF data, processing parameters (e.g., ToF parameters, tuning parameters, etc.), metadata, and / or any type of data. In some examples, the memory 122 can be used to store data from and / or used by the ToF system 102, the image sensor 104, the storage 106, the other computational components 108, the application processor 110, the ToF engine 130, the image processing engine 134, and / or the rendering engine 136.

[0044] Cache 120 may include one or more hardware and / or software components that store data so that future requests for the data may be serviced faster than if the data were stored on memory 122 or storage 106. For example, cache 120 may include any type of cache or buffer, such as, for example, a system cache or an L2 cache. Cache 120 may be faster and / or more cost-effective than memory 122 and storage 106. Moreover, cache 120 may have lower power and / or operating requirements or footprint than memory 122 and storage 106. Thus, in some cases, cache 120 may be used to store / buffer and quickly service certain types of data, such as image data or ToF data, that are expected to be processed and / or requested in the future by one or more components of image processing system 100 (e.g., application processor 110).

[0045] In some examples, operations for the ToF engine 130, image processing engine 134, and rendering engine 136 (as well as any other processing engines) may also be implemented by any of the computational components in the application processor 110. In one illustrative example, operations of the rendering engine 136 may be implemented by the GPU 114, and operations of the ToF engine 130, image processing engine 134, and / or one or more other processing engines may be implemented by the CPU 112, the DSP 116, and / or the ISP 118. In some examples, operations of the ToF engine 130 and image processing engine 134 may be implemented by the ISP 118. In other examples, operations of the ToF engine 130 and / or image processing engine 134 may be implemented by the ISP 118, the CPU 112, the DSP 116, and / or a combination of the ISP 118, the CPU 112, and the DSP 116.

[0046] In some cases, the application processor 110 may include other electronic circuitry or hardware, computer software, firmware, or any combination thereof for performing any of the various operations described herein. In some examples, the ISP 118 may receive data (e.g., image data, ToF data, etc.) captured or generated by the ToF system 102 and / or the image sensor 104 and process the data to generate an output depth map and / or frame. A frame may include a video frame of a video sequence or a still image. A frame may include an array of pixels representing a scene. For example, a frame may be a Red-Green-Blue (RGB) frame having red, green, and blue color components per pixel, a Luma, Red-Difference, Blue-Difference (YCbCr) frame having a Luma component and two Chroma (color) components (red difference and blue difference) per pixel, or any other suitable type of color or monochrome picture.

[0047] In some examples, the ISP 118 may implement one or more processing engines (e.g., ToF engine 130, image processing engine 134, etc.) and may perform ToF signal processing and / or image processing operations such as depth calculation, depth mapping, filtering, demosaicing, scaling, color correction, color conversion, noise reduction filtering, spatial filtering, artifact correction, etc. The ISP 118 may process data from other components in the ToF system 102, image sensor 104, storage 106, memory 122, cache 120, application processor 110, and / or data received from remote sources such as a remote camera, server, or content provider.

[0048] Although image processing system 100 is shown as including several components, one skilled in the art will appreciate that image processing system 100 may include more or fewer components than those shown in Figure 1. For example, image processing system 100 may in some cases also include one or more other memory devices (e.g., RAM, ROM, cache, etc.), one or more networking interfaces (e.g., wired and / or wireless communication interfaces, etc.), one or more display devices, and / or other hardware or processing devices not shown in Figure 1. Illustrative examples of computing devices and hardware components that may be implemented with image processing system 100 are described below with respect to Figure 10.

[0049] 2A is a simplified block diagram illustrating an example of a direct ToF sensing procedure 200. In this example, the ToF system 102 first emits a pulse of light 202 toward a target 210. The target 210 may include, for example, a scene, one or more objects, one or more animals, one or more people, etc. The pulse of light 202 may travel toward the target 210 until it strikes the target 210. When the pulse of light 202 strikes the target 210, at least some portion of the pulse of light 202 may be reflected back to the ToF system 102.

[0050] Thus, the ToF system 102 may receive a reflected light pulse 204 that includes at least some portion of the light pulse 202 that was reflected back from the target 210. The ToF system 102 may detect the reflected light pulse 204 and calculate a distance 206 to the target 210 based on the reflected light pulse 204. To calculate the distance 206, the ToF system 102 may calculate the total time traveled by the emitted light pulse 202 and the reflected light pulse 204 (e.g., the time from when the light pulse 202 is emitted to when the reflected light pulse 204 is received). The ToF system 102 may multiply the total time traveled by the emitted light pulse 202 and the reflected light pulse 204 by the speed of light (c) to determine the total distance traveled by the light pulse 202 and the reflected light pulse 204 (e.g., the round trip time). The ToF system 102 may then divide the total time traveled by two to obtain the distance 206 from the ToF system 102 to the target 210.

[0051] 2B is a simplified block diagram illustrating an example of an indirect ToF sensing procedure 220. In this example, a phase shift of the reflected light may be calculated to determine the depth and distance for the target 210. Here, the ToF system 102 first emits modulated light 222 toward the target 210. The modulated light 222 may have a constant, known, or predetermined frequency. The modulated light 222 may travel toward the target 210 until it strikes the target 210. When the modulated light 222 strikes the target 210, at least some portion of the modulated light 222 may be reflected back to the ToF system 102.

[0052] The ToF system 102 can receive the reflected light 224 and determine the phase shift 226 of the reflected light 224 and the distance 206 to the target 210 using the formula distance (206) = (c / 2f) x phase shift / 2π, where f is the frequency of the modulated light 222 and c is the speed of light.

[0053] In some cases, when calculating depth and distance (e.g., 206), one or more factors that affect how light is reflected can be taken into account or used to adjust the calculation. For example, objects and surfaces can have certain properties that can cause light to be reflected differently. By way of example, different surfaces may have different refractive indices, which can affect how light travels or comes into contact with the surface and / or materials in the surface. Furthermore, non-uniformities such as material irregularities or scattering centers can cause light to be reflected, refracted, transmitted, or absorbed, sometimes resulting in a loss of energy. Thus, when light hits a surface, it may be absorbed, reflected, transmitted, etc. The percentage of light that is reflected by a surface is called the reflectance of the light. However, the reflectance does not only depend on the surface (e.g., refractive index, material properties, uniformity or non-uniformity, etc.), but may also depend on the type of light reflected and the surrounding environment (e.g., temperature, ambient light, water vapor, etc.). Thus, as will be further described below, in some cases, information about the surrounding environment, the type of light, and / or characteristics of the target 210 can be taken into account when calculating distance 206 and / or depth information for the target 210.

[0054] 3 illustrates an example technique for measuring the phase angle of an optical signal. In this example, the ToF system 102 emits a modulated optical signal 222 and receives reflected light 224. The ToF system 102 measures the amplitude (A) of the reflected light 224 at four different points 302, 304, 306, and 308 to determine an amplitude A1 at measurement point 302, an amplitude A2 at measurement point 304, an amplitude A3 at measurement point 306, and an amplitude A4 at measurement point 308. The measurement points 302, 304, 306, and 308 may be equally spaced points (e.g., 0°, 90°, 180°, 270°).

[0055] The phase angle of the reflected light 224 can be expressed by the following equation:

[0056]

number

[0057] Equation 1 shows the relationship between the amplitude measurements at points 302, 304, 306, and 308 and the phase angle of the reflected light 224. The ratio of the difference between A1 and A3 to the difference between A2 and A4 is equal to the tangent of the phase angle.

[0058] FIG. 4 illustrates a continuous wave method 400 for ToF sensing. In this example, an emitted optical signal 402 and a reflected optical signal 404 may be cross-correlated to determine a phase delay of the reflected optical signal 404. The optical signal 402 may first be emitted and sampled quantities 406-412 (Q1, Q2, Q3, Q4) of the reflected optical signal 404 may be measured using four out-of-phase time windows (e.g., sensor integration time windows) C1 414, C2 416, C3 418, and C4 420, where each window (414-420) is phase-staged such that C1 is 0 degrees, C2 is 180 degrees, C3 is 90 degrees, and C4 is 270 degrees. As shown in FIG. 4, the measured quantities 406-412 correspond to the overlap regions between the measurement time windows and the reflected optical signal 404.

[0059] The phase delay between the emitted optical signal 402 and the reflected optical signal 404, i.e.

[0060]

number

[0061] , and the distance, i.e., d, can then be calculated by the following formula:

[0062]

number

[0063] In some examples, the quantities 406-412 (Q1, Q2, Q3, Q4) can be used to calculate pixel intensities and offsets. Additionally, in some cases, the terms (Q3-Q4) and (Q1-Q2) can reduce the effect of constant offsets from the quantities (406-412). The quotients in the phase equation, i.e., Equation 2, can reduce the effects of system or environmental variations from the distance measurements, such as, for example, system amplification, system attenuation, reflected intensity, etc.

[0064] 5 is a diagram illustrating an example image processing system 500 for ToF signal processing. In some cases, the image processing system 500 may correspond to all or a portion of the image processing system 100 of FIG. 1. As shown in FIG. 5, the image processing system 500 may include a projection system 502, a receiving system 508, and a processor 518. These and / or other components of the image processing system 500 may be configured to generate a depth map 516. The depth map 516 may include depth information associated with targets (e.g., objects) within the FOV of the image processing system 500.

[0065] In some cases, the projection system 502 may include one or more light emitting devices 504. The one or more light emitting devices 504 may include any device configured to emit light for the purpose of obtaining depth information (e.g., distance measurement) about one or more targets. In an illustrative example, the one or more light emitting devices 504 may include one or more vertical cavity surface emitting lasers (VCSELs) or other laser devices. For example, the one or more light emitting devices 504 may include multiple VCSELs arranged in a VCSEL array. As described in more detail below, the projection system 502 may include a single VCSEL array, or multiple VCSEL arrays. In some cases, the one or more light emitting devices 504 may be configured to emit light having a certain wavelength (or a certain range of wavelengths). In a non-limiting example, the one or more light emitting devices 504 may be configured to emit narrowband near infrared (NIR) light (e.g., light having a wavelength between 800 and 2500 nanometers). The one or more light emitting devices 504 may emit light having any wavelength suitable for obtaining ToF data.

[0066] As shown, the projection system 502 may include one or more light directing devices 506. The one or more light directing devices 506 may include any device configured to change, adjust, or otherwise direct the angle of light emitted by the one or more light emitting devices 504. In one example, the one or more light directing devices 506 may include one or more projection lenses that disperse light using refraction. In another example, the light directing device 506 may include one or more mirrors that reflect light at a desired angle based on the laws of specular reflection. As described in more detail below, the one or more light directing devices 506 may be configured to selectively direct light emitted by the one or more light emitting devices 504 to specific portions of the FOV of the image processing system 500.

[0067] In some cases, the projection system 502 can generate and project an illumination signal 512. The illumination signal 512 can correspond to light emitted by the one or more light emitting devices 504 and directed by the one or more light directing devices 506. In one example, the illumination signal 512 can include multiple illumination signals emitted at different times. For example, the illumination signal 512 can include a series of illumination signals that are directed sequentially to different FOV portions of the image processing system 500. The illumination signal 512 can be absorbed, transmitted, and / or reflected by one or more targets. All or a portion of the reflected illumination signal can be captured by the receiving system 508. The reflected and captured illumination signal can correspond to the illumination reflection 514 of FIG. 5. In one example, the illumination reflection 514 can be received by a sensor 510 of the receiving system 508. The sensor 510 can include any type or form of device configured to receive, capture, and / or process light having wavelengths of light emitted by the one or more light emitting devices 504. In one example, the sensor 510 may include a spectral bandpass filter that filters out all or a portion (e.g., a majority) of the environmental interference and transmits light having approximately the same wavelength as the illumination signal 512 (e.g., within a certain tolerance thereof). In some cases, the sensor 510 may include a ToF sensor. For example, the sensor 510 may correspond to all or a portion of the ToF system 102 and / or the image sensor 104 of the image processing system 100. The sensor 510 may process the illumination reflections 514 to determine distance information associated with one or more targets. For example, the sensor 510 may determine a distance to a target (in accordance with the direct ToF technique described above) based on an amount of time between the projection of an illumination signal and the receipt of a corresponding illumination reflection. In another example, the sensor 510 may determine a distance to a target (in accordance with the indirect ToF technique described above) based on a phase shift between the illumination signal and the corresponding illumination reflection. In some cases, the resolution (e.g., image resolution) of the sensor 510 corresponds to the number of pixels (e.g., photosensitive elements) in the sensor 510.All or a portion of the pixels of the sensor 510 can receive illumination reflections in response to the projected illumination signal. In one example, the sensor 510 can determine a distance measurement corresponding to each pixel of the sensor 510.

[0068] In some examples, the processor 518 may generate the depth map 516 based at least in part on the distance measurements determined by the sensor 510. For example, the sensor 510 and / or the processor 518 may combine distance measurements associated with individual FOV portions to generate a complete set of distance measurements for the entire FOV of the image processing system 500. The processor 518 may use the complete set of distance measurements to determine a depth map for the entire FOV.

[0069] 6A is a diagram of an example FOV 602 of the image processing system 500 of FIG. 5. In this example, the FOV 602 may correspond to the cumulative FOV of the projection system 502. For example, the FOV 602 may represent the total diffusion angle of the projection system 502. As described above, the one or more light directing devices 506 may be configured to direct the light emitted by the one or more light emitting devices 504 at various angles (e.g., to reduce the diffusion angle of the projection system 502 to a particular FOV portion). In one example, each FOV portion may be associated with a particular configuration of components of the projection system 502. Thus, the FOV 602 may represent a combination of FOVs resulting from each possible projection configuration of the projection system 502.

[0070] FIG. 6B is a diagram of example FOV portions of FOV 602. In this example, FOV 602 may include nine FOV portions. In some cases, the resolution (e.g., image resolution) of the ToF data associated with each FOV portion may correspond to the full resolution of sensor 510. For example, the scene in each FOV portion is imaged continuously across sensor 510. In this manner, the full resolution of sensor 510 may be used in each configuration. Furthermore, the full illumination power of projection system 502 may be used in each configuration. As described in more detail below, image processing system 500 may combine ToF data associated with multiple FOV portions to generate a depth map having a resolution greater than the resolution of sensor 510. For example, the resolution of depth map 516 may correspond to the resolution of sensor 510 multiplied by the number of FOV portions into which FOV 602 is divided. In an illustrative example, the resolution of sensor 510 may be given as r=n×m, and FOV 602 may be divided into N×M FOV portions. In this example, the resolution of the cumulative depth map corresponding to the entire FOV can be given as R=N×M. Furthermore, one FOV portion has a horizontal projection angle θ x and the vertical projection angle θ y Then, the horizontal and vertical projection angles of the FOV 602 are θ X =Nθ x and θ Y =Mθ y For reference, see Figures 6A and 6B.

[0071] In some cases, the projection system 502 can sequentially illuminate each FOV portion of the FOV 602. For example, the projection system 502 can illuminate a first FOV portion by projecting a first illumination signal while in a first projection configuration. The receiving system 508 can capture and process the first illumination reflection at the resolution of the entire sensor 510, and a full resolution ToF frame of the first illumination is stored in a memory buffer. After the first illumination reflection is received and / or processed, the projection system 502 can illuminate a second FOV portion by projecting a second illumination signal while in a second projection configuration. The receiving system 508 can capture and process the second illumination reflection at the resolution of the entire sensor 510, and a full resolution ToF frame of the second illumination is also stored in a memory buffer. In some examples, the image processing system 500 can repeat the process of projecting illumination signals and receiving illumination reflections until an illumination reflection corresponding to each FOV portion is received. During this process, a single FOV portion can be illuminated at a time. For example, the projection system 502 may be configured to stop projecting light toward one FOV portion before projecting light toward another FOV portion.

[0072] In one example, the projection system 502 can illuminate the FOV portions according to a sequential illumination order. The sequential illumination order can correspond to a predetermined order in which the projection system 502 "scans" the FOV portion. The sequential illumination order can include scanning across multiple rows of the FOV portion, scanning across multiple columns of the FOV portion, diagonally scanning the FOV portion, among other illumination orders. In an illustrative example, the sequential illumination order for illuminating the FOV portions shown in FIG. 6B can be: 1,1 , FOV 1,2 , FOV 1,3 , FOV 2,1and so on. In some cases, the receiving system 508 may receive the illumination reflections in a sequential receiving order that corresponds to the sequential illumination order. For example, the sensor 510 may determine a distance measurement in response to each illumination reflection received by the sensor 510. The sensor 510 may record a particular distance measurement in association with an appropriate FOV portion based on the sequential illumination order. For example, the image processing system 500 may send a synchronization signal to the projection system 502 and the receiving system 508. The synchronization signal may indicate the currently illuminated FOV to both the projection system 502 and the receiving system 508.

[0073] As described above, image processing system 500 can combine (e.g., “stitch”) distance measurements associated with individual FOV portions to generate a complete set of distance measurements for the entire FOV. To combine distance measurements associated with individual FOV portions, image processing system 500 may generate a ToF frame for each FOV portion (e.g., a ToF frame FOV for the FOV portion of FIG. 6B ). 1,1 , FOV 1,2 , FOV 1,3 , FOV 2,1 From FOV 3,3) may be stored in a memory buffer built into the sensor 510. In one example, the image processing system 500 may store the ToF frames in a system memory external to the sensor 510 (e.g., the storage 106 of the image processing system 100). In some cases, the image processing system 500 may combine the distance measurements associated with the FOV portions in real-time. For example, the sensor 510 and / or the processor 518 may incorporate the distance measurements into a complete set of distance measurements as new illumination reflections are received and processed according to the sequential illumination sequence. In other examples, the sensor 510 and / or the processor 518 may generate a complete set of distance measurements after each FOV portion is scanned. The complete set of distance measurements may be stored in a single location (e.g., the same file or data structure) and used to generate the depth map 516. The image stitching operation may be implemented by the sensor electronics of the image processing system 500 or external to the sensor electronics (eg, by the device CPU or other processor).

[0074] In some examples, one FOV portion may overlap another FOV portion. For example, FIG. 1,1 and FOV 1,2 6 shows an example overlap 612 between the FOV 1,1 The horizontal resolution of n is n=n x +x, where n x is FOV 1,1 x corresponds to a unique pixel in the FOV 1,1 and FOV 1,2 Similarly, the FOV 1,1 and FOV 2,1 may share overlap 614. In one example, 1,1 The vertical resolution m of is m=m y + y, where m y is FOV1,1 corresponds to a unique pixel in the FOV 1,1 and FOV 2,1 6B , all or a portion of the remaining FOV portions may include similar overlap. In some cases, overlap between FOV portions may facilitate implementation of an image stitching scheme that uses automated digital alignment techniques to seamlessly "stitch" the FOV portions together. For example, image processing system 500 may locate pixels within the FOV portion relative to the entire FOV based at least in part on detecting overlapping (e.g., matching) distance measurements. Image processing system 500 may implement any suitable size of overlap, including no overlap.

[0075] FIG. 7 is a diagram of a device 700 corresponding to an example implementation of the projection system 502 of FIG. 5. In some examples, the device 700 may include multiple light emitting devices. In an illustrative example, the device 700 may have multiple VCSEL arrays. As shown, the device 700 may include one or more light emitting devices, including light emitting devices 708(A), 708(B), and 708(C). Although FIG. 7 shows three light emitting devices 708(A), 708(B), and 708(C), the device 700 may include any suitable number of light emitting devices arranged in any suitable configuration. In an illustrative example, the device 700 may include nine light emitting devices arranged in a 3×3 grid. In some cases, the number of light emitting devices of the device 700 may correspond to the number of FOV portions into which the FOV 602 is divided. In these cases, each light emitting device may correspond to a different FOV portion. For example, the projection system 502 may activate (eg, turn on) each light emitting device in sequence to scan each FOV portion.

[0076] In some cases, device 700 may include one or more projection lenses and / or additional light directing components configured to project light emitted by the light emitting devices. For example, device 700 may include multiple projection lenses (e.g., a projection lens array including projection lenses 710(A), 710(B), and / or 710(C)). In some cases, each projection lens of the projection lens array may correspond to one of the light emitting devices (e.g., light from light emitting device 708(A) is emitted through a corresponding projection lens 710(A) of the multiple projection lenses, light from light emitting device 708(B) is emitted through a corresponding projection lens 710(B), and so on). In an illustrative example, the projection lens array may include multiple individual (e.g., separate) projection lenses. In some examples, device 700 does not include a projection lens array (i.e., the projection lens array may be optional). In some cases, the focal lengths of the projection lenses 710(A), 710(B), and 710(C) of the projection lens array may be such that each of the FOV portions, e.g., FOV 1,1 , FOV 1,2 ,...FOV n,m , .... In some cases, the focal lengths of the projection lenses 710(A), 710(B), and 710(C) along with the light directing elements (e.g., prism 714) determine the amount of overlap between the FOV portions.

[0077] In some cases, the prism 714 shown in FIG. 7 can be configured to direct light at different angles (or different ranges of angles). In some examples, each projection lens of the projection lens array, along with the prism 714, can be configured to direct light at a particular angle. For example, when one light emitting device 708(A), 708(B), or 708(C) is activated, the prism 714 (or the projection lens prism assembly corresponding to the particular light emitting device) can bend the light from the light emitting device (e.g., the light emitting device 708(A), 708(B), or 708(C)) toward a desired FOV portion. In one example, the prism 714 can be a segmented prism array including multiple prism segments configured to refract light in different directions. For example, each prism on the segmented prism array can bend the light from the corresponding light emitting device to a desired angle associated with a particular FOV portion. Although a prism 714 is shown in FIG. 7, other types of optical elements other than prisms, such as a diffractive optical element (DOE), may be used. The direction in which a prism segment (or other optical element) refracts light may depend on the refractive index of the prism segment, the apex angle of the prism segment (e.g., the angle between two faces on the prism segment), among other properties. In some cases, the properties of a prism segment that corresponds to a light emitting device (e.g., a prism segment located above the light emitting device) may be selected such that the prism segment bends light emitted by the light emitting device toward a desired FOV portion.

[0078] In some examples, device 700 may include multiple diffusers (e.g., diffuser array 712). Each diffuser of diffuser array 712 may correspond to (e.g., be located above) a respective projection lens of the projection lens array (e.g., a first diffuser above projection lens 710(A), a second diffuser above projection lens 710(B), and a third diffuser above projection lens 710(C). In some cases, diffuser array 712 (e.g., together with projection lens array) may diffuse light emitted from pixelated VCSELs (or other types of light sources) to increase the uniformity of the illumination projection generated by device 700. In some cases, diffuser 712 may be located opposite projection lenses 710(A), 710(B), and 710(C) of the projection lens array, above VCSEL array 708, or in any other suitable configuration. Device 700 may include any additional or alternative light directing devices not shown in FIG. 7.

[0079] FIG. 8 is a diagram of a device 800 corresponding to another example implementation of the projection system 502 of FIG. 5. In some examples, the device 800 may include a single light emitting device 808. In an illustrative example, the light emitting device 808 may be a single VCSEL array. The single light emitting device 808 may be used to sequentially illuminate each FOV portion into which the full FOV is divided. In some cases, the light emitted by the light emitting device 808 may be guided by a scanning mirror 814. In an illustrative example, the scanning mirror 814 may be a microelectromechanical system (MEMS) mirror. A MEMS mirror scanner may provide various benefits, including high reliability, long lifetime, compact size, low cost, among other benefits. The scanning mirror 814 may include and / or correspond to any additional or alternative types of scanning mirrors.

[0080] In some cases, the device 800 can adjust the orientation of the scan mirror 814 to adjust the direction of the emitted light. For example, the scan mirror 814 can have multiple orientations. Each orientation can correspond to illuminating a different FOV portion. In some cases, a particular orientation of the scan mirror 814 can correspond to a particular angle between the scan mirror 814 and a reference plane. In one example, the reference plane can be a plane (e.g., a face) of the light emitting device 808. FIG. 8 shows three example orientations of the scan mirror 814. The orientation O1 of the scan mirror 814 is shown in FIG. 8 with a dashed line. The direction of the light projected by the scan mirror 814 in the orientation O1 is shown with a corresponding dashed arrow. The orientations O2 and O3 (and their corresponding directions of the projected light) are shown with solid and dotted lines, respectively.

[0081] The device 800 may include one or more additional light directing devices, such as a projection lens 810 and / or a diffuser 812. In some cases, the diffuser 812 (e.g., in some cases together with the lens 810) can increase the uniformity of the illumination projection generated by the device 800. In some cases, the diffuser 812 may be located opposite the projection lens 810 or above the light emitting device 808 (e.g., a VCSEL array). The focal length of the projection lens 810 allows the projection lens 810 to focus the light in each of the FOV portions, e.g., the FOV 1,1 , FOV 1,2 ,...FOV n,m , .... In some cases, the focal length of the projection lens 810 along with the angle of the light directing mirror 814 determine the amount of overlap between the FOV portions. Device 800 may include any additional or alternative light directing device not shown in FIG. 8. In some examples, device 800 does not include a projection lens 810 (i.e., the projection lens 810 may be optional).

[0082] 9A is a diagram of a device 900 corresponding to an example implementation of the receiving system 508 of FIG. 5. In some examples, the device 900 may include a sensor 910 (e.g., corresponding to the sensor 510 of FIG. 5). The device 900 may also include one or more components configured to focus an illumination reflection (e.g., an illumination reflection 514 based on an illumination signal 512 emitted by the projection system 502 of FIG. 5, the device 700 of FIG. 7, or the device 800 of FIG. 8) onto the sensor 910. For example, the device 900 may include an imaging lens array 902 including one or more imaging lenses (e.g., imaging lenses 902(A), 902(B), and / or 902(C)). In an illustrative example, the imaging lens array 902 may include a 3×3 matrix of imaging lenses. In some cases, the refractive properties (e.g., index of refraction, angle, tilt, curvature, tilt, etc.) of the imaging lens array 902 may be selected to focus illumination reflections corresponding to one FOV portion onto the sensor 910. Furthermore, the sensor 910 may be located at a distance from the imaging lens array 902 (or other imaging lens arrays not shown in FIG. 9) that facilitates capture of illumination reflections. For example, the sensor 910 may be located in a back focal plane of the imaging lens array 902. In one illustrative example, the imaging lens array 902 may have a focal length f. The sensor 910 may be located at a distance f from the imaging lens array 902 to facilitate and / or ensure that a scene illuminated by the projector of FIG. 7 or FIG. 8 is focused or imaged onto the sensor 910. Furthermore, in some cases, the sensor 910 may be centered relative to the center and / or central lens segment of the imaging lens array 902. In these cases, the center-to-center distance between the imaging lens segments may be designed to facilitate illumination of a FOV portion of a particular size (e.g., a particular projection angle) and / or to facilitate focused illumination reflection. For example, the location of imaging lenses 902(A), 902(B), and / or 902(C) may be designed to facilitate a desired level of light diffusion and / or concentration.Each imaging lens of imaging lens array 902 (imaging lenses 902(A), 902(B), and 902(C)) covers a portion of the FOV, i.e., the FOV. 1,1 , FOV 1,2 ,...FOV n,m , ... to the sensor 910. In some cases, the number of imaging lenses is the same as the number of FOV portions.

[0083] In some examples, the imaging lens array 902 can be configured for off-axis imaging. For example, in an off-axis imaging configuration, the optical axis of the imaging lens is not aligned with the center of the sensor. In some cases, such off-axis imaging can result in aberrations (e.g., undesirable diffusion of light). To correct and / or eliminate aberrations associated with off-axis imaging, one or more imaging lenses in the imaging lens array 902 may be tilted relative to one another, as shown in FIG. 9A. For example, the sensor 910 may be aligned with (e.g., located directly below) the central imaging lens 902(B) of the imaging lens array 902, and the other imaging lenses of the imaging lens array 902 (e.g., imaging lenses 902(A) and 902(C)) may be offset relative to the sensor 910.

[0084] In some cases, device 900 may include one or more filters to facilitate capture of illumination reflections. For example, device 900 may include a bandpass filter configured to pass light having a frequency corresponding to a frequency of light emitted by one or more light emitting devices (e.g., the light emitting devices of device 700 of FIG. 7 and / or device 800 of FIG. 8). In an illustrative example, device 900 may include a narrow bandpass filter with a center frequency corresponding to NIR light. The bandpass filter may pass light corresponding to illumination reflections and at least partially block optical interference from external light sources, e.g., sunlight, thereby increasing the signal-to-noise or signal-to-background noise ratio and thus increasing the accuracy and / or quality of the generated depth map. FIG. 9A illustrates an exemplary filter 922 that may be implemented within device 900. As shown, filter 922 may be located above sensor 910.

[0085] In conjunction with an optical projection system (e.g., projection system 502 of FIG. 5, which may include device 700 of FIG. 7 and / or device 800 of FIG. 8), device 900 can perform a continuous ToF image capture process. FIGS. 9B and 9C show an exemplary continuous ToF image capture process that may be performed by device 900 and devices 700 and / or 800. The following description gives examples of the continuous ToF image capture process performed by device 900 and device 700. At time T1, device 700 can activate light emitting device 708(A), causing light to be directed toward a first FOV portion by prism 714 and / or a corresponding projection lens 710(A) of the projection lens array. In response (e.g., at a time synchronized with time T1), sensor 910 captures the first FOV portion (e.g., the ToF frame FOV of FIG. 6B). 1,1It is possible to receive light corresponding to the irradiation reflection of the FOV portion corresponding thereto (the FOV portion corresponding to the ToF frame FOV in FIG. 6B). This light is shown as light L1 in FIG. 9C. As shown in FIGS. 9B and 9C, the received light L1 can be guided to the sensor 910 by the image lens 902(A) and / or by additional light guiding components of the device 900. At time T2, the device 700 can activate the light emitting device 708(B) and guide light toward the second FOV portion by the prism 714 and / or the corresponding projection lens 710(B). In response (e.g., at a time synchronized with time T2), the sensor 910 can receive light corresponding to the irradiation reflection of the second FOV portion (e.g., the FOV portion corresponding to the ToF frame FOV in FIG. 6B) 1,2 It is possible to receive light corresponding to the irradiation reflection of the FOV portion corresponding thereto (the FOV portion corresponding to the ToF frame FOV in FIG. 6B). This light is shown as light L2 in FIG. 9C. The received light L2 can be guided to the sensor 910 by the image lens 902(B) and / or by additional light guiding components of the device 900. At time T3, the device 700 can activate the light emitting device 708(C) and guide light toward the third FOV portion by the prism 714 and / or the corresponding projection lens 710(C). In response (e.g., at a time synchronized with time T3), the sensor 910 can receive light corresponding to the irradiation reflection of the third FOV portion (e.g., the FOV portion corresponding to the ToF frame FOV in FIG. 6B) 1,3 It is possible to receive light corresponding to the irradiation reflection of the FOV portion corresponding thereto (the FOV portion corresponding to the ToF frame FOV in FIG. 6B). This light is shown as light L3 in FIG. 9C. The received light L3 can be guided to the sensor 910 by the image lens 902(C) and / or by additional light guiding components of the device 900.

[0086] In some cases, the amount of time between time T1 and time T2 (and between time T2 and time T3) may be sufficient for device 700 to project illumination signals into one FOV, device 900 to receive illumination reflections corresponding to the illumination signals, and device 900 (and / or additional processors or computing devices) to process the illumination reflections for incorporation into a depth map. In some examples, the continuous ToF illumination process may be directed at least in part by a driver 702 (shown in FIG. 7 ) connected to device 700. For example, driver 702 may send control signals 716(A), 716(B), and 716(C) to light emitting devices 708(A), 708(B), and 708(C), respectively. The control signals may direct a single light emitting device to be active at a time. For example, the control signals may alternate activation of the light emitting devices.

[0087] The following description gives examples of the continuous ToF image capture process performed by device 800 and device 900. At time T1, device 800 can turn scanning mirror 814 in orientation O1 to direct light toward a first FOV portion. In response (e.g., at a time synchronized with time T1), sensor 910 can capture the first FOV portion (e.g., the ToF frame FOV of FIG. 6B ) and the second FOV portion (e.g., the ToF frame FOV of FIG. 6B ). 1,1 At time T2, device 800 can turn scanning mirror 814 in orientation O2 to direct the light toward the second FOV portion. In response (e.g., at a time synchronized with time T2), sensor 910 can receive light corresponding to the illumination reflection of the second FOV portion (e.g., the ToF frame FOV of FIG. 6B ). 1,2 At time T3, device 800 can turn its scanning mirror to an orientation O3, directing the light toward the third FOV portion. In response (e.g., at a time synchronized with time T3), sensor 910 can receive light corresponding to the illumination reflection of the third FOV portion (e.g., the ToF frame FOV of FIG. 6B ). 1,38. In some instances, the amount of time between time T1 and time T2 (and between time T2 and time T3) may be sufficient for device 800 to project an illumination signal into one FOV, device 900 to receive illumination reflections corresponding to the illumination signal, and device 900 (and / or additional processors or computing devices) to process the illumination reflections for incorporation into the depth map. In some instances, the continuous illumination process may be directed at least in part by a driver 802 communicatively coupled to device 800. For example, driver 802 may send a control signal 816 to scanning mirror 814. Control signal 816 may instruct scanning mirror 814 to adjust the orientation of scanning mirror 814 at the appropriate time to scan each FOV portion.

[0088] In some cases, device 700 can facilitate sequentially obtaining distance measurements for multiple FOV portions without mechanical movement. For example, device 700 can illuminate a desired FOV portion by selective activation of a light emitting device, rather than by adjusting the physical position and / or orientation of components of device 700. Eliminating the need for mechanical movement can reduce the risk of failure within device 700. As described above, device 800 can obtain distance measurements for multiple FOV portions by mechanical adjustment of a scanning mirror. However, mechanical fatigue (and corresponding performance degradation) caused by mechanical adjustment of the scanning mirror can be negligible. Additionally, device 800 can include a single light emitting device, which can reduce the size, cost, and / or complexity of the device.

[0089] Returning to FIG. 5, as described above, the image processing system 500 can synchronize the projection system 502 and the receiving system 508 with a synchronization signal. For example, the image processing system 500 can implement a synchronization scheme that includes sending a first control signal to the projection system 502 and sending a second control signal to the receiving system 508. In one example, the first control signal can indicate a sequential illumination order, and the second control signal can indicate a corresponding sequential reception order. In some cases, the first control signal and the second control signal can be the same (or similar) time-synchronized signals.

[0090] FIG. 10A is a diagram of an exemplary synchronized ToF system 1000(A). In one example, all or a portion of the synchronized ToF system 1000(A) may correspond to the device 700 of FIG. 7 and the device 900 of FIG. 9A. As shown, the synchronized ToF system 1000(A) may include a controller 1002. The controller 1002 may send a control signal 1004 to a driver (e.g., driver 702) of the device 700. In some cases, the control signal 1004 may represent a VCSEL selection signal. For example, the control signal 1004 may instruct the driver 702 to selectively activate the light emitting device 708 of the device 700 at the appropriate time. In one example, the control signal 1004 may instruct the driver 702 to send the control signals 716(A), 716(B), and / or 716(C) shown in FIG. 7. In some cases, the control signals 716(A), 716(B), and / or 716(C) may include a drive current suitable for operation of the light emitting device 708 (e.g., a pulse train drive current or a continuous wave (CW) modulated drive current). The controller 1002 may also send the control signal 1006 to the sensor electronics 1008 of the device 900. The sensor electronics 1008 may include hardware and / or software components configured to determine a distance measurement associated with the illumination reflection. In some cases, the control signal 1006 may be synchronized with the control signal 1004. For example, the control signal 1006 may indicate to the sensor electronics 1008 which FOV portion is currently illuminated and therefore which FOV portion corresponds to the current illumination reflection. Based on the control signals 1004 and 1006, the synchronized ToF system 1000(A) may associate and label the distance measurement with the corresponding FOV portion.

[0091] FIG. 10B is a diagram of an exemplary synchronized ToF system 1000(B). In one example, all or a portion of the synchronized ToF system 1000(B) may correspond to the device 800 of FIG. 8 and the device 900 of FIG. 9A. As shown, the synchronized ToF system 1000(B) may include a controller 1012. The controller 1012 may send a control signal 1014 to a driver (e.g., driver 802) of the device 800. In some cases, the control signal 1014 may represent an angle selection signal. For example, the control signal 1014 may instruct the driver 802 to adjust the angle of the scanning mirror 814 at the appropriate time. In some examples, the controller 1012 may send a control signal 1020 to an additional driver 1016. In one example, the driver 1016 may be a driver for the light emitting device 808 of the device 800. In some cases, the control signal 1020 may represent a VCSEL synchronization signal. For example, the control signal 1020 may instruct the driver 1016 to provide an appropriate drive current to the light emitting device 808 to continuously illuminate the FOV portion. As shown, the controller 1012 may send a control signal 1022 to the sensor electronics 1018 of the device 900. The sensor electronics 1018 may include hardware and / or software components configured to determine a distance measurement associated with the illumination reflection. In some cases, the control signal 1022 may be synchronized with the control signal 1014 and / or the control signal 1020. For example, the control signal 1022 may indicate to the sensor electronics 1018 which FOV portion is currently illuminated and therefore which FOV portion corresponds to the current illumination reflection. Based on the control signals 1014, 1020 and / or 1022, the synchronized ToF system 1000(B) may associate and label a distance measurement with the corresponding FOV portion.

[0092] Using the systems and techniques described herein, a ToF system can generate depth maps (e.g., depth image maps) having a resolution that exceeds the full resolution of the ToF sensor of the ToF system. The systems and techniques also enable the ToF system to generate high-resolution depth maps without incurring significant delays in processing time and without increasing the size of the sensor (thus reducing the cost of the sensor).

[0093] 11 is a flow diagram illustrating an example process 1100 for high-resolution ToF depth imaging. For clarity, the process 1100 is described with reference to the systems and devices of FIGs. 5, 7, 8, and 9A. The steps or operations outlined herein are examples and can be implemented in any combination thereof, including combinations in which some steps or operations are removed, added, or modified.

[0094] In operation 1102, the process 1100 includes illuminating at least one portion of an FOV of a projection system including one or more light emitting devices, where the entire FOV includes a plurality of FOV portions. In some examples, a first FOV portion of the plurality of FOV portions may overlap at least a second FOV portion of the plurality of FOV portions. In some examples, illuminating the plurality of FOV portions includes illuminating each of the plurality of FOV portions in a sequential illumination order (e.g., by illuminating a single FOV portion at a time). In such examples, receiving each illumination reflection of the plurality of illumination reflections includes receiving each reflection in a sequential reception order corresponding to the sequential illumination order. The process 1100 may include generating each image of the plurality of images based on each illumination reflection of the plurality of illumination reflections. Referring to FIG. 5 as an illustrative example, the projection system 502 of the image processing system 500 may illuminate each of the plurality of FOV portions in a sequential illumination order, at least in part, by illuminating a single FOV portion at a time. In some examples, the projection system 502 can continuously project the illumination signal 512 using the light emitting device 504 and / or the light directing device 506 .

[0095] In one illustrative example, the light emitting device 504 can include multiple light emitting devices, and the light directing device 506 can include multiple projection lenses. Each projection lens of the multiple projection lenses can be configured to project light emitted by one of the multiple light emitting devices toward a different FOV portion of the multiple FOV portions (e.g., together with a prism such as prism 714). In some examples, the projection system 502 can include a segmented prism array configured to direct light to each FOV portion of the multiple FOV portions. For example, the multiple light emitting devices 504 can include the light emitting device 708 of FIG. 7, and the multiple light directing device can include the prism 714 of FIG. 7. In one example, the prism 714 can be a segmented prism array including multiple prism segments. Each prism segment of the prism 714 can be located above one of the light emitting devices 708. Furthermore, each prism segment of the prism 714 can be oriented at a different angle relative to a plane of the light emitting device 708. In some examples, the projection system 502 may include one or more diffusers positioned relative to the scanning mirror. The one or more diffusers are configured to diffuse the light emitted by the single light emitting device. For example, the device 700 of FIG. 7 (as an example of the projection system 502) may include a diffuser array 712. As another example, the device 800 of FIG. 8 may include a diffuser 812.

[0096] In another illustrative example, light emitting device 504 may include a single light emitting device and light directing device 506 may include a scanning mirror. The scanning mirror may be configured to project light emitted by the single light emitting device toward different ones of the FOV portions when oriented in different orientations. For example, the single light emitting device may include light emitting device 808 of FIG. 8 and the single light directing device may include scanning mirror 814 of FIG. 8. Scanning mirror 814 may be a scanning mirror. In some cases, scanning mirror 814 may be located above light emitting device 808. Furthermore, each different orientation of scanning mirror 814 may correspond to a different orientation angle between scanning mirror 814 and a plane of light emitting device 808.

[0097] In operation 1104, the process 1100 includes sequentially capturing a plurality of images (e.g., by a sensor of a receiving system, such as the receiving system 508) based on a plurality of illumination reflections corresponding to light emitted by one or more light emitting devices. Each image of the plurality of images corresponds to one of the plurality of FOV portions. Furthermore, an image resolution associated with each image of the plurality of images corresponds to the full resolution of the sensor. In one illustrative example with reference to FIG. 5, the receiving system 508 of the image processing system 500 may receive the illumination reflections 514 in a sequential reception order corresponding to the sequential illumination order of the illumination signal 512. The sensor of the receiving system 508 may capture an image for each illumination reflection. For example, the projection system 502 may illuminate a first FOV portion of the plurality of FOV portions, the receiving system 508 may receive the first illumination reflection, and the sensor may capture a first image corresponding to the first FOV portion. After receiving the first illumination reflection, the projection system 502 may illuminate a second FOV portion of the plurality of FOV portions. The receiving system 508 can then receive a second illumination reflection, and the sensor can capture a second image corresponding to a second FOV portion.

[0098] In one example, the projection system 502 and the receiving system 508 may be synchronized. For example, the processor 518 of the image processing system 500 may send a first control signal to the projection system 502 instructing the projection system 502 to illuminate a particular FOV portion of the multiple FOV portions. The processor 518 may also send a second control signal to the receiving system 508 instructing the receiving system 508 to associate the illumination reflection received by the sensor 510 with the particular FOV portion. The first control signal and the second control signal may be time synchronized.

[0099] In some examples, the receiving system 508 can include an array of imaging lenses, with each imaging lens of the array configured to project light associated with a different portion of a scene corresponding to a respective FOV portion of the multiple FOV portions onto a sensor of the receiving system 508. For example, the device 900 of FIG. 9A (as an example of the receiving system 508) can include an imaging lens array 902. In some cases, the receiving system 508 can include a filter located above the sensor 510. For example, the device 900 can include a filter 922. The filter 922 can be configured to transmit light having a frequency corresponding to a frequency of light emitted by the light emitting device 504 (e.g., the light emitting device 708 of FIG. 7 and / or the light emitting device 808 of FIG. 8).

[0100] In operation 1106, the process 1100 includes using the multiple images (and / or illumination reflections) to generate a high-resolution depth map associated with the entire FOV. For example, the processor 518 can generate a depth map 516, which may include a high-resolution depth map based on various full resolution images associated with each FOV portion of the entire FOV. In one example, the processor 518 can generate a plurality of partial distance measurements (e.g., as images), each corresponding to one of the illumination reflections 514. The processor 518 can then generate the depth map 516 by combining the plurality of partial distance measurements. Furthermore, in one example, the image resolution of the depth map 516 may correspond to the maximum resolution of the sensor 510 multiplied by the number of individual FOV portions.

[0101] In some examples, the process 1100 and / or other processes described herein may be implemented by one or more computing devices or apparatuses. In some examples, the process 1100 and / or other processes described herein may be implemented by one or more computing devices having the image processing system 500 shown in FIG. 5, the device 700 shown in FIG. 7, the device 800 shown in FIG. 8, the device 900 shown in FIG. 9A, the synchronized ToF system 1000(A) shown in FIG. 10A, the synchronized ToF system 1000(B) shown in FIG. 10B, and / or the computing device architecture 1200 shown in FIG. 12. In some cases, such computing devices or apparatuses may include a processor, a microprocessor, a microcomputer, or other components of a device configured to perform steps of the process 1100. In some examples, such computing devices or apparatuses may include one or more sensors configured to capture image data. For example, the computing device may include a smartphone, a camera, a head-mounted display, a mobile device, or other suitable device. In some examples, such computing devices or apparatuses may include a camera configured to capture one or more images or videos. In some cases, such a computing device may include a display for displaying images. In some examples, the one or more sensors and / or cameras are separate from the computing device, in which case the computing device receives the sensed data. Such a computing device may further include a network interface configured to communicate data.

[0102] The components of a computing device may be implemented in a circuit configuration. For example, the components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein. The computing device may further include a display (as an example of an output device or in addition to an output device), a network interface configured to communicate and / or receive data, any combination thereof, and / or other components. The network interface may be configured to communicate and / or receive Internet Protocol (IP)-based data or other types of data.

[0103] Process 1100 is illustrated as a logical flow diagram, whose operations represent sequences of operations that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the described operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0104] Additionally, process 1100, and / or other processes described herein, may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0105] 12 is a diagram illustrating an example of a system for implementing some aspects of the present technology. Specifically, FIG. 12 illustrates an example of a computing system 1200, which may be, for example, an internal computing system, a remote computing system, a camera, or any computing device whose components constitute any of those components that communicate with each other using a connection 1205. The connection 1205 may be a physical connection using a bus, or a direct connection to a processor 1210, such as in a chipset architecture. The connection 1205 may also be a virtual connection, a network connection, or a logical connection.

[0106] In some examples, computing system 1200 is a distributed system in which the functionality described in this disclosure may be distributed across a data center, multiple data centers, a peer network, etc. In some examples, one or more of the described system components represent many components for which the components each perform some or all of the functionality described. In some cases, the components may be physical or virtual devices.

[0107] The exemplary system 1200 includes at least one processing unit (CPU or processor) 1210 and connections 1205 coupling various system components to the processor 1210, including system memory 1215, such as read only memory (ROM) 1220 and random access memory (RAM) 1225. The computing system 1200 may include a cache of high-speed memory 1212 directly connected to the processor 1210, in close proximity to it, or integrated as part of it.

[0108] Processor 1210 can include any general purpose processor and hardware or software services, such as services 1232, 1234, and 1236 stored in storage device 1230, configured to control processor 1210, as well as special purpose processors where software instructions are incorporated into the actual processor design. Processor 1210 can essentially be a completely self-contained computing system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0109] To enable user interaction, computing system 1200 includes input device(s) 1245, which may represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gestural or graphical input, a keyboard, a mouse, motion input, speech, etc. Computing system 1200 may also include output device(s) 1235, which may be one or more of a number of output mechanisms. In some instances, a multi-modal system may enable a user to provide multiple types of input / output to communicate with computing system 1200. Computing system 1200 may include a communication interface 1240, which may generally govern and manage user input and system output.The communications interface may be an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, an optical fiber port / plug, a proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® low energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.12 The communication interface 1240 may implement or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers, including those utilizing Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or any combination thereof. The communication interface 1240 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine a position of the computing system 1200 based on reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States Global Positioning System (GPS), the Russian Global Navigation Satellite System (GLONASS), the Chinese BeiDou Navigation Satellite System (BDS), and the European Galileo GNSS.Since there is no constraint to operating on any particular hardware configuration, the basic features herein may be easily replaced by improved hardware or firmware configurations as they are developed.

[0110] The storage device 1230 may be a non-volatile and / or non-transitory and / or computer readable memory device, such as a hard disk, or a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid state memory, a compact disk read only memory (CD-ROM) optical disk, a re-writable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disk (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, It may be another type of computer readable medium capable of storing data that is accessible by a computer, such as an EMV chip, a Subscriber Identity Module (SIM) card, a mini / micro / nano / pico SIM card, another Integrated Circuit (IC) chip / card, a Random Access Memory (RAM), a Static RAM (SRAM), a Dynamic RAM (DRAM), a Read Only Memory (ROM), a Programmable Read Only Memory (PROM), an Erasable Programmable Read Only Memory (EPROM), an Electrically Erasable Programmable Read Only Memory (EEPROM), a Flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a Resistive Random Access Memory (RRAM / ReRAM), a Phase Change Memory (PCM), a Spin Transfer Torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0111] The storage devices 1230 may include software services, servers, services, etc. that cause the system to perform functions when code defining such software is executed by the processor 1210. In some examples, hardware services that perform particular functions may include software components stored in a computer-readable medium that are in communication with the necessary hardware components, such as the processor 1210, connections 1205, output devices 1235, etc., to perform the functions.

[0112] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media that can store, contain, or transport instructions and / or data. Computer-readable media may include non-transitory media that do not include carrier waves and / or transitory electronic signals that propagate wirelessly or over wired connections, with data stored thereon. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted using any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0113] In some examples, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and signals in and of itself.

[0114] Specific details are given in the above description to provide a thorough understanding of the examples provided herein. However, it will be understood by those skilled in the art that the examples may be practiced without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including individual functional blocks, including functional blocks comprising devices, device components, and steps or routines in a method embodied in software, or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, or other components may be shown as components in block diagram form so as not to obscure the examples with unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the examples.

[0115] Individual examples may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0116] The processes and methods according to the examples described above can be implemented using computer-executable instructions stored or otherwise available from a computer-readable medium. Such instructions can include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a processing device to perform or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code, and the like. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, network-attached storage devices, and the like.

[0117] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, and the like. The functionality described herein may also be embodied in a peripheral device or an add-in card. Such functionality may also be implemented on a circuit board, in different chips, or on different processes executing in a single device, as further examples.

[0118] The instructions, media for carrying such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0119] In the above description, aspects of the present application have been described with respect to specific examples thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concepts may be embodied and employed in various other ways, and that the appended claims are intended to be construed to include such variations, except as limited by the prior art. Various features and aspects of the application examples described above may be used individually or jointly. Moreover, the examples may be used in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the present specification. Thus, the specification and drawings should be regarded as illustrative and not restrictive. For purposes of illustration, the methods have been described in a particular order. It should be appreciated that in alternative examples, the methods may be performed in an order different from that described.

[0120] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of the description.

[0121] When a component is described as being "configured to" perform some operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operation, or any combination thereof.

[0122] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component and / or any component that is in communication, either directly or indirectly, with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communications interface).

[0123] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, a claim language reciting "at least one of A and B" means A, B, or A and B. In another example, a claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, a claim language reciting "at least one of A and B" can mean A, B, or A and B, plus can further include items not listed in the set of A and B.

[0124] The various exemplary logic blocks, modules, circuits, and algorithm steps described with respect to the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been generally described above with respect to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0125] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a magnetic or optical data storage medium, or the like. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0126] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided in dedicated software or hardware modules configured for encoding and decoding, or may be incorporated into a composite video encoder-decoder (codec).

[0127] Illustrative aspects of the present disclosure include the following.

[0128] Aspect 1: An apparatus for high-resolution time-of-flight depth imaging, comprising: a projection system including one or more light emitting devices, each light emitting device of the one or more light emitting devices configured to illuminate at least a portion of an entire field of view (FOV) of the projection system, the entire FOV including a plurality of FOV portions; a receiving system including a sensor configured to sequentially capture a plurality of images based on a plurality of illumination reflections corresponding to light emitted by the one or more light emitting devices, each image of the plurality of images corresponding to one of the plurality of FOV portions, and an image resolution associated with each image of the plurality of images corresponding to a full resolution of the sensor; and a processor configured to use the plurality of images to generate a high-resolution depth map associated with the entire FOV.

[0129] Aspect 2: An apparatus according to aspect 1, wherein the projection system is configured to illuminate each of the multiple FOV portions in a sequential illumination sequence, the sequential illumination sequence including illuminating a single FOV portion at a time, and the receiving system is configured to receive each illumination reflection of the multiple illumination reflections in a sequential reception sequence corresponding to the sequential illumination sequence, and to generate each image of the multiple images based on each illumination reflection.

[0130] Aspect 3: An apparatus according to aspect 2, wherein to illuminate each of the multiple FOV portions in a sequential illumination order, the projection system is configured to illuminate a first FOV portion of the multiple FOV portions, receive a first illumination reflection corresponding to the first FOV portion, and, after receiving the first illumination reflection, illuminate a second FOV portion of the multiple FOV portions, and receive a second illumination reflection corresponding to the second FOV portion.

[0131] Aspect 4: An apparatus according to aspect 1, wherein the one or more light emitting devices include a plurality of light emitting devices, and the projection system includes a plurality of projection lenses, each projection lens of the plurality of projection lenses configured to project light emitted by one of the plurality of light emitting devices toward a different FOV portion of the plurality of FOV portions.

[0132] Aspect 5: An apparatus according to aspect 4, wherein the projection system includes one or more diffusers positioned relative to the multiple projection lenses, the one or more diffusers configured to diffuse light emitted by the multiple light emitting devices.

[0133] Example 6: An apparatus according to example 4, wherein each of the multiple projection lenses is positioned above a multiple light emitting device.

[0134] Example 7: An apparatus according to Example 1, wherein the projection system includes a segmented prism array, the segmented prism array configured to direct light to each FOV portion of the multiple FOV portions.

[0135] Example 8: An apparatus according to Example 1, wherein the one or more light emitting devices include a single light emitting device and the projection system includes a scanning mirror, which, when oriented in different orientations, is configured to project light emitted by the single light emitting device toward different FOV portions of the multiple FOV portions.

[0136] Example 9: The apparatus according to example 8, wherein the scanning mirror comprises a microelectromechanical system (MEMS) mirror.

[0137] Example 10: An apparatus according to example 8, wherein the scanning mirror is positioned above the single light emitting device, and each different orientation of the scanning mirror corresponds to a different orientation angle between the scanning mirror and a plane of the single light emitting device.

[0138] Example 11: An apparatus according to Example 8, wherein the projection system includes one or more diffusers positioned relative to the scanning mirror, the one or more diffusers configured to diffuse light emitted by the single light emitting device.

[0139] Aspect 12: An apparatus according to aspect 1, wherein the receiving system includes an array of imaging lenses, and each imaging lens of the array of imaging lenses is configured to project onto the sensor light associated with a different portion of the scene corresponding to a respective FOV portion of the multiple FOV portions.

[0140] Aspect 13: An apparatus according to aspect 1, wherein the receiving system further comprises a filter positioned above the sensor, the filter configured to transmit light having a frequency corresponding to a frequency of light emitted by the one or more light emitting devices.

[0141] Aspect 14: An apparatus according to aspect 1, wherein the processor is configured to synchronize the projection system and the receiving system based at least in part on sending a first control signal to the projection system instructing the projection system to illuminate a specific FOV portion among the multiple FOV portions, and sending a second control signal to the receiving system instructing the receiving system to associate the illumination reflection received by the sensor with the specific FOV portion, wherein the first control signal and the second control signal are time synchronized.

[0142] Example 15: An apparatus according to example 1, wherein a first FOV portion of the plurality of FOV portions partially overlaps with at least a second FOV portion of the plurality of FOV portions.

[0143] Aspect 16: To generate a high-resolution depth map associated with an entire FOV, the processor generates a plurality of partial distance measurements, each of the plurality of partial distance measurements corresponding to one of the plurality of illumination reflections, and combines the plurality of partial distance measurements. The apparatus according to aspect 1, configured as follows.

[0144] Aspect 17: An apparatus according to aspect 1, wherein the image resolution of the high resolution depth map corresponds to the maximum resolution of the sensor multiplied by the number of FOV portions.

[0145] Aspect 18: A method for high-resolution time-of-flight depth imaging, comprising: illuminating a plurality of FOV portions of an entire field of view (FOV) of the projection system using one or more light emitting devices of a projection system, wherein each light emitting device of the one or more light emitting devices is configured to illuminate at least a portion of the entire FOV; sequentially capturing a plurality of images by a sensor of a receiving system based on a plurality of illumination reflections corresponding to light emitted by the one or more light emitting devices, wherein each image of the plurality of images corresponds to one of the plurality of FOV portions, and an image resolution associated with each image of the plurality of images corresponds to a full resolution of the sensor; and generating a high-resolution depth map associated with the entire FOV using the plurality of images.

[0146] Aspect 19: A method according to aspect 18, wherein the step of illuminating the multiple FOV portions includes a step of illuminating each of the multiple FOV portions in a sequential illumination order, the sequential illumination order including illuminating a single FOV portion at a time, and the step of receiving each illumination reflection of the multiple illumination reflections includes a step of receiving each reflection in a sequential reception order corresponding to the sequential illumination order, and further including a step of generating each image of the multiple images based on each illumination reflection of the multiple illumination reflections.

[0147] Aspect 20: A method according to aspect 19, wherein the step of illuminating each of the multiple FOV portions in a sequential illumination order includes the steps of illuminating a first FOV portion of the multiple FOV portions, receiving a first illumination reflection corresponding to the first FOV portion, and after receiving the first illumination reflection, illuminating a second FOV portion of the multiple FOV portions, and receiving a second illumination reflection corresponding to the second FOV portion.

[0148] Aspect 21: A method according to aspect 18, wherein the one or more light emitting devices include a plurality of light emitting devices, and the projection system includes a plurality of projection lenses, each projection lens of the plurality of projection lenses configured to project light emitted by one of the plurality of light emitting devices toward a different FOV portion of the plurality of FOV portions.

[0149] Aspect 22: A method according to aspect 21, wherein the projection system includes one or more diffusers positioned relative to the multiple projection lenses, the one or more diffusers configured to diffuse light emitted by the multiple light emitting devices.

[0150] Example 23: The method according to example 21, wherein each of the multiple lenses is positioned above a multiple number of light emitting devices.

[0151] Example 24: The method according to example 18, wherein the projection system includes a segmented prism array, the segmented prism array configured to direct light to each FOV portion of the multiple FOV portions.

[0152] Example 25: A method according to example 18, wherein the one or more light emitting devices include a single light emitting device, and the projection system includes a scanning mirror, which, when oriented in different orientations, is configured to project light emitted by the single light emitting device toward different FOV portions of the multiple FOV portions.

[0153] Example 26: The method according to example 25, wherein the scanning mirror comprises a microelectromechanical system (MEMS) mirror.

[0154] Example 27: A method according to example 25, wherein the scanning mirror is positioned above the single light emitting device, and each of the different orientations of the scanning mirror corresponds to a different orientation angle between the scanning mirror and a surface of the single light emitting device.

[0155] Example 28: A method according to example 25, wherein the projection system includes one or more diffusers positioned relative to the scanning mirror, the one or more diffusers configured to diffuse light emitted by the single light emitting device.

[0156] Aspect 29: A method according to aspect 18, wherein the receiving system includes an array of imaging lenses, and each imaging lens of the array of imaging lenses is configured to project onto the sensor light associated with a different portion of the scene corresponding to each FOV portion of the multiple FOV portions.

[0157] Example 30: The method according to example 18, further comprising a filter positioned above the sensor, the filter configured to transmit light having a frequency corresponding to a frequency of light emitted by the one or more light emitting devices.

[0158] Aspect 31: The method according to aspect 18, further comprising a step of synchronizing the projection system and the receiving system based at least in part on sending a first control signal to the projection system instructing the projection system to illuminate a specific FOV portion among the multiple FOV portions, and sending a second control signal to the receiving system instructing the receiving system to associate the illumination reflection received by the sensor with the specific FOV portion, wherein the first control signal and the second control signal are time synchronized.

[0159] Example 32: The method according to example 18, wherein a first FOV portion of the plurality of FOV portions partially overlaps with at least a second FOV portion of the plurality of FOV portions.

[0160] Aspect 33: A method according to aspect 18, wherein the step of generating a high-resolution depth map associated with the entire FOV includes the steps of generating a plurality of partial distance measurements, each of the plurality of partial distance measurements corresponding to one of the plurality of illumination reflections, and combining the plurality of partial distance measurements.

[0161] Aspect 34: A method according to aspect 18, wherein the image resolution of the high resolution depth map corresponds to the maximum resolution of the sensor multiplied by the number of FOV portions.

[0162] Aspect 35: A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations according to any of aspects 1 to 34.

[0163] Example 36: An apparatus comprising means for performing the operations according to any of Examples 1 to 34. [Explanation of symbols]

[0164] 100 Image Processing System 102 ToF System 104 Image Sensor 106 Storage 108 Computational Components 110 Application Processor 112 CPU 114 GPU 116 DSP 118 ISP 120 Cache 122 Memory 130 ToF Engine 134 Image Processing Engine 136 Rendering Engine 202 Emitted light pulse 204 Reflected Light Pulse 206 distance 210 Target 222 Modulated Light 224 Reflected light 402 Emitted Light Signal 404 Reflected Light Signal 500 Image Processing System 502 Projection System 504 Light Emitting Devices 506 Light Induction Device 508 Receiving System 510 Sensors 512 Irradiation signal 514 Irradiation reflection 516 Depth Map 518 processor 602 FOV 700 devices 702 Driver 708 Light Emitting Devices 710 Projection Lens 712 Diffuser Array, Diffuser 714 Prism 716 Control Signals 800 devices 802 Driver 808 Light Emitting Devices 810 Projection lens, lens 812 Diffuser 814 Scanning mirror, light guide mirror 816 Control Signals 900 devices 902 Imaging lenses, imaging lens arrays 910 Sensor 922 Filters 1000 Synchronized ToF System 1002 Controller 1004 Control Signal 1006 Control Signal 1008 Sensor Electronics 1012 Controller 1016 Driver 1018 Sensor Electronics 1020 Control Signal 1022 Control Signal 1200 Computing system, system 1210 Processor 1205 Connection 1212 Cache 1215 System Memory 1220 Read-Only Memory (ROM) 1225 Random Access Memory (RAM) 1230 Storage Device 1235 Output Device 1240 Communication Interface 1245 Input Devices

Claims

1. 1. An apparatus for high resolution time-of-flight depth imaging, comprising: a projection system including one or more light emitting devices, each light emitting device of the one or more light emitting devices configured to illuminate at least a portion of an overall field of view (FOV) of the projection system, the overall FOV including a plurality of FOV portions; a receiving system including a sensor configured to sequentially capture a plurality of images based on a plurality of illumination reflections corresponding to light emitted by the one or more light emitting devices, each image of the plurality of images corresponding to one of the plurality of FOV portions, and an image resolution associated with each image of the plurality of images corresponding to a full resolution of the sensor; and a processor configured to combine the multiple images and generate a high-resolution depth map associated with the entire FOV, the high-resolution depth map having a resolution greater than the full resolution of the sensor.

2. the projection system is configured to illuminate each of the plurality of FOV portions in a sequential illumination sequence, the sequential illumination sequence including illuminating a single FOV portion at a time; The apparatus of claim 1 , wherein the receiving system is configured to receive each illumination reflection of the plurality of illumination reflections in a successive receiving order corresponding to the successive illumination order, and to generate each image of the plurality of images based on each illumination reflection.

3. To illuminate each of the plurality of FOV portions in the sequential illumination sequence, the projection system comprises: illuminating a first FOV portion of the plurality of FOV portions; receiving a first illumination reflection corresponding to the first FOV portion; illuminating a second FOV portion of the plurality of FOV portions after receiving the first illumination reflection; and receiving a second illumination reflection corresponding to the second FOV portion.

4. the one or more light emitting devices include a plurality of light emitting devices; 2. The apparatus of claim 1, wherein the projection system includes a plurality of projection lenses, each projection lens of the plurality of projection lenses configured to project light emitted by one of the plurality of light emitting devices toward a different one of the plurality of FOV portions.

5. the projection system includes one or more diffusers positioned relative to the plurality of projection lenses, the one or more diffusers configured to diffuse the light emitted by the plurality of light emitting devices; and / or Each of the plurality of projection lenses is positioned above the plurality of light emitting devices.

5. The apparatus of claim 4.

6. 10. The apparatus of claim 1, wherein the projection system includes a segmented prism array configured to direct the light to each FOV portion of the multiple FOV portions.

7. the one or more light emitting devices include a single light emitting device; 2. The apparatus of claim 1, wherein the projection system includes a scanning mirror configured to project light emitted by the single light emitting device toward different ones of the multiple FOV portions when oriented in different orientations.

8. the scanning mirror comprises a microelectromechanical system (MEMS) mirror; the scanning mirror is positioned above the single light emitting device, and each of the different orientations of the scanning mirror corresponds to a different orientation angle between the scanning mirror and a plane of the single light emitting device; and / or The projection system includes one or more diffusers positioned relative to the scanning mirror, the one or more diffusers configured to diffuse the light emitted by the single light emitting device.

8. The apparatus of claim 7.

9. the receiving system includes an array of imaging lenses, each imaging lens of the array of imaging lenses configured to project onto the sensor light associated with a different portion of a scene corresponding to a respective FOV portion of the plurality of FOV portions; and / or The receiving system further comprises a filter positioned above the sensor, the filter configured to transmit light having a frequency corresponding to a frequency of light emitted by the one or more light emitting devices.

2. The apparatus of claim 1.

10. The processor, sending a first control signal to the projection system instructing the projection system to illuminate a particular FOV portion of the plurality of FOV portions; 2. The apparatus of claim 1, configured to synchronize the projection system and the receiving system based at least in part on: sending a second control signal to the receiving system instructing the receiving system to associate illumination reflections received by the sensor with the particular FOV portion, the first control signal and the second control signal being time synchronized.

11. The apparatus of claim 1 , wherein a first FOV portion of the plurality of FOV portions overlaps at least a second FOV portion of the plurality of FOV portions.

12. To generate the enhanced resolution depth map associated with the entire FOV, the processor: generating a plurality of partial distance measurements, each of the plurality of partial distance measurements corresponding to one of the plurality of illumination reflections; and combining the plurality of partial distance measurements.

13. The apparatus of claim 1 , wherein an image resolution of the enhanced depth map corresponds to a maximum resolution of the sensor multiplied by a number of the plurality of FOV portions.

14. 1. A method for high resolution time-of-flight depth imaging, comprising: illuminating a plurality of field of view (FOV) portions of an entire FOV of the projection system using one or more light emitting devices of the projection system, each light emitting device of the one or more light emitting devices being configured to illuminate at least one portion of the entire FOV; capturing a plurality of images successively by a sensor of a receiving system based on a plurality of illumination reflections corresponding to light emitted by the one or more light emitting devices, each image of the plurality of images corresponding to one of the plurality of FOV portions, and an image resolution associated with each image of the plurality of images corresponding to a full resolution of the sensor; and combining the multiple images to generate a high-resolution depth map associated with the entire FOV, the high-resolution depth map having a resolution greater than the full resolution of the sensor.

15. 1. A non-transitory computer readable storage medium for high resolution time-of-flight depth imaging, comprising:

15. A non-transitory computer-readable storage medium comprising instructions stored therein that, when executed by one or more processors, cause the one or more processors to perform the method of claim 14.

Images