Imaging system configured to use time-of-flight imaging and stereoscopic imaging

Abstract

An imaging system is configured to use a time-of-flight (ToF) pixel array to determine depth information using a ToF imaging method and / or a stereoscopic imaging method. The light emitting component emits light to illuminate the scene, and the light detecting component detects reflected light via the ToF pixel array. The ToF pixels are configured to determine phase shift data based on a phase shift between the emitted light and the reflected light, and to determine intensity data based on an amplitude of the reflected light. The plurality of ToF pixels are shared by a single microlens. This enables multiple offset images to be generated using the intensity data measured by each ToF pixel. Accordingly, depth information may be determined using both a ToF imaging method and a stereoscopic imaging method via a configuration in which a plurality of ToF pixels share a single microlens.

Description

[0001] Related Applications

[0002] This application is a divisional application of the invention patent application with application number 202080080975.3 and invention name “Imaging system configured to use time-of-flight imaging and stereoscopic imaging”. Background Art

[0003] A depth camera may use any of a variety of types of pixels to determine the depth of an object in a scene. For example, a depth camera may include a stereo pixel array and may implement a passive imaging method configured to determine depth information from disparity data. This passive imaging method may be referred to as a "stereo" imaging method, which creates an illusion of depth in an image by means of stereoscopic imaging for binocular vision. More specifically, a stereo pixel typically includes a pair of adjacent photodiodes covered by a single microlens. A stereo pixel array may generate two offset images (e.g., a left image and a right image) using incident light detected separately by individual adjacent photodiode pairs. The disparity data includes the distance between two corresponding points in the two offset images. The stereo imaging method uses the known distance between the photodiode pairs to obtain disparity data for the two offset images, and then uses the disparity data to determine the depth information. The advantage of the stereo imaging method is that no artificial light source is required. Figure 1A A voxel 102 is illustrated having a photodiode pair 104 , 106 that shares a single microlens 108 .

[0004] Alternatively, the depth camera may include a time-of-flight pixel array and may implement an active imaging method that uses phase shift data to determine depth information. This active imaging method may be referred to as a "time-of-flight" imaging method. A time-of-flight pixel typically includes a photodiode and two photogates or two transfer gates. More specifically, the time-of-flight imaging method illuminates a scene with light emitted from an artificial light source and detects the reflected light. The phase shift between the emitted light and the reflected light is measured, and depth information may be determined based on the phase shift. Figure 1B A time-of-flight (ToF) pixel 110 with a single photodiode 112 behind a single microlens 114 is illustrated.

[0005] Unfortunately, both stereoscopic imaging methods and time-of-flight imaging methods may encounter problems with accurately determining depth information for a scene. Stereoscopic imaging methods struggle to accurately determine depth information when the scene does not include a recognizable pattern that enables effective correspondence between a point in a first image of two offset images and the same point in a second image of the two offset images. Time-of-flight imaging methods struggle to accurately determine depth information when there is strong ambient light (e.g., sunlight) that interferes with reflected light.

[0006] It is with respect to these and other considerations that the disclosure presented herein is presented. Summary of the invention

[0007] The technology disclosed herein describes an imaging system that is configured to use a time-of-flight pixel array and a corresponding controller to determine depth information for a scene based on a time-of-flight imaging method and / or a stereo imaging method. That is, the technology described herein allows the time-of-flight imaging method and the stereo imaging method to be used in a complementary manner to improve the accuracy of determining depth information for a scene.

[0008] As described above, stereoscopic imaging methods require recognizable patterns in the scene to effectively determine corresponding points in the two offset images. If the pattern cannot be recognized, the stereoscopic imaging method is generally ineffective with respect to accurately determining the depth. Time-of-flight imaging methods do not require the recognition of such patterns to accurately determine the depth. In contrast, when strong ambient light interferes with the reflected light, the accuracy of determining depth information for the time-of-flight imaging method is affected. However, ambient light helps to expose a pattern that, when recognized, enables effective correspondence between points in the two offset images. Therefore, ambient light is beneficial for stereoscopic imaging methods.

[0009] The time-of-flight pixel array and corresponding controller described herein address deficiencies in both time-of-flight imaging methods and stereoscopic imaging methods. Therefore, the techniques described herein improve the accuracy of determining depth information because the techniques can adapt to changing conditions that affect the determination of depth. These conditions may include the degree of change in the recognizable pattern in the scene, the amount of change in ambient light, and / or other conditions that may change from one scene to the next.

[0010] The imaging system described herein may include a depth camera configured using an integrated circuit. The imaging system includes a light emitting component configured to emit light to illuminate a scene and a light detecting component for detecting reflected light. The light detecting component includes a time-of-flight pixel array. The time-of-flight pixel is configured to determine phase shift data based on a phase shift between the emitted light and the reflected light. The time-of-flight pixel array can be used to determine depth information via a time-of-flight imaging method using phase shift data measured by the time-of-flight pixel. In addition, the time-of-flight pixel is configured to determine intensity data using the amplitude of the reflected light.

[0011] The light detection assembly also includes a plurality of microlenses. Each microlens is shared by at least two time-of-flight pixels. This enables the plurality of time-of-flight pixels to have overlapping fields of view, so that at least two offset images can be generated using intensity data measured by each time-of-flight pixel. Disparity data can be determined based on intensity values ​​corresponding to the same point in the offset images. Therefore, via a configuration in which at least two time-of-flight pixels share a single microlens, the time-of-flight pixel array can also be used to determine depth information by a stereoscopic imaging method using intensity data measured by the time-of-flight pixels.

[0012] The imaging system also includes a controller. The controller is configured to generate a first depth image of the scene using a time-of-flight imaging method. That is, the controller can generate the first depth image using phase shift data determined for each time-of-flight pixel in the array. The controller is also configured to determine disparity data based on intensity data measured for each pixel in at least two time-of-flight pixels shared by individual microlenses. The controller can generate a second depth image of the scene using the disparity data determined for each microlens.

[0013] Therefore, the imaging system is configured to generate depth data using both a time-of-flight imaging method and a stereoscopic imaging method. The controller can use one or both of the first depth image generated via the time-of-flight imaging method or the second depth image generated via the stereoscopic imaging method to determine the distance between the object in the scene and the imaging system. In one example, the controller can combine the depth data in the first depth image with the depth data in the second depth image by averaging the corresponding depth values ​​for individual pixels to generate a representative depth image. The controller can then use the representative depth image to determine the distance between the object in the scene and the imaging system.

[0014] In another example, the controller may compare a first depth quality of a first depth image to a second depth quality of a second depth image. To determine the quality, the controller may segment the depth image and perform a segmentation analysis in which edge sharpness and / or uniformity between segmentations of the depth image are evaluated. The controller may determine that one of the depth images has a higher quality and select such a depth image as the depth image used to determine the distance between an object in the scene and the imaging system.

[0015] In addition to those technical benefits discussed above, implementations of the disclosed technology can produce improved focus of a camera. For example, the determined distance between an object in a scene and an imaging system can be used to focus the camera on the object and / or select (e.g., activate) a preconfigured mode for the camera to capture a photo of the object in the scene. Alternatively, implementations of the disclosed technology can produce improved motion recognition (e.g., the position and movement of a human body in physical space), which can be used for input to applications executed on a computing system such as a game console. Other technical benefits not specifically mentioned herein can also be achieved through implementations of the disclosed subject matter.

[0016] This summary is provided to introduce in simplified form some concepts that will be further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to help determine the scope of the claimed subject matter. For example, the term "technique" may refer to (multiple) systems, (multiple) methods, computer-readable instructions, (multiple) modules, algorithms, hardware logic, and / or (multiple) operations as allowed by the above context and the entire document. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The detailed description is provided with reference to the accompanying drawings. In the accompanying drawings, the leftmost digit(s) of a reference number indicates the drawing in which the reference number first appears. The same reference numbers in different drawings indicate similar or identical items. References to individual items of a plurality of items may use reference marks with letters in the alphabetical sequence to refer to each individual item. General references to items may use specific reference marks without the alphabetical sequence.

[0018] Figure 1A A conventional voxel is illustrated.

[0019] Figure 1B A conventional time-of-flight pixel is illustrated.

[0020] Figure 2 An example environment is illustrated in which the imaging systems described herein may be used.

[0021] Figure 3 An example of a depth camera implemented via an integrated circuit is illustrated.

[0022] Figure 4 An example is illustrated of how two time-of-flight pixels have a common field of view of an object when they share a microlens, and thus the configuration can be used to implement a stereoscopic imaging method in addition to the time-of-flight imaging method.

[0023] Figure 5 Another example of a depth camera implemented on an integrated circuit is illustrated.

[0024] Figure 6 An example environment is illustrated in which an imaging system may use time-of-flight imaging methods and stereoscopic imaging methods in a complementary manner.

[0025] Figure 7 is a flow chart illustrating aspects of a routine for determining the depth of an object in a scene using one or both of stereoscopic imaging methods and time-of-flight imaging methods.

[0026] Figure 8 is a computing architecture diagram illustrating aspects of the configuration and operation of a device that may implement aspects of the techniques disclosed herein. DETAILED DESCRIPTION

[0027] The specific embodiments disclose aspects of an imaging system configured to use a time-of-flight pixel array and a corresponding controller to determine depth information for a scene based on a time-of-flight imaging method and / or a stereoscopic imaging method. The imaging system includes a light emitting component configured to emit light to illuminate a scene and a light detecting component for detecting reflected light. The light detecting component includes a time-of-flight pixel array. The time-of-flight pixels are configured to determine phase shift data based on a phase shift between the emitted light and the reflected light. Therefore, the time-of-flight pixel array can be used to determine depth information via a time-of-flight imaging method using phase shift data measured by the time-of-flight pixels. In addition, the time-of-flight pixels are configured to determine intensity data using the amplitude of the reflected light.

[0028] The light detection assembly also includes a plurality of microlenses. Each microlens is shared by at least two time-of-flight pixels. This enables the plurality of time-of-flight pixels to have overlapping fields of view, so that two offset images can be generated using the intensity data measured by each time-of-flight pixel. Therefore, via a configuration in which at least two time-of-flight pixels share a single microlens, the time-of-flight pixel array can also be used to determine depth information via a stereoscopic imaging method using the intensity data measured by the time-of-flight pixels.

[0029] Figure 2An example environment 200 in which the imaging system 202 described herein can be effectively used is illustrated. The imaging system 202 may include a depth camera, which may be part of or connected to a device 204. The device 204 may be configured to use the depth of an object 206 (e.g., a dog in this example) in a scene 208 for various purposes. For example, the depth of the object 206 may be used to focus a camera for a photo. Alternatively, the depth of the object 206 may be used to better detect the motion of a user interacting with a game console while playing a game. Additionally, the depth of the object 206 may be used to present associated virtual content via a head-mounted display device. Thus, the device 204 may include a smart phone, a head-mounted display device, a game console, a tablet device, a laptop device, a camera, etc. The scene 208 may include a physical space in front of the device 204 or a physical space surrounding the device 204.

[0030] As shown in example environment 200, imaging system 202 includes a light emitting component 210 configured to emit light 212 to illuminate scene 208 and / or object 206. Imaging system 202 also includes a light detecting component 214 to detect reflected light 216 from object 206. Light detecting component 214 includes a time-of-flight pixel array 218. The techniques described herein can be used with various types of indirect time-of-flight pixels, including photogate time-of-flight pixels, transfer gate time-of-flight pixels, current-assisted time-of-flight pixels, and the like.

[0031] The time-of-flight pixel is configured to determine phase shift data based on the phase shift between the emitted light 212 and the reflected light 216. For example, the light emitting component 210 can illuminate the scene 208 using a modulated light source that generates a pulsed wave or a continuous wave (e.g., a sine wave or a square wave). The modulated light source can be a solid-state laser or a light emitting diode operating in the near infrared range (e.g., ~850nm), which is invisible to the human eye. The light detecting component 214 can observe the reflection. The phase shift between the illumination and the reflection is measured and converted into a distance.

[0032] In the example where the light emitting assembly 210 illuminates the scene 208 with a continuous wave, the light detecting assembly 214 illuminates the scene for a period of time and can measure the reflected energy using 90 degree phase stepped samples. The charge accumulated during these samples (Q1, Q2, Q3, Q4) can be measured. The phase angle between illumination and reflection is And the distance (d) can be calculated as follows:

[0033]

[0034] In the above equation, c is the speed of light constant. Further to this example, the time-of-flight pixel is configured to determine the intensity data A based on the following calculation:

[0035]

[0036] As this article refers to Figure 3 As further described, the light detection assembly 214 also includes a plurality of microlenses. Each microlens is shared by at least two time-of-flight pixels. In other words, the plurality of time-of-flight pixels sharing a single microlens have overlapping fields of view, so that the intensity data determined by each time-of-flight pixel can be used to generate an offset image.

[0037] The imaging system 202 also includes a controller 220. The controller 220 is configured to generate a first depth image 224 (e.g., a depth map having a depth value for each pixel) of the scene 208 using the phase shift data 222 from the time-of-flight pixel array 218. Thus, the controller 220 can generate the first depth image 224 using a time-of-flight imaging method or a depth value determined based on the phase shift data 222.

[0038] The controller 220 is also configured to determine disparity data 226 (e.g., a measurable distance between two corresponding points in an offset image pair) using the intensity data from the time-of-flight pixel array 218. The intensity data aggregated across the plurality of microlenses may be used to generate an offset image of the scene 208 and determine disparity data 226 between corresponding points. The disparity data 226 is used to generate a second depth image 228. Thus, the controller 220 may generate the second depth image 228 using a stereo imaging method other than the time-of-flight imaging method as described above.

[0039] Using the above techniques, the imaging system 202 can adapt to changing scene conditions because the imaging system 202 is configured to generate depth data using both the time-of-flight imaging method and the stereoscopic imaging method. The controller 220 can use one or both of the first depth image 224 or the second depth image 228 to determine the distance 230 between the object 206 in the scene 208 and the imaging system 202. For example, if there is limited ambient light, the first depth image 224 generated using the time-of-flight method is more likely to include accurate depth information. Conversely, if there is strong ambient light, the second depth image 228 generated using the stereoscopic method is more likely to include accurate depth information.

[0040] In one example, the controller 220 may combine the depth data in the first depth image 224 with the depth data in the second depth image 226 by averaging the depth values ​​for the individual pixels to generate a representative depth image 232. The controller 220 may then use the representative depth image 232 to determine the distance 230 between the object 206 and the imaging system 202.

[0041] In another example, the controller 220 may compare a first depth quality of the first depth image 224 with a second depth quality of the second depth image 228. To determine the quality, the controller 220 may segment the depth images 224, 228 and perform a segmentation analysis in which edge sharpness and / or uniformity between the segmentations is evaluated. The controller 220 may determine that one of the depth images 224, 228 has a higher quality relative to the segmentation(s) containing the object 206, and select such a depth image 224, 228 as the depth image used to accurately determine the distance 230 between the object 206 in the scene 208 and the imaging system 202. Other ways of determining quality may also be used.

[0042] The depth camera may include an integrated circuit. Figure 3 An example of a depth camera implemented via an integrated circuit (IC) 300 is shown. The depth camera includes a two-dimensional array 302 of time-of-flight pixels (e.g., Figure 2 The time-of-flight pixel array 218 in FIG. 210 ). Individual time-of-flight pixels include dedicated circuitry for processing the detection charge output. In a specific example, the time-of-flight pixel array 302 may include 100*100 pixels, and thus the time-of-flight pixel array 302 may include 100*100 processing circuitry.

[0043] In order to implement a stereoscopic imaging method other than a time-of-flight imaging method for a depth camera, a single microlens 304 is shared (eg, covered) by a plurality of time-of-flight pixels. Figure 3 As shown in the example of FIG. 3 , a single microlens 304 is shared by two time-of-flight pixels 306, 308, and the method is propagated throughout the entire time-of-flight pixel array 302. The integrated circuit 300 also includes a controller 310, such as a microprocessor (e.g., Figure 2 220 in the controller 310), the controller 310 is configured to receive data from the processing circuit of the flight time pixel, and also includes a memory 312 (e.g., a random access memory and / or a read-only memory), which is configured to store data of the flight time pixel array 302, and also includes a clock 314 and / or an input / output (I / O) circuit 316.

[0044] The controller 310 is configured to cause the light emitting component (eg, a laser diode or LED device) to emit light into the scene 208 and / or toward the target object 206 (eg, Figure 2As described above, some of the emitted light will reflect from the object 206 and fall on the time-of-flight pixel array 302. Each time-of-flight pixel is configured to measure both the amplitude (e.g., intensity) of the reflected light and the phase shift of the light as it travels from the light emitting assembly to the object 206 and then back to the time-of-flight pixel array 302.

[0045] Figure 4 An example 400 illustrating how two time-of-flight pixels have a common field of view of an object when the two time-of-flight pixels share a microlens. As shown, a single microlens 402 covers a first time-of-flight pixel 404 and a second time-of-flight pixel 406 separated by a distance 408. The single microlens 402 enables the first time-of-flight pixel 404 and the second time-of-flight pixel 406 to share a field of view (e.g., covering the shadow area of ​​a dog), and thus two offset images (e.g., a left image and a right image) can be generated based on the intensity data measured by the time-of-flight pixels. Disparity data between the two offset images can then be determined, for example, based on the measured offset between the left and right images and the distance 408.

[0046] Figure 4 The composition of an example time-of-flight pixel is also illustrated. As shown, Figure 4 The time-of-flight pixel diagram on the left may include a photodiode 412 , a transfer gate transistor or photogate transistor 414 , a reset transistor 416 , a floating diffusion region 418 , an amplifier transistor 420 , and / or a select transistor 422 .

[0047] Figure 5 Another example of a depth camera implemented on an integrated circuit (IC) 500 is shown. The integrated circuit 500 is similar to Figure 3 However, in the two-dimensional array 302 of time-of-flight pixels, a single microlens 502 shares four time-of-flight pixels 504, 506, 508, 510. In this example, the microprocessor 310 can be configured to generate a depth image using a stereo imaging method using disparity data based on the four offset images.

[0048] Figure 6 An example environment 600 is illustrated in which an imaging system can use time-of-flight imaging methods and stereoscopic imaging methods in a complementary manner. Figure 2As described, the imaging system 202 may be part of a device 204 configured with a camera. The camera may be configured to capture a field of view 602 for a photograph. When the photographed object 604 is within a certain distance 606 of the imaging system, a time-of-flight imaging method may be used to focus the camera on the object 604 because the object 604 is close enough to provide a strong reflected light signal. The strong reflected light signal is less likely to be disturbed due to conditions where ambient light is present.

[0049] However, when the object being photographed 604 is located beyond a certain distance 606 from the imaging system (as captured by the dotted form of the object 604 and outside the focus distance 608), stereoscopic imaging methods can be used to focus the camera on the object 604 because the reflected light signal is attenuated due to the greater distance and ambient light.

[0050] Figure 7 is a flow chart illustrating a routine 700 that describes aspects of the present disclosure. In various examples, the operations of the routine 700 may be performed by the imaging system 202. Figure 7 In any of the processes, the logical operations described herein may be implemented as (1) a sequence of computer-implemented actions or program modules running on a computing device, and / or (2) interconnected machine logic circuits or circuit modules within a computing device.

[0051] For ease of understanding, the processes discussed in this disclosure are depicted as separate operations represented as independent boxes. However, these separately described operations should not be interpreted as necessarily order-dependent in their performance. The order in which the processes are described is not intended to be interpreted as limiting, and any number of the described process boxes can be combined in any order to implement the process or an alternative process. In addition, one or more of the operations provided may also be modified or omitted.

[0052] The specific implementation of the technology disclosed herein is a selection problem that depends on the performance of computing equipment and other requirements. Therefore, the logical operation described herein is referred to as state, operation, structural device, action, or module in different ways. These states, operations, structural devices, actions, and modules can be realized with hardware, software, firmware, special digital logic, and any combination thereof. It should be understood that more or less operations than those shown in the figure and described herein can be performed. These operations can also be performed in a sequence different from the sequence described herein.

[0053] It should also be understood that the illustrated method can be terminated at any time and does not need to be performed as a whole. Some or all of the operations of the method and / or substantially equivalent operations can be performed by executing computer-readable instructions included on a computer-readable medium. The term "computer-readable instructions" and its variations used in the specification and claims are expandably used herein to include routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. Computer-readable instructions can be implemented on various system configurations, including processing units in single-processor systems or multi-processor systems, minicomputers, mainframe computers, personal computers, head-mounted display devices, handheld computing devices, microprocessor-based programmable consumer electronics, and combinations thereof, etc.

[0054] For example, the operations of routine 700 may be implemented by a dynamic link library ("DLL"), a static link library, a function generated by an application programming interface ("API"), a compiled program, an interpreted program, a script, a network service or site, or any other executable instruction set. Data may be stored in a data structure in one or more memory components. Data may be retrieved from a data structure by addressing a link or reference to the data structure.

[0055] Although the following diagrams can refer to the components of the accompanying drawings, it should be understood that the operation of routine 700 can also be implemented in many other ways. For example, routine 700 can be implemented at least in part by another remote computer, processor, or circuit. In addition, one or more operations of routine 700 are alternatively or additionally implemented at least in part by a chipset that works alone or in combination with other software modules. In the example described below, one or more modules of the computing system can receive and / or process data disclosed herein. Any service, circuit, or application for providing the technology disclosed herein can be used in the described operation.

[0056] Reference Figure 7 , routine 700 begins at operation 702, where light is emitted to illuminate a scene. At operation 704, reflected light is detected by a time-of-flight pixel array, and individual time-of-flight pixels in the array (i) determine phase shift data based on a phase shift between the emitted light and the reflected light, and (ii) determine intensity data based on the amplitude of the reflected light. As described above, the time-of-flight pixel array includes a plurality of microlenses, and each microlens is shared by at least two time-of-flight pixels, so that a stereoscopic imaging method can be used as a complementary method to the time-of-flight imaging method. At operation 706, a first depth image of the scene is generated using the phase shift data determined for each time-of-flight pixel in the array.

[0057] At operation 708, disparity data is determined using intensity data determined by each of the at least two time-of-flight pixels sharing the microlens. At operation 710, a second depth image is generated using the disparity data determined for each microlens.

[0058] At operation 712, at least one of the first depth image or the second depth image is used to determine a distance between an object in the scene and an imaging system including a time-of-flight pixel array. For example, the distance can be used to focus a camera on the object and / or select (e.g., activate) a preconfigured mode of the camera to capture a photograph of the object. In another example, the distance can be used to improve motion recognition (e.g., the position and movement of a human body in a physical space), which can be used for input to an application executed on a computing system such as a game console. In yet another example, the distance can be used to display virtual content associated with the object via a head-mounted display device.

[0059] Figure 8 A computing device architecture 800 of a computing device capable of executing the various components described herein is illustrated. The computing device architecture 800 is applicable to computing devices that facilitate computing operations due in part to form factor, wireless connectivity, and / or battery power. In some configurations, the computing device includes, but is not limited to, a head mounted display device, a smartphone device, a tablet device, a slate device, a video game device, etc.

[0060] The computing device architecture 800 includes a processor 802, a memory component 804, a network connectivity component 806, a sensor component 808, an input / output component 810, and a power component 812. In the illustrated configuration, the processor 802 is in communication with the memory component 804, the network connectivity component 806, the sensor component 808, the input / output ("I / O") component 810, and the power component 812.

[0061] The processor 802 may include a central processing unit ("CPU") configured to process data, execute computer-executable instructions of one or more applications, and communicate with other components of the computing device architecture 800 to perform the various functions described herein. The processor 802 may be used to execute various aspects of the software components presented herein.

[0062] In some configurations, the processor 802 includes a graphics processing unit (“GPU”) configured to accelerate operations performed by the CPU, including, but not limited to, operations performed by executing general scientific and / or engineering computing applications and graphics-intensive computing applications such as high-resolution video (e.g., 720P, 1080P, and higher resolutions), video games, three-dimensional (“3D”) modeling applications, etc. In some configurations, the CPU and GPU may be configured according to a co-processing CPU / GPU computing model, where sequential portions of an application are executed on the CPU and computationally intensive portions are accelerated by the GPU.

[0063] In some configurations, the processor 802 is or is included in a system on a chip ("SoC") along with one or more of the other components described herein. For example, the SoC may include the processor 802, a GPU, one or more network connectivity components 806, and / or one or more sensor components 808. In some configurations, the processor 802 is manufactured in part using package-on-package ("PoP") integrated circuit packaging technology. The processor 802 may be a single-core package-on-package technology or a multi-core processor.

[0064] The memory component 804 includes random access memory (“RAM”) 814, read-only memory (“ROM”) 816, integrated storage memory (“integrated storage”) 818, and / or removable storage memory (“removable storage”) 820. In some configurations, RAM 814 or portions thereof, ROM 816 or portions thereof, and / or some combination of RAM 814 and ROM 816 are integrated into the processor 802. In some configurations, ROM 816 is configured to store firmware, an operating system or portions thereof (e.g., an operating system kernel), and / or a boot loader for loading an operating system kernel from the integrated storage 818 and / or the removable storage 820.

[0065] The integrated storage device 818 may include a solid-state memory, a hard disk, or a combination of a solid-state memory and a hard disk. The integrated storage device 818 may be soldered or otherwise connected to a logic board to which the processor 802 and other components described herein may also be connected. The integrated storage device 818 may be configured to store an operating system or portions thereof, application programs, data, and other software components described herein.

[0066] The removable storage device 820 may include a solid-state memory, a hard disk, or a combination of a solid-state memory and a hard disk. In some configurations, the removable storage device 820 is provided in place of the integrated storage device 818. In other configurations, the removable storage device 820 is provided as additional optional storage. In some configurations, the removable storage device 820 is logically combined with the integrated storage device 818 so that the total available storage is used as the total combined storage capacity. In some configurations, the total combined capacity of the integrated storage device 818 and the removable storage device 820 is displayed to the user, rather than the individual storage capacities of the integrated storage device 818 and the removable storage device 820.

[0067] The removable storage device 820 is configured to be inserted into a removable storage memory slot or other mechanism through which the removable storage device 820 is inserted and secured to facilitate a connection through which the removable storage device 820 can communicate with other components of the computing device, such as the processor 802. The removable storage device 820 can be implemented in various memory card formats, including but not limited to PC card, compact flash card, memory stick, secure digital ("SD"), miniSD, microSD, universal integrated circuit card ("UICC") (e.g., subscriber identity module ("SIM") or universal SIM ("USIM")), proprietary formats, etc.

[0068] The network connectivity component 806 includes a wireless wide area network component ("WWAN component") 822, a wireless local area network component ("WLAN component") 824, and a wireless personal area network component ("WPAN component") 826. The network connectivity component 806 facilitates communication to and from a network 856 or another network (which may be a WWAN, a WLAN, or a WPAN). Although only network 856 is illustrated, the network connectivity component 806 may facilitate simultaneous communication with multiple networks. For example, the network connectivity component 806 may facilitate simultaneous communication with multiple networks via one or more of a WWAN, a WLAN, or a WPAN.

[0069] The network 856 may be or may include a WWAN, such as a mobile telecommunications network that utilizes one or more mobile telecommunications technologies to provide voice and / or data services to computing devices utilizing the computing device architecture 800 via the WWAN component 822 .

[0070] The network 856 may be a WLAN operating in accordance with one or more Institute of Electrical and Electronics Engineers (“IEEE”) 802.11 standards, such as IEEE 802.11a, 802.11b, 802.11g, 802.11n, etc. (collectively referred to herein as Wi-Fi). In some configurations, the WLAN is implemented using one or more wireless Wi-Fi access points. In some configurations, one or more of the wireless Wi-Fi access points is another computing device with connectivity to a WWAN that acts as a Wi-Fi hotspot. The WLAN component 824 is configured to connect to the network 856 via a Wi-Fi access point. Such a connection may be protected by various encryption techniques, including but not limited to Wi-Fi Protected Access (“WPA”), WPA2, Wired Equivalent Privacy (“WEP”), etc.

[0071] The network 856 may be a WPAN operating in accordance with infrared data association ("Irda"), Bluetooth, wireless universal serial bus ("USB"), Z-wave, ZIGBEE, or some other short-range wireless technology. In some configurations, the WPAN component 826 is configured to facilitate communication with other devices (e.g., other devices) via the WPAN.

[0072] The sensor assembly 808 includes a magnetometer 828, a time-of-flight sensor 830 (e.g., a time-of-flight sensor array as described herein), a proximity sensor 832, an accelerometer 834, a gyroscope 836, and a global positioning system sensor (“GPS sensor”) 838. It is contemplated that other sensors such as, but not limited to, temperature sensors or shock detection sensors may also be incorporated into the computing device architecture 800.

[0073] The magnetometer 828 is configured to measure the strength and direction of the magnetic field. In some configurations, the magnetometer 828 provides measurements to a compass application stored in a component in the memory component 804 to provide the user with accurate directions in a reference system including the cardinal directions (north, south, east, and west). Similar measurements can be provided to a navigation application that includes a compass component.

[0074] The proximity sensor 832 is configured to detect the presence of an object or thing near the computing device without direct contact. In some configurations, the proximity sensor 832 detects the presence of a user's body (e.g., the user's face) and provides this information to an application stored in a memory in the memory component 804, and the application uses the proximity information to enable or disable some functions of the computing device. For example, a phone application can automatically disable the touch screen in response to receiving the proximity information so that the user's face does not inadvertently end a call or enable / disable other functions within the phone application during a call.

[0075] The accelerometer 834 is configured to measure appropriate accelerations. In some configurations, the output from the accelerometer 834 is used by the application as an input mechanism to control some functions of the application. For example, the application may be a video game in which a character, a portion of a character, or an object is moved or otherwise manipulated in response to input received via the accelerometer 834. In some configurations, the output from the accelerometer 834 is provided to the application for switching between landscape mode and portrait mode, calculating coordinate acceleration, or detecting a fall.

[0076] The gyroscope 836 is configured to measure direction and maintain orientation. In some configurations, the output from the gyroscope 836 is used by the application as an input mechanism to control some functions of the application. For example, the gyroscope 836 can be used for accurate recognition of motion within a 3D environment of a video game application or some other application. In some configurations, the application utilizes the output from the gyroscope 836 and the output of the accelerometer 834 to enhance control of some functions of the application.

[0077] The GPS sensor 838 is configured to receive signals from GPS satellites for calculating a position. The position calculated by the GPS sensor 838 can be used by any application that needs or benefits from position information. For example, the position calculated by the GPS sensor 838 can be used with a navigation application to provide directions from a location to a destination or from a destination to a location. The GPS sensor 838 can utilize one or more network connectivity components 806 to obtain location information generated via Wi-Fi, WIMAX, and / or cellular triangulation technology to help the GPS sensor 838 obtain a location fix.

[0078] The I / O component 810 includes a display 840, a touch screen 842, a data I / O interface component ("data I / O") 844, an audio I / O interface component ("audio I / O") 846, a video I / O interface component ("video I / O") 848, and a camera 850. In some configurations, the display 840 and the touch screen 842 are combined. In some configurations, two or more of the data I / O component 844, the audio I / O component 846, and the video I / O component 848 are combined. The I / O component 810 may include a discrete processor configured to support the various interfaces described below, or may include processing functionality built into the processor 802.

[0079] The display 840 is an output device configured to present information in a visual form. Specifically, the display 840 can present graphical user interface ("GUI") elements, text, images, videos, notifications, virtual buttons, virtual keyboards, message data, Internet content, device status, time, date, calendar data, preferences, map information, location information, and any other information that can be presented in a visual form. In some configurations, the display 840 is a liquid crystal display ("LCD") using any active matrix technology or passive matrix technology and any backlight technology (if used). In some configurations, the display 840 is an organic light emitting diode ("OLED") display.

[0080] The touch screen 842 (also referred to herein as a "touch-enabled screen") is an input device configured to detect the presence and location of a touch. The touch screen 842 may be a resistive touch screen, a capacitive touch screen, a surface acoustic wave touch screen, an infrared touch screen, an optical imaging touch screen, a dispersive signal touch screen, an acoustic pulse recognition touch screen, or may utilize any other touch screen technology. In some configurations, the touch screen 842 is incorporated on top of the display 840 as a transparent layer to enable a user to interact with objects or other information presented on the display 840 using one or more touches. In other configurations, the touch screen 842 is a touch pad incorporated on a surface of a computing device that does not include the display 840.

[0081] The data I / O interface component 844 is configured to facilitate inputting data to and outputting data from the computing device. In some configurations, the data I / O interface component 844 includes a connector configured to provide wired connectivity between the computing device and the computer system, such as for the purpose of synchronous operation. The connector can be a proprietary connector or a standardized connector such as USB, micro-USB, mini-USB, etc. In some configurations, the connector is a docking connector for docking the computing device with another device such as a docking station, an audio device (e.g., a digital music player), or a video device.

[0082] The audio I / O interface component 846 is configured to provide audio input and / or output capabilities to the computing device. In some configurations, the audio I / O interface component 846 includes a microphone configured to collect audio signals. In some configurations, the audio I / O interface component 846 includes a headphone jack configured to provide connectivity for headphones or other external speakers. In some configurations, the audio I / O interface component 846 includes a speaker for outputting audio signals. In some configurations, the audio I / O interface component 846 includes an optical audio cable output.

[0083] The video I / O interface component 848 is configured to provide video input and / or output capabilities to the computing device. In some configurations, the video I / O interface component 848 includes a video connector that is configured to receive video as input from another device or send video as output to another device (e.g., a monitor, television, or some other external display). In some configurations, the video I / O interface component 948 includes a High Definition Multimedia Interface (“HDMI”), mini-HDMI, micro-HDMI, DisplayPort, or a proprietary connector to input / output video content. In some configurations, the video I / O interface component 848 or a portion thereof is combined with the audio I / O interface component 846 or a portion thereof.

[0084] The camera 850 may be configured to capture still images and / or video. The camera 850 may utilize a charge coupled device (“CCD”) or a complementary metal oxide semiconductor (“CMOS”) image sensor to capture images. In some configurations, the camera 850 includes a flash to assist in taking pictures in low light environments. The settings of the camera 850 may be implemented as hardware or software buttons.

[0085] The illustrated power assembly 812 includes one or more batteries 852, which can be connected to a battery meter 854. The batteries 852 can be rechargeable or disposable. Rechargeable battery types include, but are not limited to, lithium polymer, lithium ion, nickel cadmium, and nickel metal hydride. Each battery in the battery 852 can be composed of one or more cells.

[0086] The battery meter 854 can be configured to measure battery parameters such as current, voltage, and temperature. In some configurations, the battery meter 854 is configured to measure the effects of the battery's discharge rate, temperature, aging, and other factors to predict the remaining life within a certain error percentage. In some configurations, the battery meter 854 provides the measurements to an application, which is configured to use the measurements to present useful power management data to the user. The power management data can include one or more of the percentage of battery used, the percentage of battery remaining, battery condition, remaining time, remaining capacity (e.g., in Watt-hours), current consumption, and voltage.

[0087] The power supply component 812 may also include a power connector, which may be combined with one or more components of the above-mentioned I / O component 810. The power supply component 812 may interface with an external power system or charging device via the I / O component.

[0088] The disclosure presented herein also includes the subject matter set forth in the following clauses.

[0089] Example clause A, an imaging system, comprising: a light emitting component configured to emit light to illuminate a scene; a light detecting component, the light detecting component comprising: a time-of-flight pixel array, wherein each time-of-flight pixel is configured to detect reflected light based on the emitted light, and (i) determine phase shift data based on a phase shift between the emitted light and the reflected light, and (ii) determine intensity data based on the amplitude of the reflected light; and a plurality of microlenses, wherein each microlens is shared by at least two time-of-flight pixels in the array; and a controller, the controller being configured to: generate a first depth image of the scene using phase shift data determined for each time-of-flight pixel in the array; determine disparity data using intensity data for each time-of-flight pixel in at least two time-of-flight pixels shared by the individual microlenses; generate a second depth image of the scene using disparity data determined for each microlens in the plurality of microlenses; and determine a distance between an object in the scene and the imaging system using at least one of the first depth image or the second depth image.

[0090] Example clause B, an imaging system as in clause A, wherein using at least one of the first depth image or the second depth image to determine the distance between an object in the scene and the imaging system includes: combining the first depth image and the second depth image by averaging the depth values ​​of the pixels to generate a representative depth image; and using the representative depth image to determine the distance between the object in the scene and the imaging system.

[0091] Example clause C, an imaging system as in clause A, wherein using at least one of the first depth image or the second depth image to determine the distance between an object in the scene and the imaging system includes: comparing a first depth quality of the first depth image with a second depth quality of the second depth image; determining that one of the first depth quality or the second depth quality is better than the other of the first depth quality or the second depth quality; and selecting one of the first depth image or the second depth image to determine the distance between the object in the scene and the imaging system based at least in part on determining that one of the first depth quality or the second depth quality is better than the other of the first depth quality or the second depth quality.

[0092] Example Clause D. An imaging system as in Clause C, wherein the first depth quality of the first depth image and the second depth quality of the second depth image are based on a segmentation analysis of uniformity and edge sharpness.

[0093] Example clause E. An imaging system as in any of example clauses A to D, wherein the distance is used to focus on an object in a scene with a camera lens.

[0094] Example clause F. An imaging system as in any of example clauses A to D, wherein the distance is used to select a preconfigured mode that can be used to capture a photograph of an object in a scene.

[0095] Example clause G. The imaging system of any of example clauses A to F, wherein the imaging system is configured as an integrated circuit, the integrated circuit being part of the device.

[0096] Example Clause H, an imaging system as in Clause G, wherein the device uses distance to display virtual content in association with objects in the scene.

[0097] Example Item I, a method comprising: emitting light to illuminate a scene; detecting reflected light based on the emitted light by a time-of-flight pixel array, wherein the time-of-flight pixel array includes a plurality of microlenses and each microlens is shared by at least two time-of-flight pixels; determining by a single time-of-flight pixel in the array (i) phase shift data based on a phase shift between the emitted light and the reflected light, and (ii) intensity data based on the amplitude of the reflected light; generating a first depth image of the scene using the phase shift data determined for each time-of-flight pixel in the array; determining disparity data using the intensity data of each time-of-flight pixel of at least two time-of-flight pixels shared by the individual microlenses; generating a second depth image of the scene using the disparity data determined for each microlens of the plurality of microlenses; and determining a distance between an object in the scene and an imaging system including the time-of-flight pixel array using the first depth image or at least one of the second depth images.

[0098] Example clause J, a method as described in Example clause I, wherein using at least one of the first depth image or the second depth image to determine the distance between an object in the scene and the imaging system includes: combining the first depth image and the second depth image by averaging the depth values ​​of the pixels to generate a representative depth image; and using the representative depth image to determine the distance between the object in the scene and the imaging system.

[0099] Example clause K, a method as described in Example clause I, wherein using at least one of the first depth image or the second depth image to determine the distance between an object in the scene and the imaging system includes: comparing a first depth quality of the first depth image with a second depth quality of the second depth image; determining that one of the first depth quality or the second depth quality is better than the other of the first depth quality or the second depth quality; and at least partially determining that based on one of the first depth quality or the second depth quality being better than the other of the first depth quality or the second depth quality, select one of the first depth image or the second depth image to determine the distance between the object in the scene and the imaging system.

[0100] Example Clause L, the method of Example Clause K, wherein the first depth quality of the first depth image and the second depth quality of the second depth image are based on a segmentation analysis of uniformity and edge sharpness.

[0101] Example clause M, a method as in any of example clauses I to L, wherein the distance is used to focus the camera lens on an object in the scene.

[0102] Example clause N, a method as in any of example clauses I to L, wherein the distance is used to select a preconfigured mode that can be used to capture a photograph of an object in a scene.

[0103] Example clause O, a method as in any of example clauses I to N, wherein the imaging system is configured as an integrated circuit, the integrated circuit being part of a device including the camera.

[0104] Example clause P, the method of example clause O, wherein the device uses distance to display virtual content in association with objects in the scene.

[0105] Example clause Q, an imaging system, comprising: a light emitting component configured to emit light to illuminate a scene; a light detecting component, the light detecting component comprising: a time-of-flight pixel array, wherein each time-of-flight pixel is configured to detect reflected light based on the emitted light, and (i) determine phase shift data based on a phase shift between the emitted light and the reflected light, and (ii) determine intensity data based on the amplitude of the reflected light; and a plurality of microlenses, wherein each microlens is shared by at least two time-of-flight pixels in the array; and a controller, the controller being configured to: determine disparity data using intensity data of each time-of-flight pixel of at least two time-of-flight pixels shared by the individual microlenses; and determine the distance between objects in the scene based in part on at least one of the phase shift data determined for each time-of-flight pixel in the array or the disparity data determined for each microlens in the plurality of microlenses.

[0106] Example clause R, the imaging system of example clause Q, wherein the distance is used to focus the camera lens on an object in the scene.

[0107] Example clause S, the imaging system of example clause Q, wherein the distance is used to select a preconfigured mode that can be used to capture a photograph of an object in a scene.

[0108] Example clause T, the imaging system of example clause Q, the imaging system being configured as an integrated circuit, the integrated circuit being part of a device, wherein the device uses distance to display virtual content in association with objects in a scene.

[0109] The terms "a," "an," "the," and similar referents used in the context of describing the invention (especially in the context of the following claims) should be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms "based on," "based on," and similar referents should be construed to mean "based at least in part on," which includes "based in part on" and "based entirely on," unless otherwise indicated herein or clearly contradicted by context.

[0110] It should be understood that any reference to "first", "second", etc. elements in the Summary of the Invention and / or the Detailed Description is not intended to and should not be interpreted as necessarily corresponding to any reference to "first", "second", etc. elements in the claims. On the contrary, any use of "first" and "second" in the Summary of the Invention, the Detailed Description, and / or the Claims can be used to distinguish two different examples of the same element (e.g., two different depth images, two different ToF pixels, etc.).

[0111] It should be understood that, unless otherwise specifically stated, conditional language used herein, such as "may," "can," "might," or "could" is understood in context to mean that certain examples include certain features, elements, and / or steps, while other examples do not include certain features, elements, and / or steps. Therefore, such conditional language is generally not intended to imply that one or more examples require certain features, elements, and / or steps in any way, or that one or more examples must include logic for determining whether to include or perform certain features, elements, and / or steps in any particular example with or without user input or prompting. Unless otherwise explicitly stated, connection language such as the phrase "at least one of X, Y, or Z" should be understood to mean that an item, term, etc. can be X, Y, or Z, or a combination thereof.

[0112] It should also be understood that many variations and modifications may be made to the above examples, and the elements of these variations and modifications should be understood as other acceptable examples. All these modifications and variations are intended to be included within the scope of this disclosure and protected by the accompanying claims.

[0113] Finally, although various configurations have been described using language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the accompanying representations is not necessarily limited to the specific features or acts described. Instead, these specific features and acts are disclosed as example forms of implementing the claimed subject matter.

Claims

1. An imaging system configured to use time-of-flight imaging and stereoscopic imaging, the imaging system include: a light emitting assembly configured to emit a first light to illuminate a scene; A light detection component, the light detection component comprising: Time-of-flight pixel array, where Each time-of-flight pixel is configured to detect a second light reflected from the first light and determine (i) phase shift data based on a phase shift between the first light and the second light and (ii) intensity data based on an amplitude of the second light; and Each time-of-flight pixel includes a single photodiode and at least one of: (i) two transfer gates, (ii) two photogates; and a plurality of microlenses, wherein each microlens is shared by at least two time-of-flight pixels in the array; and A controller, the controller being configured to: determining disparity data using the intensity data for each of the at least two time-of-flight pixels shared by an individual microlens; and Distances between objects in the scene are determined based in part on at least one of the phase shift data determined for each time-of-flight pixel in the array or the disparity data determined in association with the individual microlenses in the plurality of microlenses.

2. The imaging system of claim 1, wherein the distance is used to focus a camera lens on the object in the scene.

3. The imaging system of claim 1, wherein the distance is used to select a preconfigured mode that can be used to capture a photograph of the object in the scene. 4 . The imaging system of claim 1 , configured as an integrated circuit that is part of a device that uses the distance to display virtual content in association with the object in the scene.

5. The imaging system of claim 1, wherein the distance is used to improve motion recognition for an application executing on a connected computing system.

6. The imaging system of claim 1, wherein when the object is within a particular distance of the imaging system, the distance is determined using the phase shift data determined for each time-of-flight pixel in the array.

7. The imaging system of claim 1, wherein when the object is located outside a specific distance of the imaging system, the distance is determined using the disparity data determined in association with the individual microlenses of the plurality of microlenses.

8. A method for using time-of-flight imaging and stereoscopic imaging, the method include: emitting a first light to illuminate a scene; detecting a second light emission from the first light by a time-of-flight pixel array, wherein: Each time-of-flight pixel includes a single photodiode and at least one of: (i) two transfer gates, (ii) two photogates; and The time-of-flight pixel array includes a plurality of microlenses, and each microlens is shared by at least two time-of-flight pixels; determining, by individual time-of-flight pixels in the array, (i) phase shift data based on a phase shift between the first light and the second light, and (ii) intensity data based on an amplitude of the second light; determining disparity data using the intensity data for each of the at least two time-of-flight pixels shared by an individual microlens; and Distances between objects in the scene are determined based in part on at least one of the phase shift data determined for each time-of-flight pixel in the array or the disparity data determined in association with the individual microlenses in the plurality of microlenses.

9. The method of claim 8, wherein the distance is used to focus a camera lens on the object in the scene.

10. The method of claim 8, wherein the distance is used to select a preconfigured mode that can be used to capture a photograph of the object in the scene. The method of claim 8 , wherein the distance is used to display virtual content in association with the object in the scene.

12. The method of claim 8, wherein the distance is used to improve motion recognition for an application executing on a connected computing system.

13. The method of claim 8, wherein when the object is within a particular distance of the imaging system, the distance is determined using the phase shift data determined for each time-of-flight pixel in the array.

14. The method of claim 8, wherein when the object is located outside a certain distance of the imaging system, the distance is determined using the disparity data determined in association with the individual microlenses of the plurality of microlenses.

15. One or more non-transitory computer storage media storing instructions that, when executed by an imaging system, cause the imaging system to perform operations, the operations include: determining, by individual time-of-flight pixels in the time-of-flight pixel array, (i) phase shift data based on a phase shift between first light emitted by the imaging system to illuminate a scene and second light detected based on reflection of the first light, and (ii) intensity data based on an amplitude of the second light; determining disparity data using the intensity data for each of at least two time-of-flight pixels shared by individual microlenses in a plurality of microlenses included in the time-of-flight pixel array; and Distances between objects in the scene are determined based in part on at least one of the phase shift data determined for each time-of-flight pixel in the array or the disparity data determined in association with the individual microlenses in the plurality of microlenses.

16. The one or more non-transitory computer storage media of claim 15, wherein the distance is used to focus a camera lens on the object in the scene.

17. The one or more non-transitory computer storage media of claim 15, wherein the distance is used to select a preconfigured mode that can be used to capture a photograph of the object in the scene.

18. The one or more non-transitory computer storage media of claim 15, wherein the distance is used to display virtual content in association with the object in the scene.

19. The one or more non-transitory computer storage media of claim 15, wherein when the object is within a particular distance of the imaging system, the distance is determined using the phase shift data determined for each time-of-flight pixel in the array.

20. The one or more non-transitory computer storage media of claim 15, wherein when the object is located outside a particular distance of the imaging system, the distance is determined using the disparity data determined in association with the individual microlenses of the plurality of microlenses.

Images