Depth-sensing techniques for virtual, augmented, and mixed reality systems

Abstract

To provide depth sensing techniques for virtual, augmented and mixed reality systems.SOLUTION: The invention provides a system and method for operating a sensor which has at least two modes of operation. The sensor may be provided with a sequence of common operation steps which are included in both a first sequence of operation steps which define a first mode of operation and a second sequence of operation steps which define a second mode of operation. The sensor may also be provided with one or more dummy operation steps which relate to the difference between the first mode of operation and the second mode of operation. The sensor can be operated in the first mode of operation by causing the sensor to execute at least the common operation steps, and the sensor can be operated in the second mode of operation by causing the sensor to execute the common operation steps and at least one dummy operation step.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] (Citation of Related Applications) Any and all applications for foreign and domestic priority identified in the application data sheet filed with respect to this application are hereby incorporated by reference into this specification under 37 CFR 1.57. That is, this application claims priority to U.S. Provisional Patent Application No. 62 / 474,503, filed on March 21, 2017, entitled "DEPTH SENSING TECHNIQUES FOR VIRTUAL, AUGMENTED, AND MIXED REALITY SYSTEMS", and the above application is hereby incorporated by reference into this specification in its entirety. (Field)

[0002] This disclosure relates to depth sensors such as those that can be used in imaging and visualization systems for virtual reality, augmented reality, and mixed reality.

Background Art

[0003] Modern computing and display technologies are facilitating the development of virtual reality, augmented reality, and mixed reality systems. A virtual reality or "VR" system creates a simulated environment for a user to experience. This can be done by presenting computer-generated images to the user through a head-mounted display. This image generates a sensory experience that immerses the user in the simulated environment. Virtual reality scenarios typically also involve not only the presentation of computer-generated images but also real-world images.

[0004] Augmented reality systems generally complement the real-world environment with simulated elements. For example, an augmented reality or "AR" system can provide a user with a view of the surrounding real-world environment via a head-mounted display. However, computer-generated images can also be presented on the display to enhance the real-world environment. These computer-generated images can include elements that are contextually relevant to the real-world environment. Such elements may include simulated text, images, objects, etc. Mixed reality or "MR" systems are a type of AR system that also introduces simulated objects into the real-world environment, but these objects typically feature a greater degree of interaction. The simulated elements can often be real-time and interactive.

[0005] Figure 1 depicts an exemplary AR / MR scene 1, in which the user is viewing a real-world park-like setting 6 featuring people, trees, buildings in the background, and a concrete platform 20. In addition to these items, computer-generated images are also presented to the user. The computer-generated images may include, for example, a robot figure 10 standing on the real-world platform 20, and a flying cartoon-like avatar character 2 that looks like an anthropomorphic bumblebee, even if these elements 2 and 10 do not actually exist in the real-world environment. [Overview of the Initiative] [Means for solving the problem]

[0006] In some embodiments, a method for operating a sensor having at least two operating modes includes providing the sensor with a set of common operating steps that are included in both a first set of operating steps defining a first operating mode and a second set of operating steps defining a second operating mode; providing the sensor with one or more dummy operating steps relating to the difference between the first operating mode and the second operating mode; operating the sensor in the first operating mode by having the sensor perform at least the common operating steps; and operating the sensor in the second operating mode by having the sensor perform the common operating steps and at least one dummy operating step.

[0007] In some embodiments, the sensor may be a depth sensor. A first operating mode may include a depth sensing mode with a first frame rate, and a second operating mode may be a depth sensing mode with a second frame rate that is slower than the first frame rate. For example, one or more of the dummy operating steps may include a delay.

[0008] In some embodiments, a system for operating a sensor having at least two operating modes includes a processor configured to perform a method comprising: providing the sensor with a set of common operating steps that are included in both a first set of operating steps defining a first operating mode and a second set of operating steps defining a second operating mode; providing the sensor with one or more dummy operating steps relating to the difference between the first operating mode and the second operating mode; and causing the sensor to operate in the first operating mode by having the sensor perform at least the common operating steps; and causing the sensor to operate in the second operating mode by having the sensor perform the common operating steps and at least one dummy operating step. The present invention provides, for example, the following: (Item 1) A method for operating a sensor, wherein the method is To provide the sensor with a series of common operation steps that are included in both a first series of operation steps defining a first operation mode and a second series of operation steps defining a second operation mode, To provide the sensor with one or more dummy operation steps relating to the difference between the first operation mode and the second operation mode, The sensor is made to operate in the first operating mode by causing it to perform at least the common operation steps, The sensor is made to operate in the second operating mode by causing it to perform the common operation step and at least one dummy operation step. Methods that include... (Item 2) The method according to item 1, wherein the first operating mode includes performing at least the common operating step at a first rate, and the second operating mode includes performing the common operating step and at least one dummy operating step at a second rate that is slower than the first rate. (Item 3) The method according to item 2, wherein at least one of the one or more dummy operation steps includes a delay. (Item 4) The method according to item 1, wherein providing the series of common operation steps and the one or more dummy operation steps to the sensor includes storing those operation steps in the sensor memory. (Item 5) The method according to item 4, wherein switching the sensor between the first operating mode and the second operating mode does not require any additional actions to store the operating steps in the sensor memory. (Item 6) The method according to item 1, wherein the sensor is equipped with a depth sensor. (Item 7) The method according to item 6, wherein the depth sensor is equipped with a time-of-flight camera. (Item 8) The method according to item 6, wherein the first operating mode includes a depth sensing mode with a first frame rate, and the second operating mode includes a depth sensing mode with a second frame rate that is slower than the first frame rate. (Item 9) The method according to item 6, further comprising providing depth information from the depth sensor to a virtual reality, augmented reality, or mixed reality display system. (Item 10) A system for operating a sensor, the system comprising a processor configured to perform a method, the method being To provide the sensor with a series of common operation steps that are included in both a first series of operation steps defining a first operation mode and a second series of operation steps defining a second operation mode, To provide the sensor with one or more dummy operation steps relating to the difference between the first operation mode and the second operation mode, The sensor is made to operate in the first operating mode by causing it to perform at least the common operation steps, The sensor is made to operate in the second operating mode by causing it to perform the common operation step and at least one dummy operation step. A system that includes this. (Item 11) The system according to item 10, wherein the first operating mode includes performing at least the common operating step at a first rate, and the second operating mode includes performing the common operating step and at least one dummy operating step at a second rate that is slower than the first rate. (Item 12) The system according to item 11, wherein at least one of the one or more dummy operation steps includes a delay. (Item 13) Providing the series of common operation steps and the one or more dummy operation steps to the sensor includes storing those operation steps in a sensor memory, the system according to item 10. (Item 14) Switching the sensor between the first operation mode and the second operation mode does not require any additional act for storing operation steps in the sensor memory, the system according to item 13. (Item 15) The sensor includes a depth sensor, the system according to item 10. (Item 16) The depth sensor includes a time-of-flight camera, the system according to item 15. (Item 17) The first operation mode includes a depth sensing mode with a first frame rate, and the second operation mode includes a depth sensing mode with a second frame rate that is slower than the first frame rate, the system according to item 15. (Item 18) The system is integrated within a virtual reality, augmented reality, or mixed reality display system, the system according to item 15. (Item 19) The processor includes a state machine, the system according to item 10. (Item 20) The system further includes an arbiter for receiving requests to operate the sensor in the first mode or the second mode, and the arbiter is configured to schedule and prioritize the requests, the system according to item 10. (Item 21) The system further includes the sensor, the system according to item 10. (Item 22) A method of operating a depth sensor, the method comprising: Receiving a first request for a first type of depth measurement; Receiving a second request for a second type of depth measurement; Assigning a first priority to the first request, Assigning a second priority to the second request, If the first priority is higher than the second priority, causing the depth sensor to first acquire the first type of depth measurement, or if the second priority is higher than the first priority, causing the depth sensor to first acquire the second type of depth measurement A method comprising. (Item 23) The method according to item 22, wherein the first priority is assigned based on the priority of a first application requesting the first type of depth measurement, and the second priority is assigned based on the priority of a second application requesting the second type of depth measurement. (Item 24) A system for operating a depth sensor, the system comprising: An arbiter configured to receive a first request for a first type of depth measurement and a second request for a second type of depth measurement, the arbiter being configured to assign a first priority to the first request and a second priority to the second request, the arbiter; A processor Comprising, The system, wherein the processor is configured to cause the depth sensor to first acquire the first type of depth measurement if the first priority is higher than the second priority, or to cause the depth sensor to first acquire the second type of depth measurement if the second priority is higher than the first priority. (Item 25) The system according to item 24, wherein the arbiter is configured to assign the first priority based on the priority of a first application requesting the first type of depth measurement and to assign the second priority based on the priority of a second application requesting the second type of depth measurement. (Item 26) The system described in item 24 is integrated within a virtual, augmented, or mixed reality display system. (Item 27) A method for operating a depth sensor, wherein the method is Performing configuration operations for a depth sensor, the configuration operations being: A first set of operation steps defining a first depth sensing operation mode is stored in the memory of the depth sensor, The depth sensor's memory stores a second set of operation steps that define a second depth sensing operation mode. This includes, Receiving a first request for depth measurement in accordance with the first depth sensing operation mode, In response to the first request, the depth sensor is made to perform the first series of operation steps, thereby operating the depth sensor in the first operating mode, Receiving a second request for depth measurement in accordance with the second depth sensing operation mode, In response to the second request, the depth sensor is made to operate in the second operating mode by causing it to perform the second set of operating steps without performing any additional configuration operations. Methods that include... (Item 28) The depth sensor is equipped with a time-of-flight camera, as described in item 27. (Item 29) The method according to item 27, wherein the first depth sensing operation mode corresponds to a first measurement distance range, and the second depth sensing operation mode corresponds to a second measurement distance range different from the first measurement distance range. (Item 30) The method according to item 27, wherein the first depth sensing operation mode corresponds to a first frame rate, and the second depth sensing operation mode corresponds to a second frame rate that is slower than the first frame rate. (Item 31) The method according to item 27, further comprising providing depth information from the depth sensor to a virtual reality, augmented reality, or mixed reality display system. (Item 32) A system for operating a depth sensor, the system comprising a processor configured to perform a method, the method being Performing a configuration operation for a depth sensor, the configuration operation including storing a first set of operation steps defining a first depth sensing operation mode in the memory of the depth sensor, and storing a second set of operation steps defining a second depth sensing operation mode in the memory of the depth sensor. Receiving a first request for depth measurement in accordance with the first depth sensing operation mode, In response to the first request, the depth sensor is made to perform the first series of operation steps, thereby operating the depth sensor in the first operating mode, Receiving a second request for depth measurement in accordance with the second depth sensing operation mode, In response to the second request, the depth sensor is made to operate in the second operating mode by causing it to perform the second set of operating steps without performing any additional configuration operations. A system that includes this. (Item 33) The depth sensor is a system according to item 32, comprising a time-of-flight camera. (Item 34) The system according to item 32, wherein the first depth sensing operation mode corresponds to a first measurement distance range, and the second depth sensing operation mode corresponds to a second measurement distance range different from the first measurement distance range. (Item 35) The system according to item 32, wherein the first depth sensing operation mode corresponds to a first frame rate, and the second depth sensing operation mode corresponds to a second frame rate that is slower than the first frame rate. (Item 36) The system described in item 32 is integrated within a virtual, augmented, or mixed reality display system. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 illustrates the user's field of view in an augmented reality (AR) scenario using an exemplary AR system.

[0010] [Figure 2] Figure 2 illustrates an example of a wearable VR / AR / MR display system.

[0011] [Figure 3] Figure 3 illustrates an exemplary depth sensing system.

[0012] [Figure 4] Figure 4 illustrates an example of an improved method for efficiently operating a depth sensor in multiple depth sensing modes.

[0013] [Figure 5] Figure 5 is an illustrative schematic diagram illustrating how to efficiently operate a depth sensor in multiple depth sensing modes.

[0014] [Figure 6] Figure 6 illustrates another example of an improved method for efficiently operating a depth sensor in multiple depth sensing modes.

[0015] [Figure 7] Figure 7 is an illustrative table showing common and dummy operation steps for multiple depth sensing modes.

[0016] [Figure 8] Figure 8 is an illustrative table illustrating how the common operation steps and dummy operation steps in Figure 7 can be used to operate efficiently in multiple depth sensing modes.

[0017] [Figure 9] Figure 9 is an illustrative timing diagram for operation in high dynamic range (HDR) depth sensing mode. [Modes for carrying out the invention]

[0018] Virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems may include displays that present computer-generated images to a user. In some embodiments, the display system is wearable, which, as an advantage, can provide a more immersive VR / AR / MR experience. Computer-generated images presented through a display can create the impression of being three-dimensional. This can be done, for example, by presenting stereoscopic images to the user.

[0019] Figure 2 illustrates an example of a wearable VR / AR / MR display system 80. The VR / AR / MR display system 80 includes a display 62 and various mechanical and electronic modules and systems to support the functions of the display 62. The display 62 may be coupled to a frame 64, which is wearable by a wearable user 60 and positions the display 62 in front of the user's eyes. A speaker 66 may be coupled to the frame 64 and positioned adjacent to the user's ear canal. Another speaker, not shown, may be positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control. The display 62 is operably coupled to a local data processing module 70, which can be mounted in various configurations, such as fixedly attached to the frame 64 by wired or wireless connectivity 68, fixedly attached to a helmet or hat worn by the user, built into headphones, or otherwise detachably attached to the user 60 (e.g., in a backpack configuration, in a belt-mounted configuration, etc.).

[0020] The local processing and data module 70 may include a processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be used to assist in the processing and storage of data. This includes data captured from sensors such as image acquisition devices (e.g., cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes. Sensors may be operably coupled to frame 64 or otherwise attached to user 60. In some embodiments, all data is stored and all calculations are performed within the local processing and data module to enable fully autonomous use. Alternatively, or in addition, sensor data may be retrieved and / or processed using the remote processing module 72 and / or remote data repository 74. The local processing and data module 70 may be operably coupled to the remote processing module 72 and the remote data repository 74 by communication links (76, 78) via wired or wireless communication links, etc., so that these remote modules (72, 74) are operably coupled to each other and available as resources to the local processing and data module 70. In some embodiments, the remote processing module 72 may include one or more processors configured to analyze and process data (e.g., sensor data and / or image information). The remote data repository 74 may comprise a digital data storage facility, which may be available through the Internet or other networking configurations in a “cloud” resource configuration.

[0021] The VR / AR / MR system 80 may also include a depth sensor 100. The depth sensor 100 measures the user's surroundings and determines information about the distances to various objects and features present within those surroundings. VR / AR / MR applications can utilize various types of depth information, including short-range depth information (e.g., 0-2 meters), long-range depth information (e.g., 2-4 meters and beyond), and high dynamic range (HDR) depth information. The depth information provided by the depth sensor 100 can be used to enable the user to interact with the VR / AR / MR system and / or to enable the system to project virtual images into the user's real-world environment.

[0022] One application of long-range depth sensing in VR / AR / MR systems is to use depth information to model the user's environment. For example, depth sensor 100 can be used to determine the distance to walls and objects in a room. The resulting depth information can be used to create a 3D model of the room and its contents. In AR / MR systems in particular, this can enable the system to project virtual images into the room in a realistic and interactive manner. An exemplary application of short-range depth sensing in VR / AR / MR systems is gesture recognition. For example, VR / AR / MR system 80 can use depth sensing to track the user's hand movements to facilitate gesture recognition. VR / AR / MR system 80 can then perform an action in response to the user's gesture.

[0023] Given that depth information can be used by the VR / AR / MR system 80 to provide the user with an interactive, immersive experience, it is advantageous for the depth sensor 100 to collect depth information relatively quickly and efficiently. This is because it allows the VR / AR / MR system 80 to be more responsive. This is particularly true for AR / MR applications, as AR / MR applications can be highly sensitive to discontinuities between real-world content surrounding the user and virtual content projected into the user's environment by the system 80. This disclosure therefore describes improved techniques that can increase the efficiency and / or speed at which various depth-sensing information can be collected.

[0024] As background, one type of depth sensor is the 3D time-of-flight (TOF) camera. Generally, a 3D TOF camera uses a light source to illuminate a scene. The TOF camera then observes and processes the light reflected from the scene to determine information about the distance to various points / objects / features in the scene. Some TOF cameras perform depth measurement by emitting pulses of infrared light toward one or more points in the scene and then measuring the elapsed time until the light is reflected from the scene. Based on the elapsed time combined with knowledge of the speed of light, the camera can then determine the distance the light has traveled. In addition, some TOF cameras can perform depth measurement by emitting a modulated light signal (e.g., square or sine wave) and then measuring the phase shift between the illumination light signal and the reflected light signal. These phase shift measurements are then converted into distance measurements.

[0025] In most depth-sensing TOF cameras, illumination comes from a solid-state laser or light-emitting diode (LED) operating in the near-infrared range (e.g., about 850 nm), which is invisible to the human eye. Typically, the illumination from the light source into the scene is designed to be relatively uniform. An imaging sensor, designed to respond to the same spectrum as the illumination light, receives the light reflected from the scene and converts the light into an electrical signal. In some embodiments, the imaging sensor can be a CCD or CMOS sensor with a resolution of, for example, 224 × 172 pixels, although imaging sensors with larger or smaller resolutions can also be used. Each pixel is located at a point in the image plane, which corresponds to a distinct point in object space or scene within the TOF camera's field of view. Thus, the information collected at each pixel of the imaging sensor can be used to determine the distance to the point in the scene corresponding to that particular pixel.

[0026] The light received by each pixel of the imaging sensor has an ambient component and a reflected component. Depth information is incorporated only into the reflected component. To distinguish between these two components, the TOF camera may capture an image of ambient infrared light immediately before or after actively illuminating the scene with infrared light. This image of ambient infrared light may be called an intensity subframe image. By subtracting or otherwise removing the intensity subframe image from other subframe images collected while the scene is actively illuminating, the depth sensor 100 can distinguish the reflected component of infrared light from background noise in the scene.

[0027] To enable the detection of phase shifts between illumination and reflection components, the signal from the light source can be modulated. For example, a square wave modulated signal can be used. The image sensor then detects the reflected light at several different times corresponding to different phase shifts relative to the modulated signal. The different phase shifts could be, for example, angles 1, 2, 3, and 4, where angle 2 = angle 1 + Δ, angle 3 = angle 1 + 2Δ, and angle 4 = angle 1 + 3Δ, where angles 1 and Δ are predetermined angles. For example, angle 1 could be 0° and Δ could be 90°, thereby allowing the camera to detect reflected light received at each pixel during periods phase-shifted by 0°, 90°, 180°, and 270° relative to the modulated signal. Each of these measurements can result in a separate phase subframe image captured by the camera sensor. The distance to a point in the scene corresponding to each sensor pixel can then be calculated from the four phase subframes using mathematical equations known in the art. Therefore, each complete frame of depth information (from which a set of depth measurements (one per pixel) can be determined) consists of several subframes of the image data.

[0028] The modulated illumination signal is periodic and therefore automatically repeats every 360° of phase shift. Thus, the fact that some TOF cameras measure depth based on the phase shift of reflected light relative to the modulated illumination signal means that the measured distance will suffer from aliasing effects. These aliasing effects can result in ambiguity in the measured distance. The distance at which aliasing occurs (i.e., the ambiguity distance) is also the maximum unambiguous distance that the TOF camera can measure. The maximum measurable distance can be extended by reducing the modulation frequency of the illumination light, but this may come at the expense of reduced depth-measuring resolution. To resolve depth ambiguity without compromising depth-measuring resolution, the TOF camera can modulate the illumination light using two or more distinct modulation signals with different frequencies (e.g., Fmod0 and Fmod1). Depth measurement is performed by measuring the phase shift of reflected light for each of the multiple modulation frequencies. Since each modulation frequency is different, each will have a different ambiguity distance. The actual distance to a given point in the scene is the distance at which measurements taken using different modulation frequencies agree.

[0029] In a TOF camera, distance can be measured for each pixel within the camera sensor. This results in a depth map of the scene within the camera's field of view. The depth map is a collection of points in 3D space, i.e., voxels, where each voxel is located at a distance measured by the corresponding sensor pixel. The depth map can be rendered in 3D space as a collection of points, i.e., a point cloud. The 3D points can be mathematically connected to form a mesh. The mesh can be used to model a scene, detect objects, etc. In addition, virtual content can be mapped onto the mesh by VR / AR / MR systems, providing lifelike 3D virtual content that interacts with the user's real-world surroundings.

[0030] Various types of depth measurement may be advantageous for different purposes in a VR / AR / MR system 80. For example, short-range, low-frame-rate depth measurement may be sufficient for detection when the user's hand is within the field of view of the depth sensor 100. Once the fact that the user's hand is within the field of view of the depth sensor is detected, short-range, high-frame-rate depth measurement may be more useful for tracking the movement of the user's hand and thereby detecting the specific gesture being made. On the other hand, long-range depth measurement at low or high frame rates may be useful for mapping the user's environment. In addition, high dynamic range (HDR) depth measurement from short to long range may also be beneficial.

[0031] Given that many different types of depth measurement may be useful in the VR / AR / MR system 80, the depth sensor 100 can include multiple operating modes for collecting each of these different types of depth measurement. Each mode may consist of, for example, a series of operations to be performed by the depth sensor 100. Depending on the mode, each of these operations may be accompanied by different settings or parameters such as exposure time, illumination light intensity, and illumination modulation frequency. The following table illustrates exemplary series of operations and configuration settings for several depth sensing modes.

[0032] Table 1 illustrates an exemplary set of operations for a short-range, high-frame-rate depth sensing mode. In some embodiments, this operating mode is used to sense depth in the range of less than about 2 meters (depending on the modulation frequency and exposure time) using a frame rate above about 20 Hz. In this particular embodiment, the frame rate is 45 Hz, which means that one full frame of depth information is captured every 22.22 ms (1 / 45 s). In this case, each full frame of depth information is based on an intensity subframe (to measure ambient infrared light while the illuminator is off) and four phase subframes (captured while the illuminator is modulated). [Table 1]

[0033] An exemplary sequence of operations for a short-range, high-frame-rate depth-sensing mode begins with step 0, which acquires an intensity subframe. Then, during steps 1-4, four phase subframes are captured. For short-range measurements, the exposure time for each of these subframes (i.e., the time while the image sensor captures light) is typically less than approximately 0.5 ms. Each subframe includes an associated readout time for transferring the captured image data from the image sensor. The readout time is typically less than approximately 1 ms.

[0034] The short-range, high-frame-rate operating mode may optionally include a relatively short delay as step 5 of the operating sequence. This delay may be, for example, equal to the difference between the 22.22 ms duration of the operating sequence and the total time required to complete steps 0-4. In other words, the optional short delay in step 5 may occupy any additional time during the operating sequence that is not required to capture and read the intensity subframes and the four phase subframes. Table 1 lists the specific order of the operating steps for this particular depth-sensing mode, although the operating steps may be performed in a different order as an alternative. The same applies to other operating modes described herein.

[0035] Table 2 illustrates an exemplary set of operations for a short-range, low-frame-rate depth sensing mode. This operating mode can be used to sense depth in the range of less than approximately 2 meters (depending on the modulation frequency and exposure time) using a frame rate of less than approximately 20 Hz. In this particular embodiment, the frame rate is 8 Hz, which means that one complete frame of depth information is captured every 125 ms. As in the previous case, each complete frame of depth information is based on an intensity subframe and four phase subframes. While the short-range, high-frame-rate mode has the advantage of producing depth measurements with better temporal resolution, the short-range, low-frame-rate mode can be beneficial because it is less computationally intensive when a lower temporal resolution is appropriate for the task at hand, thereby allowing the system to enter a low-power mode and conserve energy. [Table 2]

[0036] An exemplary sequence of operations for the short-range, low-frame-rate mode begins with step 0, which acquires an intensity subframe. Then, during steps 1-4, four phase subframes are captured. Again, for short-range measurements, the exposure time for each of these subframes is typically less than approximately 0.5 ms, and the reading time for each subframe is typically less than approximately 1 ms. Steps 0-4 in Table 2 are identical to steps 0-4 in Table 1. Thus, the short-range, low-frame-rate and short-range, high-frame-rate operating modes have these five steps in common.

[0037] However, the short-range, low-frame-rate operating mode includes a relatively long delay as step 5 of the operating sequence. This delay can be, for example, equal to the difference between the 125 ms duration of the operating sequence and the total time required to complete steps 0-4. The relatively long delay of step 5 occupies time during the operating sequence that is not required to capture and read the intensity subframes and the four phase subframes. Thus, the difference between the two short-range operating modes shown in Tables 1 and 2 relates to the difference between the relatively long delay of step 5 in Table 2 and the optional relatively short delay of step 5 in Table 1.

[0038] Table 3 illustrates an exemplary set of operations for a long-range, high-frame-rate depth sensing mode. This operating mode can be used to sense depth in the range of approximately 2–4 meters (depending on the modulation frequency and exposure time), for example, using a frame rate above approximately 20 Hz. Similar to short-range depth data, each complete frame of long-range depth information is based on several subframes of the image data. Again, there is an intensity subframe for measuring ambient infrared light while the illumination source is off. However, for long-range depth data, there are eight phase subframes of the image data, i.e., four phase subframes for each of the two illumination modulation frequencies Fmod1 and Fmod2. [Table 3]

[0039] An exemplary sequence of operations for the long-range, high-frame-rate depth-sensing mode begins with step 0, which acquires an intensity subframe. Then, during steps 1–4, four phase subframes for a first modulation frequency Fmod1 are captured, while during steps 5–8, four subframes for a second modulation frequency Fmod2 are captured. For long-range measurements, the exposure time for each of these subframes (i.e., the time while the image sensor captures light) is longer than for short-range measurements, typically 2–3 ms. (Other parameters or settings for long-range subframes may also differ from those for short-range subframes.) Each subframe also includes an associated readout time of approximately 1 ms for transferring the captured image data from the image sensor.

[0040] The long-range, high-frame-rate operating mode may optionally include a relatively short delay as step 9 of the operating sequence. This delay may be, for example, equal to the difference between the duration of the operating sequence and the total time required to complete steps 0-8. In other words, the optional short delay in step 9 may occupy any additional time during the duration of the operating sequence that is not required to capture and read the intensity subframes and the eight phase subframes.

[0041] Table 4 illustrates an exemplary set of operations for a long-range, low-frame-rate depth sensing mode. This operating mode can be used to sense depth in the range of approximately 2–4 meters (depending on the modulation frequency and exposure time) using a frame rate of less than approximately 20 Hz. In this particular embodiment, the frame rate is 5 Hz, which means that one complete frame of depth information is captured every 200 ms. As in the previous case, each complete frame of depth information is based on an intensity subframe and eight phase subframes. [Table 4]

[0042] An exemplary sequence of operations for the long-range low-frame-rate mode begins with step 0, which acquires an intensity subframe. Then, during steps 1-8, eight phase subframes are captured. Again, for long-range measurements, the exposure time for each of these subframes is typically less than approximately 2-3 ms, and each subframe also includes an associated readout time for transferring the captured image data from the image sensor. The readout time is typically less than approximately 1 ms. Steps 0-8 in Table 4 are identical to steps 0-8 in Table 3. The long-range low-frame-rate and long-range high-frame-rate operating modes therefore have these nine steps in common.

[0043] However, the long-range low-frame-rate operating mode includes a relatively long delay as step 9 of the operating sequence. This delay can be, for example, equal to the difference between the 200ms duration of the operating sequence and the total time required to complete steps 0-9. In other words, the long delay in step 9 can occupy any additional time during the duration of the operating sequence that is not required to capture and read the intensity subframes and the eight phase subframes. The difference between the two operating modes shown in Tables 3 and 4 is therefore related to the difference between the relatively long delay in step 9 in Table 4 and the optional relatively short delay in step 9 in Table 3.

[0044] To operate in a specific depth sensing mode (e.g., one of the depth sensing modes shown in Table 1-4), the depth sensor 100 needs to be programmed with an appropriate set of operation steps (and associated settings). Conventional depth sensors typically have multiple memory bins for holding programming instructions. Each bin can hold one of the operations shown in, for example, the sequence of operations in Table 1-4. Therefore, to program the TOF camera to operate in a short-range, high-frame-rate depth sensing mode (i.e., according to Table 1), five or six programming bins would typically be required. Similarly, a short-range, low-frame-rate mode (i.e., according to Table 2) would typically require six programming bins. On the other hand, a long-range, high-frame-rate operating mode (i.e., according to Table 3) would typically require nine or ten programming bins, while a long-range, low-frame-rate operating mode (i.e., according to Table 4) would typically require ten programming bins. Therefore, using the conventional method, 6+6+10+10=32 memory bins may be required to program the depth sensor 100 to operate in all four of these depth sensing modes.

[0045] Each depth sensor 100 can be programmed to operate in any of the depth sensing modes illustrated in Table 1-4 or others by loading its respective operating steps (and associated settings) into the sensor's memory bin. This programming process can take, for example, about 160 ms in some implementations, but may take longer or shorter periods depending on the specific implementation. Therefore, if only one set of operating steps (corresponding to one depth sensing mode) is programmed into the depth sensor's memory bin at a time, there may be a cost of about 160 ms required to reprogram the depth sensor to switch operating modes. If the depth sensor is not required to change modes very often, this time cost may be acceptable. However, in VR / AR / MR systems 80, there may be a need to switch between depth sensing modes relatively frequently. The time required to reprogram the depth sensor can therefore be problematic, as it may introduce a significant delay into the system's responsiveness. This problem and other problems are solved by the depth sensing techniques described herein.

[0046] Figure 3 illustrates an exemplary depth sensing system 300. The depth sensing system 300 includes a state machine 320, an arbiter 330, and the depth sensor 100 itself. The state machine 320 and the arbiter 330 can be implemented as hardware (e.g., one or more processors including a general-purpose processor, a field-programmable gate array (FPGAS), an application-specific integrated circuit (ASIC), etc.) and / or software (e.g., computer-readable instructions stored in memory, a non-transient medium, etc.). Figure 3 also shows several mixed reality (MR) applications 310 that communicate with the depth sensing system 300. These are applications that run on the VR / AR / MR system 80. One of these applications 310 may be, for example, a gesture recognition application. Another may be a 3-D mapping application. Another may be a virtual content projection application. Each of these applications 310 may have different types of depth information needs at different times. It is not uncommon for different types of depth information to be required by different applications 310 at the same or nearly the same moment. Therefore, it is advantageous for the depth sensing system 300 to be able to switch between depth sensing modes and acquire the required depth information as quickly and efficiently as possible. Note that although only mixed reality applications are illustrated in Figure 3, virtual reality and augmented reality applications can also communicate with the depth sensing system 300 and request and receive depth information.

[0047] Each application 310 can request depth sensing system 300 for various types of depth information as needed. The arbiter 330 is involved in receiving requests for depth information and scheduling depth sensing operations that will provide the requested depth information. In some embodiments, the arbiter 330 prioritizes requests for depth measurements to serve applications with more time constraints first. For example, in some embodiments, the arbiter 330 prioritizes depth sensing requests in the following order (however, other prioritization schemes may also be used): 1) short-range high-frame-rate depth measurements, 2) high-dynamic-range depth measurements (consisting of short-range low-frame-rate depth measurements interleaved with long-range low-frame-rate depth measurements), 3) short-range low-frame-rate depth measurements, 4) long-range high-frame-rate depth measurements, 5) long-range low-frame-rate depth measurements, and 6) idle state.

[0048] In some embodiments, the order in which depth sensing requests are prioritized is based on the priority of the requesting application. For example, since VR / AR / MR systems typically rely on user hand gestures to provide control input (such systems typically do not have touch panels, keyboards, or other physical input devices), any user's hand gesture may be assigned the highest priority. Therefore, in some embodiments, the highest priority mode may be short-range, high-frame-rate depth sensing used to track hand gestures. However, it should be understood that various depth sensing modes can be prioritized in various ways to adapt to different operational needs.

[0049] Once requests for depth information are prioritized and scheduled by the arbiter 330, the state machine 320 is used to control the depth sensor 100 hardware to actually perform the requested measurements and return the requested data. As part of this task, the state machine 320 may perform various tasks, including storing operation steps (and associated settings) in the memory bin of the depth sensor 100, setting a selected depth sensing mode, and switching the depth sensing mode of the depth sensor 100 when requested to do so based on input from the arbiter 330. The operation of the state machine 320 will be described in more detail with reference to Figures 4 and 5.

[0050] Figure 4 illustrates an example of an improved method 400 for efficiently operating the depth sensor 100 in multiple depth sensing modes. Method 400 begins in block 410 with commands for configuring the depth sensor 100. This type of command may be issued, for example, when starting or resetting the depth sensing system 300.

[0051] In block 420, the depth sensing system 300 begins configuring the depth sensor 100 by loading a set of operations for a first depth sensing mode into a first group of memory bins of the depth sensor. For example, the first depth sensing mode may be a short-range, high-frame-rate mode. If so, the depth sensing system 300 will load a set of operations from Table 1 into the memory bins of the depth sensor. In a conventional depth sensing system, the depth sensor 100 would then proceed to operate in the first depth sensing mode and acquire depth measurements until a different depth sensing mode is required. However, the depth sensing system 300 described herein instead proceeds to block 430 and loads a set of operations for the second to Nth depth sensing modes into a group of memory bins of the depth sensor 100. For example, the second depth sensing mode may be a long-range, high-frame-rate mode. If applicable, the depth sensing system 300 will load a series of operations from Table 3 into the memory bin of the depth sensor 100. Additional depth sensing modes can also be programmed during this configuration series, as long as there are available memory bins within the depth sensor 100. As will be discussed further below, these configuration steps are performed before depth sensing begins, thereby avoiding configuration delays when changing between depth sensing modes.

[0052] In block 440, method 400 proceeds to a command to initiate the collection of depth information. This command can be issued based on a depth sensing task scheduled by arbiter 330. In block 450, state machine 320 defines a programmed depth sensing operating mode to be used. If a first depth sensing operating mode is defined in block 450, method 400 proceeds to block 460. In block 460, depth sensor 100 operates in the first depth sensing mode by performing a set of operations defined within a first group of memory bins. While in the first depth sensing mode, depth sensor 100 proceeds to capture one or more frames of depth information. Once this measurement is complete, method returns to block 450, where a depth sensing operating mode may be defined again.

[0053] Returning to block 450, if the depth sensing operating mode has been changed, method 400 proceeds to block 470. In block 470, the depth sensor 100 can operate in any of the second to Nth depth sensing modes by performing a set of operations defined within the corresponding group of memory bins. After collecting depth information for one or more frames according to any of the second to Nth depth sensing operating modes, method 400 returns to block 450 and iteratively repeats according to the depth sensing task scheduled by the arbiter 330.

[0054] The operating method shown in Figure 4 is advantageous because it can improve depth sensing efficiency by reducing the amount of time spent programming the depth sensor 100 due to changes in the required depth sensing operating mode. This is achieved by simultaneously programming operating steps corresponding to multiple depth sensing modes into the memory bins of the depth sensor. For example, the first six memory bins of the depth sensor can be programmed using the operating steps in Table 2, while the next ten memory bins of the depth sensor can be programmed using the operating steps in Table 4. In this way, the depth sensor can operate in a short-range, low-frame-rate mode (corresponding to Table 2) by executing instructions stored in the first six bins, or in a long-range, low-frame-rate mode (corresponding to Table 4) by executing instructions stored in the next ten bins. The depth sensor can switch between these modes without incurring the time penalty required to reprogram the memory bins. Therefore, the efficiency and speed of depth information acquisition can be increased.

[0055] In some embodiments, the state machine 320 performs an operation that causes the depth sensor 100 to switch from one depth sensing mode to another without requiring reprogramming of the memory bins. The state machine 320 provides the ability to alternate between depth sensing modes by executing a series of operational steps from different sets / subsets of memory bins within the depth sensor 100. The state machine 320 can then externally control the sets / subsets and sets of memory bins from which the depth sensor 100 executes commands. Without the state machine 320, the depth sensor 100 can simply cycle through the commands stored in its memory bins without the ability to select a particular set / subset of commands to execute in order to achieve a desired depth sensing mode.

[0056] Figure 5 is an exemplary schematic diagram 500 of states for efficiently operating the depth sensor 100 in multiple depth sensing modes. The states shown in Figure 5 can be implemented by a state machine 320 cooperating with the arbiter 330. In some embodiments, the depth sensing system 300 has three states: 1) "hot standby" state 510, 2) "on" state 520, and 3) "held hot standby" state 530. After being programmed according to blocks 410-430 in Figure 4, the depth sensor 100 can be set to the hot standby state 510.

[0057] Based on the command to initiate the collection of depth information in block 440 of the method in Figure 4, the state machine 320 sets the depth sensor 100 to the ON state 520. This state change can be performed, for example, by first releasing the frame buffer and receiving the depth information. Next, the state machine 320 can configure the host VR / AR / MR system 80 to receive the streaming depth information. The state machine 320 can then configure the depth sensing mode according to block 450 of the method shown in Figure 4. As discussed herein, the state machine 320 can configure the depth sensing mode by defining a set / subset and / or series of operation steps to be performed, stored in the memory bins of the depth sensor 100. For example, a first depth sensing mode can be configured by configuring the depth sensor 100 to perform only a series of operations defined by bin XY (where X is any integer and Y is any integer greater than X). Finally, the state machine 320 can configure the depth sensor 100 to begin streaming depth information according to a defined depth sensing mode. The depth sensor 100 remains in the ON state 520 while continuing to stream frames of depth information according to the defined mode, until each of the conditions for switching to the pending hot standby state 530 is met.

[0058] In some embodiments, the state machine 320 switches the depth sensor from the ON state 520 to the pending hot standby state 530 when the following conditions are met: 1) a complete frame of depth information has been received, and 2) the arbiter 330 indicates that a mode switch is required. When these conditions are met, the state machine 320 places the depth sensor 100 into the pending hot standby state 530. In this state, the state machine 320 sets the depth sensor to stop streaming.

[0059] While in the pending hot standby state 530, the state machine 320 ensures that the frame period of the current depth sensing mode is maintained. This is done to limit the amount of energy output by the depth sensor 100 per specified time unit for eye safety reasons. For example, if a particular depth measurement is scheduled to be performed at a frame rate of 5 Hz, the frame period for that measurement is 200 ms. Typically, the power of the light source in the depth sensor 100 is set to a safety level based on its frame period. Therefore, in some embodiments, the state machine 320 does not allow the depth sensing mode to be changed until its 200 ms frame period has elapsed, because doing so could immediately start a new depth measurement, which in turn would cause the new depth measurement to emit additional radiation during the 200 ms period, potentially exceeding the eye safety limit.

[0060] While still in the pending hot standby state 530, the state machine 320 sets the host VR / AR / MR system 80 into standby mode and closes the frame buffer used to receive depth information. Once these actions are complete, the depth sensing system state machine 320 transitions the depth sensor 100 to the hot standby state 510. Working with the arbiter 330, the state machine 320 then defines the next depth sensing mode, and the process can be repeated. Again, the next depth sensing mode is set by defining a set / subset and / or a series of operational steps stored in the memory bin of the depth sensor 100 for execution. Also, changing the depth sensing mode in this way does not require the depth sensor 100 to be reprogrammed, since the operational steps for the next depth sensing mode are already stored in the memory bin.

[0061] Method 400, shown in Figure 4, can improve the efficiency of depth sensing operations, but this may be limited by the number of available memory bins provided by the depth sensor 100. Depending on the desired number and type of depth sensing modes, there may be insufficient memory bins to accommodate the required sequence of operation commands. For example, if the depth sensor 100 provides only 15 memory bins, it would be impossible to simultaneously program the depth sensor using the sequence of operations required by all four depth sensing modes described in Table 1-4, as well as any additional depth sensing modes, according to conventional techniques. This is because those depth sensing modes collectively involve more than 15 operation steps. Therefore, even if Method 400, shown in Figure 4, is implemented, depending on the number of available memory bins, the depth sensor 100 may still need to be periodically reprogrammed to produce all of the depth sensing modes described in Table 1-4. As already discussed, this can result in a time penalty, which is undesirable. This problem can be mitigated by providing the depth sensor with additional memory bins. However, doing so would increase the size and cost of the depth sensor. However, there is another technique, illustrated in Figure 6-8, which can be used to further improve the efficiency of the depth sensor 100, even when the number of memory bins may be insufficient to accommodate the sequence of operations for all the desired depth sensing modes. This technique takes advantage of the fact that different depth sensing modes may have some operating steps in common, such that the common steps do not necessarily need to be programmed into the depth sensor 100 more than once.

[0062] Figure 6 illustrates another example of an improved method 600 for efficiently operating the depth sensor 100 in multiple depth sensing modes. Method 600 can be performed using the same depth sensing system 300 shown in Figure 3 and the same operating state shown in the schematic state diagram 500 in Figure 5. Method 600 begins in block 610 with commands for configuring the depth sensor 100. Again, this type of command may be issued, for example, when starting or resetting the depth sensing system 300. In some embodiments, the improved method 600 shown in Figure 6 allows the configuration of the depth sensor 100 to be performed only once during each session of operation. For example, in some embodiments, after being initially programmed, the depth sensor 100 may not need to be programmed again until the host puts the depth sensor into reset mode or until the depth sensor is restarted.

[0063] In block 620, the depth sensing system 300 begins configuring the depth sensor 100 by loading common operating steps across two or more depth sensing modes into the depth sensor's memory bin. These common operating steps are identical across two or more operating modes. For example, the steps (and associated settings) for capturing intensity subframes and four phase subframes are the same for both high-frame-rate short-range depth measurement and low-frame-rate short-range depth measurement. Referring to Tables 1 and 2, these common operating steps correspond to steps 0-4. Similarly, the steps (and associated settings) for capturing intensity subframes and eight phase subframes are the same for both high-frame-rate long-range depth measurement and low-frame-rate long-range depth measurement. Referring to Tables 3 and 4, these common operating steps correspond to steps 0-8.

[0064] In block 630, the depth sensing system 300 continues the configuration of the depth sensor 100 by loading one or more dummy operation steps into the memory bin. In some embodiments, the dummy operation steps relate to the difference between two or more operating modes. By executing one or more dummy operation steps together with a set of common operation steps for the two operating modes, one of the operating modes can be effectively converted to the other.

[0065] For example, as already discussed herein, the difference between the high frame rate short-range depth sensing mode (i.e., Table 1) and the low frame rate short-range depth sensing mode (i.e., Table 2) concerns the difference between each frame period, in other words, the difference between the amount of delay introduced before repeating the subframe capture sequence. In the case of high frame rate short-range depth sensing, a relatively short delay (or no delay) is used. In the case of low frame rate short-range depth sensing, a relatively long delay is introduced to reduce the frame rate (correspondingly increasing the frame period). Therefore, with respect to this pair of depth sensing modes (i.e., high frame rate short-range depth sensing and low frame rate short-range depth sensing), the dummy operation step can be defined as a delay representing the difference between the relatively long delay in the low frame rate measurement and the relatively short optional delay in the high frame rate measurement. In other words, the dummy operation step for this pair of depth sensing modes can be a delay equal to the difference between the relatively long delay in step 5 of Table 2 and the relatively short optional delay in step 5 of Table 1. Similarly, the dummy operation step for high-frame-rate long-range measurement and low-frame-rate long-range measurement can be a delay equal to the difference between the relatively long delay in step 5 of Table 4 and the relatively short voluntary delay in step 5 of Table 3.

[0066] Figure 7 is an illustrative table 700 showing common and dummy operation steps for multiple depth sensing modes. In this illustrative table 700, step 0 is a dummy operation step for a pair of long-range depth measurements. This dummy operation step is a delay that, when added to a series of operations performed to execute a high-frame-rate long-range measurement mode, converts that series of operations to a low-frame-rate long-range measurement mode. This dummy operation step can be stored in a first memory bin of the depth sensor 100.

[0067] On the other hand, steps 1 through m in Table 700 are common operation steps between the high-frame-rate long-range measurement mode and the low-frame-rate long-range measurement mode. For the exemplary TOF camera discussed herein, long-range depth measurement requires nine total subframes (one intensity subframe and eight phase subframes). Therefore, index m in Table 700 will be equal to 9. Hence, steps 1-9 will be used to capture the intensity subframe and eight phase subframes for the long-range operation mode. These operation steps can be stored in the next nine memory bins of the depth sensor 100 after the dummy operation in step 0. The next step is the eye safety dummy operation step, which is provided in step m+1 in Table 700. This dummy operation step is discussed with respect to Figure 9.

[0068] Table 700 in Figure 7 also shows the common operating steps between the high-frame-rate short-range measurement mode and the low-frame-rate short-range measurement mode. These common operations are represented in Table 700 by steps m+2 through m+n+1. For the exemplary TOF camera discussed herein, the short-range measurement requires five total subframes (one intensity subframe and four phase subframes). Thus, index n in Table 700 will be equal to 5 (as discussed just above, index m will be equal to 9). Therefore, steps 11-15 will be used to capture the intensity subframe and four phase subframes for the short-range operating mode. These operating steps can be stored in the following five memory bins of the depth sensor 100.

[0069] On the other hand, step m+n+2 in Table 700 is a dummy operation step for paired short-range depth measurement. This dummy operation step is a delay that, when added to a series of operations performed to execute a high-frame-rate short-range measurement mode, converts that series of operations to a low-frame-rate short-range measurement mode. This dummy operation step can be stored in the following memory bin of the depth sensor 100.

[0070] As will be further discussed with respect to Figure 8, various combinations of the operation steps in Table 700 can be performed in the illustrated order to achieve various depth-sensing operation modes.

[0071] After the depth sensor 100 is programmed using common operation steps and dummy operation steps according to blocks 610-630, the method 600 shown in Figure 6 continues in block 640 with a command to start collecting depth information. In block 650, the depth sensing system 300 defines a depth sensing operation mode. This can be done, for example, by defining the operation steps shown in Table 700 in Figure 7 to perform the defined depth sensing operation mode. This is discussed in relation to Figure 8.

[0072] Figure 8 is an illustrative Table 800 illustrating how the common operation steps and dummy operation steps in Figure 7 can be used to operate efficiently in multiple depth sensing modes. As shown in Figure 8, the high frame rate long-range depth sensing mode (as shown in Table 3) can be performed by executing steps 1 through m in Table 700 shown in Figure 7. By performing these steps, the depth sensor 100 will collect long-range intensity subframes and eight long-range phase subframes during steps 1 through m. On the other hand, if the depth sensing system 300 instead calls for low frame rate long-range depth measurement (as shown in Table 4), this operation mode can instead be performed by executing steps 0 through m. Since the dummy frame in step 0 represents the difference between high frame rate long-range measurement and low frame rate long-range measurement, executing that step in addition to steps 1 through m effectively converts the operation mode from high frame rate long-range measurement mode to low frame rate long-range measurement mode. The dummy operation in step 0 is shown in Table 700 to be performed before the acquisition of subframes during steps 1 through m, but in other embodiments it may be performed after the acquisition of subframes, or even between the acquisition of two subframes.

[0073] Figure 8 also shows that the high frame rate short-range depth sensing mode (as shown in Table 1) can be performed by executing steps m+2 through m+n+1 in Table 700 shown in Figure 7. By executing these steps, the depth sensor 100 will collect short-range intensity subframes and four short-range phase subframes. On the other hand, if the system instead calls for low frame rate short-range depth sensing (as shown in Table 2), this mode of operation can instead be performed by executing steps m+2 through m+n+2. Since the dummy frame in step m+n+2 represents the difference between high frame rate short-range and low frame rate short-range sensing, executing that step in addition to steps m+2 through m+n+1 effectively converts the mode of operation from high frame rate short-range mode to low frame rate short-range mode.

[0074] Table 800 in Figure 8 also shows a high dynamic range depth sensing mode, which consists of interleaved low frame rate short and long-range measurements. This depth sensing mode is discussed in relation to Figure 9.

[0075] After the depth sensing system 300 defines a depth sensing operating mode in block 650, the method 600 shown in Figure 6 proceeds to block 660 or block 670. The depth sensor 100 can operate in a first depth sensing mode by performing a group of common operating steps, as shown in block 660. The group of common operating steps could be, for example, steps 1 through m in Table 700 shown in Figure 7. This would correspond to operation in a high-frame-rate, long-range depth sensing mode. Alternatively, the group of common operating steps performed in block 660 could be steps m+2 through m+n+1 in Table 700 shown in Figure 7. This would correspond to operation in a high-frame-rate, short-range depth sensing mode.

[0076] Alternatively, the depth sensor 100 can operate in a second depth sensing mode by performing a group of common operation steps and one or more dummy operation steps, as shown in block 670. The group of common operation steps could be, for example, steps 1 through m in Table 700 shown in Figure 7, and the dummy operation step could be step 0. This would correspond to operation in a low-frame-rate, long-range depth sensing mode. Or, the group of common operation steps performed in block 660 could be steps m+2 through m+n+1 in Table 700 shown in Figure 7, and the dummy operation step could be step m+n+2. This would correspond to operation in a low-frame-rate, short-range depth sensing mode.

[0077] Regardless of whether the depth sensing system 300 moves from block 650 to block 660 or block 670, the depth sensor 100 captures one or more frames of depth information while in a defined depth sensing mode. Once the measurement is complete, method 600 returns to block 650, and the depth sensing operating mode can be defined again.

[0078] The operating method 600 shown in Figure 6 is advantageous because it can improve depth sensing efficiency by reducing the amount of time spent programming the depth sensor 100 in response to a change in the required depth sensing operating mode. According to method 600, the depth sensor can switch between multiple depth sensing modes without incurring the time penalty required to reprogram the memory bins. This can be accomplished using far fewer memory bins within the depth sensor 100 than would be required using conventional techniques. Thus, the efficiency and speed of depth information acquisition can be increased. In addition, lower-cost depth sensors with fewer memory bins can be used.

[0079] Figure 9 is an exemplary timing diagram for operation in High Dynamic Range (HDR) depth sensing mode. The HDR depth sensing mode consists of interleaved long-range and short-range measurements. Table 5 illustrates an exemplary set of operations for the HDR depth sensing mode. In some embodiments, the frame rate for HDR depth measurement is 5 Hz. The period for HDR depth measurement is T in Figure 9. fps This is labeled as such. This HDR depth sensing mode can be performed using the method shown in Figure 6 and the depth sensor memory bin programming scheme shown in Figure 7.

[0080] The HDR depth sensing sequence begins with step 0, in which a long-range intensity subframe is captured. Then, during steps 1-4, the depth sensor 100 captures four phase subframes using a first modulation frequency. As shown in Figure 9, each phase subframe is T LR-int The exposure time, i.e., the integral time, is set to the exposure time. After each of these exposures, there is a readout time T for transferring the captured image data from the sensor. readout This continues. Next, in step 5-8, the depth sensor 100 captures four phase subframes using a second modulation frequency. [Table 5]

[0081] After long-distance measurement, in step 9, ocular safety delays continued, which are shown in Figure 9 as T eye_safe_dummy This is labeled as such. This delay can prevent the eye safety circuit in the depth sensor 100 from being triggered so that the light source of the depth sensor is not interrupted in the middle of the measurement. This delay is another example of a dummy operation step, which will be discussed further below. As shown in Figure 9, the dummy operation step that constitutes the delay may be implemented in some embodiments as an intensity subframe (including an exposure period and a reading period), followed by an idle period. The intensity subframe captured during the dummy operation step is typically not used to calculate the depth.

[0082] Next, in step 10 in Table 5, the depth sensor captures a short-range intensity subframe. This is followed by four phase subframes in steps 11-14. As shown in Figure 9, each of these subframes is T SR_int After the exposure time, the reading period follows. After the short-distance measurement performed in steps 10-14, an optional delay in step 15 may follow.

[0083] The HDR depth sensing modes shown in Table 5 and Figure 9 have many operating steps, which are common to other depth sensing modes discussed herein. For example, steps 0-8 in Table 5 are identical to those used by high-frame-rate long-range depth sensing. Therefore, this part of the HDR depth sensing mode can be implemented by performing steps 1 through m in the programming scheme 700 shown in Figure 7. Similarly, steps 10-14 in Table 5 are identical to those used by high-frame-rate short-range depth sensing. Therefore, they can be implemented by performing steps m+2 through m+n+1 in the programming scheme 700 shown in Figure 7.

[0084] The differences between the HDR depth sensing mode and the high frame rate long-range mode compared to the high frame rate short-range mode are the eye safety period in step 9 and the optional delay in step 15 of Table 5. These differences can be implemented by appropriate dummy motion frames, as discussed herein. For example, the eye safety period can be implemented by an eye safety dummy motion step shown in step m+1 of the programming scheme 700 shown in Figure 7. The optional delay in step 15 of Table 5 can be implemented using a dummy motion in step m+n+2 of the programming scheme 700 shown in Figure 7.

[0085] As shown in Figure 8, the full HDR depth sensing mode (interleaved long-range and short-range depth measurement) can be implemented by performing steps 1 through m+n+2 of the programming scheme 700 shown in Figure 7. The HDR depth sensing modes in Table 5 are therefore examples of how additional modes can be implemented from the common and dummy operation steps shown in Figure 7.

[0086] The aforementioned disclosure describes various efficient depth sensing techniques for use in VR / AR / MR systems. While these techniques are specifically discussed in relation to depth sensors, the same techniques can also be applied to other types of sensors and are not strictly limited to depth sensors. (Exemplary embodiment)

[0087] In some embodiments, the method includes providing the sensor with a set of common operation steps that are included in both a first set of operation steps defining a first operating mode and a second set of operation steps defining a second operating mode; providing the sensor with one or more dummy operation steps relating to the difference between the first operating mode and the second operating mode; causing the sensor to operate in the first operating mode by having the sensor perform at least the common operation steps; and causing the sensor to operate in the second operating mode by having the sensor perform the common operation steps and at least one dummy operation step.

[0088] According to any of these embodiments, a first operating mode may include performing at least a common operating step at a first rate, and a second operating mode may include performing the common operating step and at least one dummy operating step at a second rate that is slower than the first rate.

[0089] According to any of these embodiments, at least one of the one or more dummy operation steps may include a delay.

[0090] According to any of these embodiments, providing the sensor with a series of common operating steps and one or more dummy operating steps may include storing those operating steps in sensor memory.

[0091] According to any of these embodiments, switching the sensor between a first operating mode and a second operating mode may not require any additional actions to store the operating steps in the sensor memory.

[0092] According to any of these embodiments, the sensor may include a depth sensor.

[0093] According to any of these embodiments, the depth sensor may be equipped with a time-of-flight camera.

[0094] According to any of these embodiments, the first operating mode may include a depth sensing mode with a first frame rate, and the second operating mode may include a depth sensing mode with a second frame rate that is slower than the first frame rate.

[0095] According to any of these embodiments, the method may further include providing depth information from a depth sensor to a virtual reality, augmented reality, or mixed reality display system.

[0096] In some embodiments, the system includes a processor configured to perform a method comprising: providing the sensor with a set of common operation steps that include both a first set of operation steps defining a first operating mode and a second set of operation steps defining a second operating mode; providing the sensor with one or more dummy operation steps relating to the difference between the first operating mode and the second operating mode; and causing the sensor to operate in the first operating mode by having the sensor perform at least the common operation steps; and causing the sensor to operate in the second operating mode by having the sensor perform the common operation steps and at least one dummy operation step.

[0097] According to any of these embodiments, a first operating mode may include performing at least a common operating step at a first rate, and a second operating mode may include performing the common operating step and at least one dummy operating step at a second rate that is slower than the first rate.

[0098] According to any of these embodiments, at least one of the one or more dummy operation steps may include a delay.

[0099] According to any of these embodiments, providing the sensor with a series of common operating steps and one or more dummy operating steps may include storing those operating steps in sensor memory.

[0100] According to any of these embodiments, switching the sensor between a first operating mode and a second operating mode may not require any additional actions to store the operating steps in the sensor memory.

[0101] According to any of these embodiments, the sensor may include a depth sensor.

[0102] According to any of these embodiments, the depth sensor may be equipped with a time-of-flight camera.

[0103] According to any of these embodiments, the first operating mode may include a depth sensing mode with a first frame rate, and the second operating mode may include a depth sensing mode with a second frame rate that is slower than the first frame rate.

[0104] According to any of these embodiments, the system can be integrated into a virtual reality, augmented reality, or mixed reality display system.

[0105] According to any of these embodiments, the processor may include a state machine.

[0106] According to any of these embodiments, the system may further include an arbiter for receiving requests to operate the sensor in a first or second mode, the arbiter may be configured to schedule and prioritize the requests.

[0107] According to any of these embodiments, the system may further include sensors.

[0108] In some embodiments, the method includes receiving a first request for a first type of depth measurement, receiving a second request for a second type of depth measurement, assigning a first priority to the first request, assigning a second priority to the second request, and, if the first priority is higher than the second priority, causing the depth sensor to first acquire a first type of depth measurement, or, if the second priority is higher than the first priority, causing the depth sensor to first acquire a second type of depth measurement.

[0109] According to any of these embodiments, a first priority can be assigned based on the priority of a first application requesting a first type of depth measurement, and a second priority can be assigned based on the priority of a second application requesting a second type of depth measurement.

[0110] In some embodiments, the system comprises an arbiter configured to receive a first request for a first type of depth measurement and a second request for a second type of depth measurement, the arbiter configured to assign a first priority to the first request and a second priority to the second request, and a processor configured to cause a depth sensor to first acquire a first type of depth measurement if the first priority is higher than the second priority, or to cause a depth sensor to first acquire a second type of depth measurement if the second priority is higher than the first priority.

[0111] According to any of these embodiments, the arbiter can be configured to assign a first priority based on the priority of a first application requesting a first type of depth measurement, and to assign a second priority based on the priority of a second application requesting a second type of depth measurement.

[0112] According to any of these embodiments, the system can be integrated into a virtual, augmented, or mixed reality display system.

[0113] In some embodiments, the method is to perform a configuration operation for a depth sensor, the configuration operation comprising storing in the depth sensor's memory a first set of operation steps defining a first depth sensing operation mode, and storing in the depth sensor's memory a second set of operation steps defining a second depth sensing operation mode, the method comprising receiving a first request for depth measurement according to the first depth sensing operation mode, operating the depth sensor in the first operation mode by causing the depth sensor to perform the first set of operation steps in response to the first request, the method comprising receiving a second request for depth measurement according to the second depth sensing operation mode, and operating the depth sensor in the second operation mode by causing the depth sensor to perform the second set of operation steps in response to the second request without performing any additional configuration operations.

[0114] According to any of these embodiments, the depth sensor may be equipped with a time-of-flight camera.

[0115] According to any of these embodiments, the first depth sensing operation mode can correspond to a first measurement distance range, and the second depth sensing operation mode can correspond to a second measurement distance range different from the first measurement distance range.

[0116] According to any of these embodiments, the first depth-sensing operation mode can correspond to a first frame rate, and the second depth-sensing operation mode can correspond to a second frame rate that is slower than the first frame rate.

[0117] According to any of these embodiments, the method may further include providing depth information from a depth sensor to a virtual reality, augmented reality, or mixed reality display system.

[0118] In some embodiments, the system includes a processor configured to perform a configuration operation for a depth sensor, the configuration operation comprising storing a first set of operation steps defining a first depth sensing operation mode in the depth sensor's memory, and storing a second set of operation steps defining a second depth sensing operation mode in the depth sensor's memory; receiving a first request for depth measurement according to the first depth sensing operation mode, and in response to the first request, causing the depth sensor to perform the first set of operation steps, thereby operating the depth sensor in the first operation mode; receiving a second request for depth measurement according to the second depth sensing operation mode, and in response to the second request, causing the depth sensor to perform the second set of operation steps without performing any additional configuration operations, thereby operating the depth sensor in the second operation mode.

[0119] According to any of these embodiments, the depth sensor may be equipped with a time-of-flight camera.

[0120] According to any of these embodiments, the first depth sensing operation mode can correspond to a first measurement distance range, and the second depth sensing operation mode can correspond to a second measurement distance range different from the first measurement distance range.

[0121] According to any of these embodiments, the first depth-sensing operation mode can correspond to a first frame rate, and the second depth-sensing operation mode can correspond to a second frame rate that is slower than the first frame rate.

[0122] According to any of these embodiments, the system can be integrated into a virtual, augmented, or mixed reality display system.

[0123] In some embodiments, a non-transient computer-readable medium, when read by a computing device, includes code that causes a computing device to perform a method including: providing a sensor with a set of common operation steps that include both a first set of operation steps defining a first operating mode and a second set of operation steps defining a second operating mode; providing the sensor with one or more dummy operation steps relating to the difference between the first operating mode and the second operating mode; and causing the sensor to operate in the first operating mode by causing the sensor to perform at least the common operation steps, and causing the sensor to operate in the second operating mode by causing the sensor to perform the common operation steps and at least one dummy operation step.

[0124] According to any of these embodiments, a first operating mode may include performing at least a common operating step at a first rate, and a second operating mode may include performing the common operating step and at least one dummy operating step at a second rate that is slower than the first rate.

[0125] According to any of these embodiments, at least one of the one or more dummy operation steps may include a delay.

[0126] According to any of these embodiments, providing the sensor with a series of common operating steps and one or more dummy operating steps may include storing those operating steps in sensor memory.

[0127] According to any of these embodiments, switching the sensor between a first operating mode and a second operating mode may not require any additional actions to store the operating steps in the sensor memory.

[0128] According to any of these embodiments, the sensor may include a depth sensor.

[0129] According to any of these embodiments, the depth sensor may be equipped with a time-of-flight camera.

[0130] According to any of these embodiments, the first operating mode may include a depth sensing mode with a first frame rate, and the second operating mode may include a depth sensing mode with a second frame rate that is slower than the first frame rate.

[0131] According to any of these embodiments, the computer-readable medium may further include code that causes a computing device to provide depth information from a depth sensor to a virtual reality, augmented reality, or mixed reality display system.

[0132] In some embodiments, a non-transient computer-readable medium, when read by a computing device, includes code causing a computing device to perform a method including receiving a first request for a first type of depth measurement, receiving a second request for a second type of depth measurement, assigning a first priority to the first request, assigning a second priority to the second request, and, if the first priority is higher than the second priority, causing the depth sensor to first acquire a first type of depth measurement, or if the second priority is higher than the first priority, causing the depth sensor to first acquire a second type of depth measurement.

[0133] According to any of these embodiments, a first priority can be assigned based on the priority of a first application requesting a first type of depth measurement, and a second priority can be assigned based on the priority of a second application requesting a second type of depth measurement.

[0134] In some embodiments, a non-transient computer-readable medium comprises a code, which, when read by a computing device, causes the computing device to perform a method for operating a depth sensor, the method being to perform a configuration operation for the depth sensor, the configuration operation comprising storing in the depth sensor's memory a first set of operation steps defining a first depth-sensing operation mode; storing in the depth sensor's memory a second set of operation steps defining a second depth-sensing operation mode; receiving a first request for depth measurement according to the first depth-sensing operation mode; operating the depth sensor in the first operation mode by causing the depth sensor to perform the first set of operation steps in response to the first request; receiving a second request for depth measurement according to a second depth-sensing operation mode; and operating the depth sensor in the second operation mode by causing the depth sensor to perform the second set of operation steps in response to the second request without performing any additional configuration operations.

[0135] According to any of these embodiments, the depth sensor may be equipped with a time-of-flight camera.

[0136] According to any of these embodiments, the first depth sensing operation mode can correspond to a first measurement distance range, and the second depth sensing operation mode can correspond to a second measurement distance range different from the first measurement distance range.

[0137] According to any of these embodiments, the first depth-sensing operation mode can correspond to a first frame rate, and the second depth-sensing operation mode can correspond to a second frame rate that is slower than the first frame rate.

[0138] According to any of these embodiments, the computer-readable medium may further include code that causes a computing device to provide depth information from a depth sensor to a virtual reality, augmented reality, or mixed reality display system. (Additional considerations)

[0139] For the purpose of summarizing this disclosure, certain aspects, advantages, and features of the present invention are described herein. It should be understood that not all such advantages can necessarily be achieved according to any particular embodiment of the present invention. Accordingly, the present invention may be embodied or practiced in a manner that achieves or optimizes one or more advantages or sets of advantages taught herein, without necessarily achieving other advantages that can be taught or suggested herein.

[0140] Embodiments are described in reference to the accompanying drawings. However, it should be understood that the drawings are not drawn to a specific scale. Distances, angles, etc., are illustrative only and do not necessarily have an exact relationship to the actual dimensions and layout of the illustrated devices. In addition, the embodiments described herein are described at a level of detail that will enable those skilled in the art to fabricate and use the devices, systems, methods, etc. described herein. A wide variety of modifications are possible. Components, elements, and / or steps may be modified, added, removed, or rearranged.

[0141] The devices and methods described herein can, for the benefit of our knowledge, be implemented at least partially using, for example, computer software, hardware, firmware, or any combination of software, hardware, and firmware. A software module may comprise computer executable code stored in the memory of a computer to perform the functions described herein. In some embodiments, the computer executable code is executed by one or more general-purpose computers. However, those skilled in the art will understand, in light of this disclosure, that any module that can be implemented using software running on a general-purpose computer can also be implemented using different combinations of hardware, software, or firmware. For example, such a module can be implemented entirely in hardware using a combination of integrated circuits. Alternatively, or in addition, such a module can be implemented entirely or partially using a specialized computer designed to perform the specific functions described herein, rather than by a general-purpose computer. Furthermore, where a method is described that is implemented, or can be implemented, at least partially by computer software, it should be understood that such a method may be provided on a non-transient computer-readable medium (e.g., optical discs such as CDs or DVDs, hard disk drives, flash memory, diskettes, etc.) that enables the method to be implemented when read by a computer or other processing device.

[0142] While some embodiments are explicitly described, other embodiments will also be obvious to those skilled in the art based on this disclosure.

Claims

1. A method for operating a depth sensor, the method being carried out by the processor of the depth sensor, The depth sensor receives one or more first requests for first depth measurement in accordance with a first depth sensing operation mode of the depth sensor, wherein the first depth sensing operation mode is defined by a first set of operation steps stored in the memory on the depth sensor, and the first set of operation steps includes one or more common operation steps. In response to the first request, the first depth sensing operation mode is implemented by performing the first series of operation steps, thereby generating one or more first depth measurements. The depth sensor receives one or more second requests for second depth measurement in accordance with a second depth sensing operation mode of the depth sensor, wherein the second depth sensing operation mode is defined by a combination of one or more common operation steps and a second series of operation steps stored in the memory on the depth sensor. In response to the second request, the second depth sensing operation mode is performed by executing the second series of operation steps and the one or more common operation steps without storing additional operation steps in the memory, thereby generating one or more second depth measurements. The processor includes a state machine, which causes the depth sensor to switch between the first depth sensing operation mode and the second depth sensing operation mode without reprogramming the memory. A method wherein the second set of operation steps includes at least one dummy operation step defined as a delay.

2. The method according to claim 1, wherein the depth sensor is a time-of-flight camera.

3. The method according to claim 1, wherein the first depth sensing operation mode corresponds to a first measurement distance range, and the second depth sensing operation mode corresponds to a second measurement distance range different from the first measurement distance range.

4. The method according to claim 1, wherein the one or more first depth measurements and the one or more second depth measurements are usable in a virtual, augmented, or mixed reality display system.

5. The method according to claim 1, wherein the first depth sensing operation mode corresponds to a first frame rate, and the second depth sensing operation mode corresponds to a second frame rate different from the first frame rate.

6. The method according to claim 1, wherein the first depth sensing operation mode corresponds to a first frame rate, and the second depth sensing operation mode corresponds to a second frame rate that is slower than the first frame rate.

7. Receiving at least one configuration request, In response to at least one of the configuration requests, the first set of operation steps and the second set of operation steps are stored in the memory. The method according to claim 1, further comprising:

8. The method according to claim 1, wherein the first depth sensing operation mode or the second depth sensing operation mode is a short-range high-frame-rate operation mode.

9. The method according to claim 8, wherein the short-range high-frame-rate operating mode is used to sense depth in a range of less than about 2 meters using a frame rate exceeding about 20 Hz.

10. The method according to claim 1, wherein the first depth sensing operation mode or the second depth sensing operation mode is a short-range, low-frame-rate operation mode.

11. The method according to claim 10, wherein the short-range, low-frame-rate operating mode is used to sense depth in a range of less than about 2 meters using a frame rate of less than about 20 Hz.

12. The method according to claim 1, wherein the first depth sensing operation mode or the second depth sensing operation mode is a long-range high-frame-rate operation mode.

13. The method according to claim 12, wherein the long-range high-frame-rate operating mode is used to sense depth in a range of about 2 to 4 m using a frame rate exceeding about 20 Hz.

14. The method according to claim 1, wherein the first depth sensing operation mode or the second depth sensing operation mode is a long-range low frame rate operation mode.

15. The method according to claim 14, wherein the long-range low-frame-rate operating mode is used to sense depth in a range of about 2 to 4 m using a frame rate of less than about 20 Hz.

16. The first depth sensing operation mode is a short-range, high-frame-rate operation mode. The second depth sensing operation mode is a short-range, low-frame-rate operation mode. The one or more common operation steps include i) capturing a short-range intensity subframe using a short exposure and ii) capturing four phase subframes, The first series of operation steps includes a first delay, The method according to claim 1, wherein the second set of operation steps includes a second delay that is longer than the first delay.

17. The first depth sensing operation mode is a long-range, high-frame-rate operation mode. The second depth sensing operation mode is a long-range, low-frame-rate operation mode. The one or more common operation steps include i) capturing long-range intensity subframes using long exposure, ii) capturing four phase subframes for a first illumination modulation frequency, and iii) capturing four phase subframes for a second illumination modulation frequency. The first series of operation steps includes a first delay, The method according to claim 1, wherein the second set of operation steps includes a second delay that is longer than the first delay.

Images