Depth sensing techniques for virtual, augmented, and mixed reality systems

Abstract

The present invention relates to depth sensing techniques for virtual, augmented and mixed reality systems. A system and method for operating a sensor having at least two modes of operation. The sensor can be provided with a sequence of common operating steps that are included in both a sequence of first operating steps that define a first mode of operation and a sequence of second operating steps that define a second mode of operation. The sensor can also be provided with one or more virtual operating steps that relate to differences between the first mode of operation and the second mode of operation. The sensor is operated in the first mode of operation by causing the sensor to perform at least the common operating steps, and can be operated in the second mode of operation by causing the sensor to perform the common operating steps and at least one virtual operating step.

Description

[0001] This application is a divisional application of the application filed on March 19, 2018, with Chinese national phase application number 202110857359.0 and entitled "Depth Sensing Technology for Virtual, Augmented and Mixed Reality Systems".

[0002] By incorporating any priority claim

[0003] Any and all applications that identify foreign or domestic priority claims in the application data sheet filed in this application are hereby incorporated by reference under 37 CFR 1.57. Specifically, this application claims priority to U.S. Provisional Application No. 62 / 474,503, filed March 21, 2017, entitled “Deep Sensing Technologies for Virtual, Augmented, and Mixed Reality Systems,” which is incorporated herein by reference in its entirety. Technical Field

[0004] This disclosure relates to depth sensors, such as depth sensors that can be used in virtual reality, augmented reality, and mixed reality imaging and visualization systems. Background Technology

[0005] Modern computing and display technologies have facilitated the development of virtual reality, augmented reality, and mixed reality systems. Virtual reality, or "VR," systems create simulated environments for users. This is accomplished by presenting computer-generated images to the user through a head-mounted display. These images create a sensory experience that immerses the user in the simulated environment. Virtual reality scenes typically involve only the presentation of computer-generated images and do not necessarily include actual, real-world images.

[0006] Augmented reality (AR) systems typically supplement the real-world environment with simulated elements. For example, an AR system can provide a user with a view of their surrounding real-world environment via a head-mounted display. However, computer-generated images can also be displayed to enhance the real-world environment. These computer-generated images can include elements that are context-dependent on the real-world environment. Such elements can include simulated text, images, objects, etc. Mixed reality (MR) systems are a type of AR system that also introduces simulated objects into the real-world environment, but these objects are typically characterized by a greater degree of interactivity. Simulated elements can often be interacted with in real time.

[0007] Figure 1An example AR / MR scene 1 is depicted, in which the user sees a real-world park setting 6 characterized by people, trees, buildings in the background, and a concrete platform 20. In addition to these items, computer-generated images are presented to the user. The computer-generated images may include, for example, a robot statue 10 standing on the real-world platform 20, and a cartoonish avatar 2 that appears to be an incarnation of a flying bee, even though these elements 2, 10 do not actually exist in the real environment. Summary of the Invention

[0008] In some embodiments, a method for operating a sensor having at least two operating modes includes: providing the sensor with a common sequence of operating steps, the common sequence of operating steps being included in both a first sequence of operating steps defining a first operating mode and a second sequence of operating steps defining a second operating mode; providing the sensor with one or more dummy operating steps relating to the difference between the first and second operating modes; operating the sensor in the first operating mode by causing the sensor to perform at least the common operating steps; and operating the sensor in the second operating mode by causing the sensor to perform the common operating steps and at least one dummy operating step.

[0009] In some embodiments, the sensor may be a depth sensor. A first operating mode may include a depth sensing mode with a first frame rate, and a second operating mode may be a depth sensing mode with a second frame rate that is slower than the first frame rate. For example, one or more virtual operation steps may include a delay.

[0010] In some embodiments, a system for operating a sensor having at least two operating modes includes: a processor configured to perform a method comprising: providing a common sequence of operating steps to the sensor, the common sequence of operating steps being included in both a first sequence of operating steps defining a first operating mode and a second sequence of operating steps defining a second operating mode; providing the sensor with one or more virtual operating steps relating to a difference between the first and second operating modes; operating the sensor in the first operating mode by causing the sensor to perform at least the common sequence of operating steps; and operating the sensor in the second operating mode by causing the sensor to perform the common sequence of operating steps and at least one virtual sequence of operating steps. Attached Figure Description

[0011] Figure 1 The example AR system shows a user's view of an augmented reality (AR) scene.

[0012] Figure 2 An example of a wearable VR / AR / MR display system is shown.

[0013] Figure 3 An example depth sensing system is shown.

[0014] Figure 4 An example of an improved method for effectively operating a depth sensor in multiple depth sensing modes is shown.

[0015] Figure 5 This is an example state diagram for effectively operating a depth sensor in multiple depth sensing modes.

[0016] Figure 6 Another example of an improved method for effectively operating a depth sensor in multiple depth sensing modes is shown.

[0017] Figure 7 This is an example table showing common and virtual operating steps for multiple depth sensing modes.

[0018] Figure 8 This demonstrates how it can be used effectively in multiple depth sensing modes. Figure 7 Example table of common operation steps and virtual operation steps.

[0019] Figure 9 This is an example timing diagram for operation in high dynamic range (HDR) depth sensing mode. Detailed Implementation

[0020] Virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems may include displays that present computer-generated images to a user. In some embodiments, the display system is wearable, which can advantageously provide a more immersive VR / AR / MR experience. The computer-generated images presented via the display can create a three-dimensional impression. This can be accomplished, for example, by presenting stereoscopic images to the user.

[0021] Figure 2An example of a wearable VR / AR / MR display system 80 is shown. The VR / AR / MR display system 80 includes a display 62, and various mechanical and electronic modules and systems supporting the functionality of the display 62. The display 62 can be coupled to a frame 64, which can be worn by a user 60 and positions the display 62 in front of the user 60's eyes. A speaker 66 can be coupled to the frame 64 and positioned near the user's ear canal. Another speaker (not shown) can be positioned near the user's other ear canal to provide stereo / shapeable sound control. The display 62 is operatively coupled to a local data processing module 70, such as via a wired or wireless connection 68, which can be mounted in various configurations, such as being fixedly attached to the frame 64, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise detachably attached to the user 60 (e.g., in a backpack configuration, a belt-coupled configuration, etc.).

[0022] The local processing and data module 70 may include a processor and digital memory, such as non-volatile memory (e.g., flash memory), both of which can be used to assist in processing and storing data. This includes data captured from sensors such as image capture devices (e.g., cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes. Sensors may be operatively coupled to frame 64 or otherwise attached to user 60. In some embodiments, all data is stored and all calculations are performed in the local processing and data module, allowing for fully autonomous use. Alternatively or additionally, remote processing module 72 and / or remote data storage 74 may be used to acquire and / or process sensor data. The local processing and data module 70 may be operatively coupled to remote processing module 72 and remote data storage 74 via communication links (76, 78) (e.g., via wired or wireless communication links), such that these remote modules (72, 74) are operatively coupled to each other and available as resources for the local processing and data module 70. In some embodiments, the remote processing module 72 may include one or more processors configured to analyze and process data (e.g., sensor data and / or image information). The remote data storage facility 74 may include a digital data storage facility accessible via the Internet or other network configurations in a “cloud” resource configuration.

[0023] The VR / AR / MR system 80 may also include a depth sensor 100. The depth sensor 100 measures the user's surrounding environment to determine information about distances to various objects and features present in that environment. VR / AR / MR applications can utilize various types of depth information, including short-range depth information (e.g., 0-2 meters), long-range depth information (e.g., 2-4 meters and above), and high dynamic range (HDR) depth information. The depth information provided by the depth sensor 100 can be used to allow the user to interact with the VR / AR / MR system and / or to allow the system to project virtual images onto the user's real-world environment.

[0024] One application of long-range depth sensing in VR / AR / MR systems is using depth information to model a user's environment. For example, depth sensor 100 can be used to determine distances to walls and objects within a room. The resulting depth information can be used to create a 3D model of the room and its contents. Particularly in AR / MR systems, this allows the system to project virtual images into the room in a realistic and interactive manner. An example application of short-range depth sensing in VR / AR / MR systems is gesture recognition. For instance, VR / AR / MR system 80 can use depth sensing to track the movement of a user's hand for gesture recognition. VR / AR / MR system 80 can then perform certain actions in response to the user's gestures.

[0025] Assuming the VR / AR / MR system 80 can use depth information to provide an interactive, immersive experience to the user, it is advantageous for the depth sensor 100 to collect depth information relatively quickly and efficiently, as this allows the VR / AR / MR system 80 to be more responsive. This is especially true for AR / MR applications, which may be highly sensitive to discontinuities between real-world content surrounding the user and virtual content projected by the system 80 into the user's environment. Therefore, this disclosure describes improved techniques that can increase the efficiency and / or speed at which various depth sensing information can be collected.

[0026] As background technology, one type of depth sensor is the 3D Time-of-Flight (TOF) camera. Generally, a 3D TOF camera uses a light source to illuminate a scene. The TOF camera then observes and processes the light reflected from the scene to determine information about the distances to various points / objects / features within the scene. Some TOF cameras perform depth measurements by emitting infrared light pulses towards one or more points in the scene and then measuring the elapsed time until the light reflects back from the scene. Based on the elapsed time, combined with knowledge of the speed of light, the camera can then determine the distance the light has traveled. Alternatively, some TOF cameras can perform depth measurements by emitting modulated light signals (e.g., square waves or sine waves) and then measuring the phase shift between the illuminating and reflected light signals. These phase shift measurements are then converted into distance measurements.

[0027] In most depth-sensing Time-of-Flight (TOF) cameras, illumination comes from solid-state lasers or light-emitting diodes (LEDs) operating in the near-infrared range (e.g., ~850 nm), which is invisible to the human eye. Typically, the illumination from the light source to the scene is designed to be relatively uniform. An imaging sensor, designed to respond to the same spectrum as the illumination light, receives reflected light from the scene and converts the light into an electrical signal. In some embodiments, the imaging sensor may be a CCD or CMOS sensor with a resolution of, for example, 224 × 172 pixels, but imaging sensors with higher or lower resolutions may also be used. Each pixel is located at a point in the image plane, corresponding to a single point in the object space or scene within the TOF camera's field of view. Therefore, the information collected at each pixel of the imaging sensor can be used to determine the distance to the point within the scene corresponding to that particular pixel.

[0028] The light received by each pixel of the imaging sensor has an ambient component and a reflection component. Depth information is embedded only in the reflection component. To distinguish between these two components, a TOF camera can capture an image of the ambient infrared light before or after actively illuminating the scene with infrared light. This image of the ambient infrared light can be called an intensity subframe image. By subtracting or otherwise removing the intensity subframe image from other subframe images collected during active scene illumination, the depth sensor 100 can distinguish the reflection component of the infrared light from background noise in the scene.

[0029] To allow detection of the phase shift between the illumination and reflection components, a signal from the light source can be modulated. For example, a square wave modulation signal can be used. The image sensor then detects reflected light at multiple different times corresponding to different phase shifts relative to the modulation signal. These different phase shifts can be, for example, angles 1, 2, 3, and 4, where angle 2 = angle 1 + Δ, angle 3 = angle 1 + 2Δ, and angle 4 = 1 + 3Δ, and where angles 1 and Δ are predetermined angles. For example, angle 1 can be 0° and Δ can be 90°, allowing the camera to detect reflected light received at each pixel during time periods of phase shifts of 0°, 90°, 180°, and 270° relative to the modulation signal. Each of these measurements results in a separate phase subframe image captured by the camera sensor. The distance to a point in the scene corresponding to each of the sensor pixels can then be calculated based on the four phase subframes using mathematical equations known in the art. Thus, each complete depth information frame of a set of depth measurements (one per pixel) can be determined to consist of several subframes of image data.

[0030] The modulated illumination signal is periodic and therefore repeats every 360° phase shift. Therefore, the fact that some Time-of-Flight (TOF) cameras measure depth based on the phase shift of the reflected light relative to the modulated illumination signal means that the measured distance will be subject to aliasing. These aliasing effects can cause blurring of the measured distance. The distance at which aliasing occurs (i.e., the blurred distance) is also the maximum blur-free distance that a TOF camera can measure. The maximum measurable distance can be extended by reducing the modulation frequency of the illumination light, but this may come at the cost of reduced depth measurement resolution. To resolve depth blur without compromising depth measurement resolution, a TOF camera can modulate the illumination light using two or more separate modulation signals (e.g., Fmod0 and Fmod1) with different frequencies. Depth measurement is performed by measuring the phase shift of the reflected light relative to each of the multiple modulation frequencies. Since each modulation frequency is different, each modulation frequency will have a different blurred distance. The actual distance to a given point in the scene is the consistent distance measured using the different modulation frequencies.

[0031] In a Time-of-Flight (TOF) camera, the distance to each pixel in the camera sensor can be measured. This results in a depth map of the scene within the camera's field of view. A depth map is a collection of points or voxels in 3D space, where each voxel is located at a distance measured by its corresponding sensor pixel. The depth map can be rendered as a set of points or a point cloud in 3D space. These 3D points can be mathematically connected to form a mesh. Meshes can be used for scene modeling, object detection, and more. Furthermore, virtual content can be mapped onto the mesh through VR / AR / MR systems to provide realistic 3D virtual content that interacts with the user's real-life environment.

[0032] In the VR / AR / MR system 80, various types of depth measurement can be advantageous for different purposes. For example, near-range low frame rate depth measurement may be sufficient to detect when a user's hand appears in the field of view of the depth sensor 100. Once the fact that the user's hand is present in the field of view of the depth sensor is detected, near-range high frame rate depth measurement may be more useful for tracking the movement of the user's hand and thus detecting a specific gesture in progress. Meanwhile, long-range depth measurement at low or high frame rates is very useful for mapping the user's environment. Furthermore, high dynamic range (HDR) depth measurement, ranging from near to long range, is also beneficial.

[0033] Considering that many different types of depth measurements may be useful in the VR / AR / MR system 80, the depth sensor 100 may include multiple operating modes to collect each of these different types of depth measurements. Each mode may include, for example, a series of operations performed by the depth sensor 100. Depending on the mode, each of these operations may involve different settings or parameters, such as exposure time, illumination intensity, illumination modulation frequency, etc. The table below illustrates some example operating sequences and configuration settings for depth sensing modes.

[0034] Table 1 shows an example operating sequence for a short-range, high-frame-rate depth sensing mode. In some embodiments, this operating mode is used to sense depth over a range of less than approximately 20 meters (depending on the modulation frequency and exposure time) at a frame rate greater than approximately 20 Hz. In this particular embodiment, the frame rate is 45 Hz, meaning that a complete depth information frame is captured every 22.22 ms (1 / 45 second). In this case, each complete depth information frame is based on an intensity subframe (used to measure ambient infrared light when the illumination source is off) and four phase subframes (captured when the illumination source is modulated).

[0035]

[0036]

[0037] Table 1

[0038] An example operational sequence for a short-range, high-frame-rate depth sensing mode begins with step 0, which acquires an intensity subframe. Then, during steps 1-4, four phase subframes are captured. For short-range measurements, the exposure time (i.e., the time during which the image sensor captures light) for each of these subframes is typically less than approximately 0.5 ms. Each subframe also includes an associated readout time for transmitting the captured image data from the image sensor. The readout time is typically less than approximately 1 ms.

[0039] The short-range, high-frame-rate operation mode may optionally include a relatively short delay, such as in step 5 of the operation sequence. This delay may be equal to, for example, the difference between the 22.22 ms time interval of the operation sequence and the total time required to complete steps 0-4. In other words, the optional short delay in step 5 can occupy any additional time during the time interval of the operation sequence, which is not necessary for capturing and reading out the intensity subframes and the four phase subframes. Although Table 1 lists the specific order of the operation steps for this particular depth sensing mode, the operation steps may alternatively be performed in a different order. The same applies to the other operation modes described herein.

[0040] Table 2 shows an example operating sequence for the short-range, low-frame-rate depth sensing mode. This operating mode can be used to sense depth over a range of less than approximately 2 meters (depending on the modulation frequency and exposure time) at a frame rate of less than approximately 20 Hz. In this particular embodiment, the frame rate is 8 Hz, meaning that a complete depth information frame is captured every 125 ms. As in the previous case, each complete depth information frame is based on an intensity subframe and four phase subframes. While the short-range, high-frame-rate mode has the advantage of producing depth measurements with better temporal resolution, the short-range, low-frame-rate mode can be beneficial—allowing the system to enter a low-power mode and conserve energy due to lower computational intensity—when the lower temporal resolution is sufficient to accomplish the task at hand.

[0041]

[0042]

[0043] Table 2

[0044] The example operating sequence for the short-range low frame rate mode begins with step 0, which acquires an intensity subframe. Then, during steps 1-4, four phase subframes are captured. Again, for short-range measurements, the exposure time of each of these subframes is typically less than approximately 0.5 ms, and the readout time of each subframe is typically less than approximately 1 ms. Steps 0-4 in Table 2 are identical to steps 0-4 in Table 1. Therefore, both the short-range low frame rate operating mode and the short-range high frame rate operating mode share these five common steps.

[0045] However, the short-range, low-frame-rate operation mode also includes a relatively long delay, such as step 5 of the operation sequence. This delay can be equal to, for example, the difference between the 125ms time interval of the operation sequence and the total time required to complete steps 0-4. The relatively long delay of step 5 occupies time during the time interval of the operation sequence, which is unnecessary for capturing and reading out the intensity subframes and the four phase subframes. Therefore, the difference between the two short-range operation modes shown in Tables 1 and 2 respectively involves the difference between the relatively long delay of step 5 in Table 2 and the optional relatively short delay of step 5 in Table 1.

[0046] Table 3 shows an example operating sequence for a long-range, high-frame-rate depth sensing mode. For example, this operating mode can be used to sense depth over a range of approximately 2–4 meters (depending on the modulation frequency and exposure time) at a frame rate greater than approximately 20 Hz. Similar to short-range depth data, each complete long-range depth information frame is based on several subframes of image data. Again, there are intensity subframes for measuring ambient infrared light when the illumination source is off. However, in the case of long-range depth data, there are eight phase subframes of image data: four phase subframes for each of the two illumination modulation frequencies, Fmod1 and Fmod2.

[0047]

[0048]

[0049] Table 3

[0050] An example operating sequence for the long-range, high-frame-rate depth sensing mode begins with step 0, which acquires an intensity subframe. Then, during steps 1-4, four phase subframes are captured for a first modulation frequency Fmod1, and during steps 5-8, four subframes are captured for a second modulation frequency Fmod2. For long-range measurements, the exposure time (i.e., the time during which the image sensor captures light) for each of these subframes is longer than for short-range measurements, typically 2-3 ms. (Other parameters or settings for the long-range subframes may also differ from those for the short-range subframes). Each subframe also includes an associated readout time of approximately 1 ms for transmitting the captured image data from the image sensor.

[0051] The long-range high frame rate operation mode may optionally include a relatively short delay, such as step 9 of the operation sequence. This delay may be equal to, for example, the difference between the duration of the operation sequence and the total time required to complete steps 0-8. In other words, the optional short delay of step 9 may occupy any additional time during the duration of the operation sequence that is not required for capturing and reading out the intensity subframes and the eight phase subframes.

[0052] Table 4 shows an example operating sequence for a long-range, low-frame-rate depth sensing mode. This operating mode can be used to sense depth over a range of approximately 2–4 meters (depending on the modulation frequency and exposure time) at a frame rate less than approximately 20 Hz. In this particular embodiment, the frame rate is 5 Hz, meaning that a complete depth information frame is captured every 200 ms. As in the previous cases, each complete depth information frame is based on an intensity subframe and eight phase subframes.

[0053]

[0054]

[0055] Table 4

[0056] The example operating sequence for the long-range low frame rate mode begins with step 0, which acquires an intensity subframe. Then, during steps 1-8, eight phase subframes are captured. Again, for long-range measurements, the exposure time for each of these subframes is typically less than approximately 2-3 ms, and each subframe also includes an associated readout time for transmitting the captured image data from the image sensor. The readout time is typically less than approximately 1 ms. Steps 0-8 in Table 4 are identical to steps 0-8 in Table 3. Therefore, both the long-range low frame rate operating mode and the long-range high frame rate operating mode share these nine common steps.

[0057] However, the long-range low frame rate operation mode also includes a relatively long delay, such as step 9 of the operation sequence. This delay can be equal to, for example, the difference between the 200ms period of the operation sequence and the total time required to complete steps 0-9. In other words, the long delay of step 9 can occupy any additional time during the period of the operation sequence, which is not necessary for capturing and reading out the intensity subframes and the eight phase subframes. Therefore, the difference between the two operation modes shown in Tables 3 and 4 involves the difference between the relatively long delay of step 9 in Table 4 and the optional relatively short delay of step 9 in Table 3.

[0058] To operate in a specific depth sensing mode (e.g., any of the depth sensing modes shown in Tables 1-4), the depth sensor 100 needs to be programmed with an appropriate sequence of operating steps (and associated settings). Conventional depth sensors typically have multiple memory boxes for holding programming instructions. Each box can hold one of the operations shown in the operating sequences of Tables 1-4, for example. Therefore, to program a TOF camera to operate in a short-range high frame rate depth sensing mode (i.e., according to Table 1), five or six programming boxes would typically be required. Similarly, a short-range low frame rate mode (i.e., according to Table 2) typically requires six programming boxes. Meanwhile, a long-range high frame rate operating mode (i.e., according to Table 3) typically requires nine or ten programming boxes, while a long-range low frame rate operating mode (i.e., according to Table 4) typically requires ten programming boxes. Therefore, using conventional methods, 6 + 6 + 10 + 10 = 32 memory boxes might be needed to program the depth sensor 100 to operate in all four depth sensing modes.

[0059] The depth sensor 100 can be programmed to operate in any of the depth sensing modes shown in Tables 1-4, as well as other depth sensing modes, by loading the corresponding operation steps (and associated settings) into the sensor's memory. In some implementations, this programming process may take, for example, approximately 160 ms, but depending on the specific implementation, it may take longer or shorter periods. Therefore, if only one set of operation steps (corresponding to one depth sensing mode) is programmed into the depth sensor's memory at a time, it may cost approximately 160 ms to reprogram the depth sensor to switch operating modes. This time cost is acceptable if the depth sensor does not need to change modes frequently. However, in the VR / AR / MR system 80, switching between depth sensing modes may be required relatively frequently. Therefore, the time required to reprogram the depth sensor can become a problem, as it may introduce significant lag in terms of system responsiveness. This and other problems are addressed by the depth sensing techniques described herein.

[0060] Figure 3 An example depth sensing system 300 is shown. The depth sensing system 300 includes a state machine 320, an arbitrator 330, and a depth sensor 100 itself. The state machine 320 and the arbitrator 330 can be implemented as hardware (e.g., one or more processors, including general-purpose processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc.) and / or software (e.g., computer-readable instructions stored in memory, non-transient media, etc.). Figure 3 Several mixed reality (MR) applications 310 communicating with the depth sensing system 300 are also shown. These are applications operating on the VR / AR / MR system 80. One of these applications 310 could be, for example, a pose recognition application. Another could be a 3D mapping application. Yet another could be a virtual content projection application. Each of these applications 310 may require various different types of depth information at different times. It is not uncommon for different applications 310 to require different types of depth information at the same or near-the same time. Therefore, it is advantageous that the depth sensing system 300 can switch between depth sensing modes to obtain the requested depth information as quickly and efficiently as possible. It should be noted that, although Figure 3 Only mixed reality applications are shown in the image, but virtual reality and augmented reality applications can also communicate with the depth sensing system 300 to request and receive depth information.

[0061] Each application 310 may request various types of depth information from the depth sensing system 300 as needed. An arbitrator 330 is responsible for receiving requests for depth information and for scheduling depth sensing operations that will provide the requested depth information. In some embodiments, the arbitrator 330 also prioritizes depth measurement requests to serve more time-critical applications first. For example, in some embodiments, the arbitrator 330 prioritizes depth sensing requests in the following order (although other prioritization schemes may also be used): 1) short-range high frame rate depth measurements; 2) high dynamic range depth measurements (consisting of short-range low frame rate depth measurements interleaved with long-range low frame rate depth measurements); 3) short-range low frame rate depth measurements; 4) long-range high frame rate depth measurements; 5) long-range low frame rate depth measurements; and 6) idle state.

[0062] In some embodiments, the order in which depth measurement requests are prioritized is based on the priority of the requesting application. For example, since VR / AR / MR systems typically rely on user gestures to provide control input (because such systems generally do not have touchpads, keyboards, or other physical input devices), the highest priority can be assigned to any user gesture. Therefore, in some embodiments, the highest priority mode could be short-range, high-frame-rate depth measurement for tracking gestures. However, it should be understood that priorities can be assigned to various depth sensing modes in various ways to suit different operational needs.

[0063] Once the request for depth information is prioritized and scheduled by the arbitrator 330, the state machine 320 controls the hardware of the depth sensor 100 to actually perform the required measurements and return the requested data. As part of this task, the state machine 320 can perform various tasks, including storing operation steps (and associated settings) in the memory of the depth sensor 100; setting the selected depth sensing mode; and switching the depth sensing mode of the depth sensor 100 when required based on input from the arbitrator 330. About Figure 4 and Figure 5 The operation of state machine 320 is described in more detail.

[0064] Figure 4 An example of an improved method 400 for effectively operating a depth sensor 100 in multiple depth sensing modes is shown. Method 400 begins at block 410 with a command to configure the depth sensor 100. This type of command can be issued, for example, during the startup or reset of the depth sensing system 300.

[0065] At block 420, the depth sensing system 300 begins configuring the depth sensor 100 by loading an operation sequence for a first depth sensing mode into a first set of memory bins of the depth sensor. For example, the first depth sensing mode could be a short-range high frame rate mode. If this is the case, the depth sensing system 300 will load the operation step sequence from Table 1 into the memory bins of the depth sensor. In conventional depth sensing systems, the depth sensor 100 would then continue operating in the first depth sensing mode to obtain depth measurements until a different depth sensing mode is required. However, the depth sensing system 300 described herein continues to block 430, where it loads operation sequences for second to Nth depth sensing modes into multiple sets of memory bins of the depth sensor 100. For example, the second depth sensing mode could be a long-range high frame rate mode. If this is the case, the depth sensing system 300 will load the operation step sequence from Table 3 into the memory bins of the depth sensor 100. Additional depth sensing modes can also be programmed during this configuration sequence, provided that an available memory bin exists in the depth sensor 100. As discussed further below, these configuration steps can be performed before depth sensing begins, thus avoiding configuration delays when changing between depth sensing modes.

[0066] At block 440, method 400 proceeds to the command to begin collecting depth information. This command can be issued based on the depth sensing task scheduled by arbitrator 330. At block 450, state machine 320 specifies which programmed depth sensing operation mode to use. If a first depth sensing operation mode is specified in block 450, method 400 proceeds to block 460. At block 460, depth sensor 100 operates in the first depth sensing mode by executing the sequence of operations specified in the first set of memory boxes. In the first depth sensing mode, depth sensor 100 continues to capture one or more frames of depth information. Once the measurement is complete, the method returns to block 450, where the depth sensing operation mode can be specified again.

[0067] Once back to box 450, if the depth sensing operation mode changes, method 400 proceeds to box 470. At box 470, the depth sensor 100 can operate in any of the second to Nth depth sensing modes by executing the sequence of operations specified in the memory bin of the corresponding group. After collecting one or more frames of depth information according to any of the second to Nth depth sensing operation modes, method 400 returns to box 450 and iteratively repeats according to the depth sensing task scheduled by arbitrator 330.

[0068] Figure 4The operating method shown is advantageous because it improves depth sensing efficiency by reducing the amount of time dedicated to programming the depth sensor 100 due to changes in the requested depth sensing operating mode. This is achieved by simultaneously programming operating steps corresponding to multiple depth sensing modes into the depth sensor's memory bins. For example, the first six memory bins of the depth sensor can be programmed using the operating steps in Table 2, while the next ten memory bins can be programmed using the operating steps in Table 4. Thus, the depth sensor can operate in a short-range, low frame rate mode (corresponding to Table 2) by executing the instructions stored in the first six bins. Alternatively, the depth sensor can operate in a long-range, low frame rate mode (corresponding to Table 4) by executing the instructions stored in the next ten bins. The depth sensor can alternate between these modes without incurring the time loss required to reprogram the memory bins. Therefore, the efficiency and speed of depth information collection can be improved.

[0069] In some embodiments, state machine 320 performs operations that switch depth sensor 100 from one depth sensing mode to another without requiring reprogramming the memory bins. State machine 320 provides the ability to alternate between depth sensing modes by executing a sequence of operation steps from memory bins of different groups / subgroups within depth sensor 100. State machine 320 can be externally controlled by multiple groups / subgroups of memory bins and the sequence of memory bins, and depth sensor 100 executes instructions according to the sequence of multiple groups / subgroups of memory bins and the sequence of memory bins. Without state machine 320, depth sensor 100 can simply cycle through commands stored in its memory bins without needing to select a specific group / subgroup of commands to execute in order to achieve the desired depth sensing mode.

[0070] Figure 5 This is an example state diagram 500 for effectively operating the depth sensor 100 in multiple depth sensing modes. Figure 5 The states shown can be achieved by the state machine 320 working in conjunction with the arbitrator 330. In some embodiments, the depth sensing system 300 has three states: 1) "hot standby" state 510; 2) "on" state 520; and 3) "pending hot standby" state 530. According to... Figure 4 After programming boxes 410-430 in the middle, the depth sensor 100 can be placed in a hot standby state 510.

[0071] Based on Figure 4At block 440 of the method, the command to collect depth information is initiated, and state machine 320 puts depth sensor 100 into an enabled state 520. This state change can be accomplished, for example, by first opening the frame buffer to receive depth information. Next, state machine 320 can configure the host VR / AR / MR system 80 to accept streaming depth information. Then, state machine 320 can... Figure 4 Block 450 of the method shown sets the depth sensing mode. As discussed herein, state machine 320 can set the depth sensing mode by specifying groups / subgroups and / or sequences of operation steps stored in the memory bin of depth sensor 100 for execution. For example, a first depth sensing mode can be specified by setting depth sensor 100 to execute only the sequence of operation specified by bin XY, where X is any integer and Y is any integer greater than X. Finally, state machine 320 can set depth sensor 100 to begin streaming depth information according to the specified depth sensing mode. Depth sensor 100 continues to stream depth information frames according to the specified mode while in the on state 520 until each condition for switching to the pending hot standby state 530 exists.

[0072] In some embodiments, state machine 320 switches the depth sensor from the ON state 520 to the Pending Hot Standby state 530 when the following conditions are met: 1) a complete depth information frame has been received; 2) arbitrator 330 indicates that a mode switch is required. Once these conditions are met, state machine 320 places the depth sensor 100 in the Pending Hot Standby state 530. In this state, state machine 320 sets the depth sensor to stop streaming.

[0073] When in a pending hot standby state 530, state machine 320 ensures that the current depth sensing mode is maintained for a specified frame period. This is done for eye safety reasons to limit the amount of energy output by depth sensor 100 per specified unit of time. For example, if a specific depth measurement is scheduled at a frame rate of 5 Hz, the frame period for that measurement is 200 ms. Typically, the power of the light source in depth sensor 100 is set at a safe level based on this frame period. Therefore, in some embodiments, state machine 320 does not allow the depth sensing mode to be changed until the 200 ms frame period has elapsed, because doing so would immediately initiate a new depth measurement, which could in turn result in additional radiation emitted during the 200 ms period, potentially exceeding eye safety limits.

[0074] While still in the pending hot standby state 530, state machine 320 sets the host VR / AR / MR system 80 to standby and disables the frame buffer used to receive depth information. Once these actions are complete, the depth sensing system state machine 320 transitions the depth sensor 100 to the hot standby state 510. State machine 320 works with arbitrator 330 to specify the next depth sensing mode, and this process can be repeated. Again, the next depth sensing mode is set by specifying a group / subgroup and / or sequence of operation steps to be performed stored in the memory of depth sensor 100. Changing the depth sensing mode in this way does not require reprogramming the depth sensor 100, because the operation steps of the next depth sensing mode are already stored in the memory.

[0075] although Figure 4 The method 400 shown can improve the efficiency of depth sensing operations, which can be limited by the number of available memory bins provided by the depth sensor 100. Depending on the required number and type of depth sensing modes, there may not be enough memory bins to hold the required sequence of operation instructions. For example, if the depth sensor 100 only provides 15 memory bins, it is impossible, according to conventional techniques, to program the depth sensor simultaneously using the operation sequences required for all four depth sensing modes described in Tables 1-4, let alone additional depth sensing modes. This is because those depth sensing modes collectively involve more than 15 operation steps. Therefore, even if implemented... Figure 4 As illustrated in method 400, depending on the number of available memory cartridges, the depth sensor 100 may still need to be periodically reprogrammed to provide all the depth sensing modes described in Tables 1-4. As already discussed, this can lead to undesirable time loss. This problem can be mitigated by providing additional memory cartridges to the depth sensor. However, doing so increases the size and cost of the depth sensor. But there is another technique, such as... Figure 6-8 As shown, even when the number of memory boxes may be insufficient to accommodate the operation sequences of all required depth sensing modes, it can still be used to further improve the efficiency of the depth sensor 100. This technique takes advantage of the fact that different depth sensing modes can have several common operation steps, so that these common steps do not have to be programmed into the depth sensor 100 all at once.

[0076] Figure 6 Another example of an improved method 600 for efficiently operating a depth sensor 100 in multiple depth sensing modes is shown. Method 600 can use... Figure 3 The same depth sensing system 300 shown and Figure 5The same operating state shown in state diagram 500 is executed. Method 600 begins at block 610 with a command to configure depth sensor 100. Again, this type of command can be issued, for example, during the startup or reset of depth sensing system 300. In some embodiments, Figure 6 The improved method 600 shown allows the configuration of the depth sensor 100 to be performed only once during each operating session. For example, in some embodiments, the depth sensor 100 may not need to be reprogrammed after initial programming until the host puts the depth sensor into reset mode or until the depth sensor is powered back on.

[0077] At box 620, the depth sensing system 300 begins configuring the depth sensor 100 by loading common operating steps between two or more depth sensing modes into the depth sensor's memory. These common operating steps are those that are identical across the two or more operating modes. For example, for both high frame rate short-range depth measurement and low frame rate short-range depth measurement, the steps (and associated settings) for capturing intensity subframes and four phase subframes are the same. Referring to Tables 1 and 2, these common operating steps correspond to steps 0-4. Similarly, for both high frame rate long-range depth measurement and low frame rate long-range depth measurement, the steps (and associated settings) for capturing intensity subframes and eight phase subframes are the same. Referring to Tables 3 and 4, these common operating steps correspond to steps 0-8.

[0078] At block 630, the depth sensing system 300 continues the configuration of the depth sensor 100 by loading one or more virtual operation steps into a memory bin. In some embodiments, a virtual operation step is a step involving a difference between two or more operating modes. By executing one or more virtual operation steps together with a common sequence of operation steps for two operating modes, one operating mode can be effectively converted to another.

[0079] For example, as already discussed herein, the difference between the high frame rate short-range depth sensing mode (i.e., Table 1) and the low frame rate short-range depth sensing mode (i.e., Table 2) is related to the difference between the corresponding frame periods (or in other words, the difference between the amounts of delay introduced before the repeated subframe capture sequence). In the case of high frame rate short-range depth measurement, a relatively short delay (or no delay at all) is used. In the case of low frame rate short-range depth measurement, a relatively long delay is introduced to reduce the frame rate (and correspondingly increase the frame period). Therefore, for this pair of depth measurements (i.e., high frame rate short-range depth measurement and low frame rate short-range depth measurement), the virtual operation step can be defined as a delay representing the difference between the relatively long delay of the low frame rate measurement and the relatively short optional delay of the high frame rate measurement. In other words, the virtual operation step of this pair of depth sensing modes can be a delay equal to the difference between the relatively long delay in step 5 of Table 2 and the relatively short optional delay in step 5 of Table 1. Similarly, the virtual operation steps for both high frame rate long-range measurements and low frame rate long-range measurements can be the difference between the relatively long delay in step 5 of Table 4 and the relatively short optional delay in step 5 of Table 3.

[0080] Figure 7 Example table 700 illustrates common and virtual operation steps for multiple depth sensing modes. In this example table 700, step 0 is a virtual operation step for the pair of long-range depth measurements. This virtual operation step is a delay that, when added to the operation sequence performed in a high frame rate long-range measurement mode, converts the operation sequence to a low frame rate long-range measurement mode. This virtual operation step can be stored in a first memory box of the depth sensor 100.

[0081] Meanwhile, steps 1 to m in Table 700 are common operating steps between the high frame rate long-range measurement mode and the low frame rate long-range measurement mode. For the example TOF camera discussed here, long-range depth measurement requires nine total subframes (one intensity subframe and eight phase subframes). Therefore, index m in Table 700 will equal 9. Thus, steps 1-9 will be used to capture the intensity subframe and eight phase subframes for the long-range operating mode. After the virtual operating steps at step 0, these operating steps can be stored in the next nine memory boxes of the depth sensor 100. The next step is the eye-safe virtual operating steps, which are provided at step m+1 in Table 700. (Regarding...) Figure 9 Discuss the virtual operation steps.

[0082] Figure 7Table 700 also shows the common operating steps between the high frame rate short-range measurement mode and the low frame rate short-range measurement mode. These common operations are represented in Table 700 by steps m+2 to m+n+1. For the example TOF camera discussed here, short-range measurement requires five total subframes (one intensity subframe and four phase subframes). Therefore, index n in Table 700 will be equal to five (and as just discussed, index m will be equal to 9). Thus, steps 11-15 will be used to capture the intensity subframe and four phase subframes for the short-range operating mode. These operating steps can be stored in the next five memory bins of the depth sensor 100.

[0083] Meanwhile, step m+n+2 in Table 700 is a virtual operation step for this pair of short-range depth measurements. This virtual operation step is a delay that, when added to the operation sequence executed in the high frame rate short-range measurement mode, converts the operation sequence to a low frame rate short-range measurement mode. This virtual operation step can be stored in the next memory compartment of the depth sensor 100.

[0084] Such as about Figure 8 As discussed further, various combinations of the operation steps in Table 700 can be performed in the order shown to achieve various depth sensing operation modes.

[0085] After the depth sensor 100 has been programmed with common and virtual operating steps according to blocks 610-630 Figure 6 The method 600 shown continues at box 640 with a command to begin collecting depth information. At box 650, the depth sensing system 300 specifies the depth sensing operation mode. This can be done, for example, by specifying the execution... Figure 7 Which of the operational steps shown in Table 700 is required to perform the specified depth sensing operation mode? This will refer to... Figure 8 Let's have a discussion.

[0086] Figure 8 It shows Figure 7 Table 800 provides examples of how common and virtual operating steps can be used to effectively operate in multiple depth sensing modes. Figure 8 As shown, it can be achieved by executing Figure 7Steps 1 through m in Table 700 are used to perform a high frame rate long-range depth sensing mode (as shown in Table 3). By performing these steps, the depth sensor 100 will collect long-range intensity subframes and eight long-range phase subframes during steps 1 through m. Conversely, if the depth sensing system 300 instead requires low frame rate long-range depth measurements (as shown in Table 4), this operating mode can be accomplished by performing steps 0 through m instead. Since the virtual frame in step 0 represents the difference between high frame rate and low frame rate long-range measurements, performing steps other than 1 through m effectively switches the operating mode from high frame rate long-range measurement mode to low frame rate long-range measurement mode. Although the virtual operation at step 0 is shown in Table 700 as being performed before subframe collection during steps 1 through m, in other embodiments, it can be performed after subframe collection, or even between the collection of two subframes.

[0087] Figure 8 The high frame rate short-range measurement depth sensing mode (as shown in Table 1) is also illustrated by performing... Figure 7 This is accomplished through steps m+2 to m+n+1 in Table 700 shown. By performing these steps, the depth sensor 100 will collect short-range intensity subframes and four short-range phase subframes. Conversely, if the system instead requires low frame rate short-range depth measurements (as shown in Table 2), this operating mode can be performed by instead executing steps m+2 to m+n+2. Since the virtual frame in step m+n+2 represents the difference between high frame rate and low frame rate short-range measurements, performing steps other than m+2 to m+n+1 effectively switches the operating mode from high frame rate short-range measurement mode to low frame rate short-range measurement mode.

[0088] Figure 8 Table 800 also shows high dynamic range depth sensing modes, which include interleaved low frame rate short-range and long-range measurements. Regarding... Figure 9 Discuss this depth sensing mode.

[0089] After the depth sensing system 300 specifies the depth sensing operation mode at frame 650. Figure 6 The method 600 shown continues to box 660 or box 670. As shown in box 660, the depth sensor 100 can operate in a first depth sensing mode by performing a set of common operating steps. This set of common operating steps may be, for example... Figure 7 Steps 1 through m in Table 700 are shown. This corresponds to the operation in high frame rate long-range depth sensing mode. Alternatively, a set of common operation steps performed in box 660 could be... Figure 7Steps m+2 to m+n+1 in Table 700 shown below correspond to the operation in high frame rate short-range depth sensing mode.

[0090] Alternatively, as shown in box 670, the depth sensor 100 can operate in a second depth sensing mode by performing a set of common operating steps and one or more virtual operating steps. Figure 7 In Table 700 shown, the set of common operation steps can be, for example, steps 1 to m, and the virtual operation step can be step 0. This corresponds to operation in a low frame rate long-range depth sensing mode. Alternatively, in Figure 7 In Table 700 shown, the set of common operation steps performed in box 660 can be steps m+2 to m+n+1, and the virtual operation step can be step m+n+2. This corresponds to the operation in a low frame rate short-range depth sensing mode.

[0091] Regardless of whether the depth sensing system 300 moves from box 650 to box 660 or to box 670, the depth sensor 100 captures one or more frames of depth information in a specified depth sensing mode. Once the measurement is complete, method 600 returns to box 650, where the depth sensing operating mode can be specified again.

[0092] Figure 6 The operating method 600 shown is advantageous because it improves depth sensing efficiency by reducing the amount of time dedicated to programming the depth sensor 100 in response to changes in the requested depth sensing operating mode. According to method 600, the depth sensor can alternate between multiple depth sensing modes without incurring the time loss required to reprogram the memory box. Furthermore, this can be achieved using fewer memory boxes in the depth sensor 100 than is required using conventional techniques. Therefore, the efficiency and speed of depth information collection can be improved. Additionally, a lower-cost depth sensor with fewer memory boxes can be used.

[0093] Figure 9 This is an example timing diagram for operation in High Dynamic Range (HDR) depth sensing mode. HDR depth sensing mode includes interleaved long-range and short-range measurements. Table 5 shows example operating sequences for HDR depth sensing mode. In some embodiments, the frame rate for HDR depth measurements is 5 Hz. The time period for HDR depth measurements is... Figure 9 The middle is marked as T fps This HDR depth sensing mode can be used Figure 6 Methods and Figure 7 The depth sensor memory box programming scheme shown is used to execute this.

[0094] The HDR depth sensing sequence begins in step 0, where long-range intensity subframes are captured. Then, during steps 1-4, the depth sensor 100 captures four phase subframes using a first modulation frequency. Figure 9 As shown, each phase subframe has T LR-int The exposure time or integration time. Each of these exposures is followed by a readout time T for transferring the captured image data from the sensor. readout Then, in steps 5-8, the depth sensor 100 captures four phase subframes using the second modulation frequency.

[0095]

[0096] Table 5

[0097] Following the long-range measurement is the eye-safe delay at step 9, which is... Figure 9 The middle is marked as T eye_safe_dummy This delay prevents the eye-safe circuitry in the depth sensor 100 from being triggered, ensuring that the light source of the depth sensor is not turned off during measurement. This delay is another example of a virtual operation step and is discussed further below. Figure 9 As shown, in some embodiments, the virtual operation step constituting the delay can be implemented as an intensity subframe (including an exposure period and a readout period), followed by an idle period. The intensity subframes captured during the virtual operation step are typically not used to calculate depth.

[0098] Next, in step 10 of Table 5, the depth sensor captures short-range intensity subframes. Four phase subframes follow in steps 11-14. (As...) Figure 9 As shown, each of these subframes has T SR_int The exposure time is followed by the readout period. An optional delay may follow the short-range measurements performed during steps 10-14 at step 15.

[0099] Table 5 and Figure 9 The HDR depth sensing mode shown has many operational steps that are common to other depth sensing modes discussed herein. For example, steps 0-8 in Table 5 are the same as those used in high frame rate long-range depth measurement. Therefore, this part of the HDR depth sensing mode can be implemented by performing... Figure 7 Steps 1 to m in the programming scheme 700 shown are implemented. Similarly, steps 10-14 in Table 5 are the same as those used in high frame rate short-range depth measurement. Therefore, they can be implemented as follows: Figure 7 Steps m+2 to m+n+1 in the programming scheme 700 shown.

[0100] The difference between the HDR depth sensing modes and the high frame rate long-range and high frame rate short-range modes lies in the eye-safe period at step 9 in Table 5 and the optional delay in step 15. As discussed herein, these differences can be achieved through appropriate virtual operating frames. For example, the eye-safe period can be achieved through… Figure 7 The eye-safe virtual operation steps shown at step m+1 in the programming scheme 700 illustrated are implemented. Furthermore, the optional delay in step 15 of Table 5 can be achieved using… Figure 7 The virtual operation at step m+n+2 of the programming scheme 700 shown is implemented.

[0101] Next, as Figure 8 As shown, it can be achieved by executing Figure 7 The programming scheme 700 shown in the diagram implements the complete HDR depth sensing mode (interleaved long-range and short-range depth measurements) through steps 1 to m+n+2. Therefore, the HDR depth sensing mode in Table 5 is based on... Figure 7 The public and virtual operation steps shown are examples of implementing additional modes.

[0102] The foregoing disclosure describes various effective depth sensing techniques for VR / AR / MR systems. Although these techniques have been discussed specifically for depth sensors, the same techniques can be applied to other types of sensors, and are not limited to depth sensors.

[0103] Example Implementation

[0104] In some embodiments, a method includes: providing a sensor with a sequence of common operating steps, the sequence of common operating steps being included in both a sequence of first operating steps defining a first operating mode and a sequence of second operating steps defining a second operating mode; providing the sensor with one or more virtual operating steps relating to a difference between the first and second operating modes; operating the sensor in a first operating mode by causing the sensor to perform at least the common operating steps; and operating the sensor in a second operating mode by causing the sensor to perform the common operating steps and at least one virtual operating step.

[0105] According to any of these embodiments, the first operating mode may include performing at least a common operating step at a first rate, and the second operating mode may include performing the common operating step and at least one virtual operating step at a second rate slower than the first rate.

[0106] According to any of these embodiments, at least one of the virtual operation steps may include a delay.

[0107] According to any of these embodiments, providing the sensor with a sequence of common operating steps and one or more virtual operating steps may include storing those operating steps in the sensor memory.

[0108] According to any of these embodiments, switching the sensor between the first and second operating modes may not require any additional action to store the operating steps in the sensor memory.

[0109] According to any of these embodiments, the sensor may include a depth sensor.

[0110] According to any of these embodiments, the depth sensor may include a time-of-flight camera.

[0111] According to any of these embodiments, the first operating mode may include a depth sensing mode having a first frame rate, and the second operating mode may include a depth sensing mode having a second frame rate that is slower than the first frame rate.

[0112] According to any of these embodiments, the method may further include providing depth information from a depth sensor to a virtual reality, augmented reality, or mixed reality display system.

[0113] In some embodiments, a system includes a processor configured to perform a method comprising: providing a sensor with a sequence of common operating steps, the sequence of common operating steps being included in both a sequence of first operating steps defining a first operating mode and a sequence of second operating steps defining a second operating mode; providing the sensor with one or more virtual operating steps relating to a difference between the first operating mode and the second operating mode; operating the sensor in a first operating mode by causing the sensor to perform at least the common operating steps; and operating the sensor in a second operating mode by causing the sensor to perform the common operating steps and at least one virtual operating step.

[0114] According to any of these embodiments, the first operating mode may include performing at least a common operating step at a first rate, and the second operating mode may include performing the common operating step and at least one virtual operating step at a second rate slower than the first rate.

[0115] According to any of these embodiments, at least one of the virtual operation steps may include a delay.

[0116] According to any of these embodiments, providing the sensor with a sequence of common operating steps and one or more virtual operating steps may include storing those operating steps in the sensor memory.

[0117] According to any of these embodiments, switching the sensor between the first and second operating modes may not require any additional action to store the operating steps in the sensor memory.

[0118] According to any of these embodiments, the sensor may include a depth sensor.

[0119] According to any of these embodiments, the depth sensor may include a time-of-flight camera.

[0120] According to any of these embodiments, the first operating mode may include a depth sensing mode having a first frame rate, and the second operating mode may include a depth sensing mode having a second frame rate that is slower than the first frame rate.

[0121] According to any of these embodiments, the system can be integrated into a virtual reality, augmented reality, or mixed reality display system.

[0122] According to any of these embodiments, the processor may include a state machine.

[0123] According to any of these embodiments, the system may further include an arbitrator for receiving a request to operate the sensor in a first mode or a second mode, and the arbitrator may be configured to schedule and prioritize the request.

[0124] According to any of these embodiments, the system may further include a sensor.

[0125] In some embodiments, a method includes: receiving a first request for a first type of depth measurement; receiving a second request for a second type of depth measurement; assigning a first priority to the first request; assigning a second priority to the second request; and causing a depth sensor to first obtain a first type of depth measurement if the first priority is higher than the second priority, or causing a depth sensor to first obtain a second type of depth measurement if the second priority is higher than the first priority.

[0126] According to any of these embodiments, a first priority can be assigned based on the priority of a first application requesting a first type of depth measurement, and a second priority can be assigned based on the priority of a second application requesting a second type of depth measurement.

[0127] In some embodiments, a system includes: an arbitrator configured to receive a first request for a first type of depth measurement and a second request for a second type of depth measurement, the arbitrator being configured to assign a first priority to the first request and a second priority to the second request; and a processor configured to cause a depth sensor to first obtain a first type of depth measurement if the first priority is higher than the second priority, or to cause the depth sensor to first obtain a second type of depth measurement if the second priority is higher than the first priority.

[0128] According to any of these embodiments, the arbitrator can be configured to assign a first priority based on the priority of a first application requesting a first type of depth measurement, and to assign a second priority based on the priority of a second application requesting a second type of depth measurement.

[0129] According to any of these embodiments, the system can be integrated into a virtual, augmented, or mixed reality display system.

[0130] In some embodiments, a method includes performing a configuration operation of a depth sensor, the configuration operation including: storing a sequence of first operation steps defining a first depth sensing operation mode in a memory of the depth sensor; and storing a sequence of second operation steps defining a second depth sensing operation mode in the memory of the depth sensor; receiving a first request for depth measurement according to the first depth sensing operation mode; operating the depth sensor in the first operation mode in response to the first request by causing the depth sensor to perform the sequence of the first operation steps; receiving a second request for depth measurement according to the second depth sensing operation mode; and operating the depth sensor in the second operation mode in response to the second request, without performing additional configuration operations, by causing the depth sensor to perform the sequence of the second operation steps.

[0131] According to any of these embodiments, the depth sensor may include a time-of-flight camera.

[0132] According to any of these embodiments, a first depth sensing operation mode may correspond to a first distance measurement range, and a second depth sensing operation mode may correspond to a second distance measurement range different from the first distance measurement range.

[0133] According to any of these embodiments, a first depth sensing operation mode may correspond to a first frame rate, and a second depth sensing operation mode may correspond to a second frame rate that is slower than the first frame rate.

[0134] According to any of these embodiments, the method may further include providing depth information from a depth sensor to a virtual reality, augmented reality, or mixed reality display system.

[0135] In some embodiments, a system includes: a processor configured to perform a method comprising: performing a configuration operation of a depth sensor, the configuration operation including storing a sequence of first operation steps defining a first depth sensing operation mode in a memory of the depth sensor, and storing a sequence of second operation steps defining a second depth sensing operation mode in a memory of the depth sensor; receiving a first request for depth measurement according to the first depth sensing operation mode; operating the depth sensor in the first operation mode in response to the first request by causing the depth sensor to perform the sequence of the first operation steps; receiving a second request for depth measurement according to the second depth sensing operation mode; and operating the depth sensor in the second operation mode in response to the second request, without performing additional configuration operations, by causing the depth sensor to perform the sequence of the second operation steps.

[0136] According to any of these embodiments, the depth sensor may include a time-of-flight camera.

[0137] According to any of these embodiments, a first depth sensing operation mode may correspond to a first distance measurement range, and a second depth sensing operation mode may correspond to a second distance measurement range different from the first distance measurement range.

[0138] According to any of these embodiments, a first depth sensing operation mode may correspond to a first frame rate, and a second depth sensing operation mode may correspond to a second frame rate that is slower than the first frame rate.

[0139] According to any of these embodiments, the system can be integrated into a virtual, augmented, or mixed reality display system.

[0140] In some embodiments, the non-transitory computer-readable medium includes code that, when read by a computing device, causes the computing device to perform a method comprising the steps of: providing a sensor with a sequence of common operating steps, the sequence of common operating steps being included in both a sequence of first operating steps defining a first operating mode and a sequence of second operating steps defining a second operating mode; providing the sensor with one or more virtual operating steps relating to a difference between the first and second operating modes; operating the sensor in a first operating mode by causing the sensor to perform at least the common operating steps; and operating the sensor in a second operating mode by causing the sensor to perform the common operating steps and at least one virtual operating step.

[0141] According to any of these embodiments, the first operating mode may include performing at least a common operating step at a first rate, and the second operating mode may include performing the common operating step and at least one virtual operating step at a second rate slower than the first rate.

[0142] According to any of these embodiments, at least one of the virtual operation steps may include a delay.

[0143] According to any of these embodiments, providing the sensor with a sequence of common operating steps and one or more virtual operating steps may include storing those operating steps in the sensor memory.

[0144] According to any of these embodiments, switching the sensor between the first and second operating modes may not require any additional action to store the operating steps in the sensor memory.

[0145] According to any of these embodiments, the sensor may include a depth sensor.

[0146] According to any of these embodiments, the depth sensor may include a time-of-flight camera.

[0147] According to any of these embodiments, the first operating mode may include a depth sensing mode having a first frame rate, and the second operating mode may include a depth sensing mode having a second frame rate that is slower than the first frame rate.

[0148] According to any of these embodiments, the computer-readable medium may include code that further enables a computing device to provide depth information from a depth sensor to a virtual reality, augmented reality, or mixed reality display system.

[0149] In some embodiments, the non-transitory computer-readable medium includes code that, when read by a computing device, causes the computing device to perform a method comprising: receiving a first request for a first type of depth measurement; receiving a second request for a second type of depth measurement; assigning a first priority to the first request; assigning a second priority to the second request; and causing a depth sensor to first acquire a first type of depth measurement if the first priority is higher than the second priority, or causing a depth sensor to first acquire a second type of depth measurement if the second priority is higher than the first priority.

[0150] According to any of these embodiments, a first priority may be assigned based on the priority of a first application requesting a first type of depth measurement, and a second priority may be assigned based on the priority of a second application requesting a second type of depth measurement.

[0151] In some embodiments, the non-transitory computer-readable medium includes code that, when read by a computing device, causes the computing device to perform a method for operating a depth sensor, the method comprising: performing a configuration operation of the depth sensor, the configuration operation comprising: storing a sequence of first operation steps defining a first depth sensing operation mode in a memory of the depth sensor, and storing a sequence of second operation steps defining a second depth sensing operation mode in a memory of the depth sensor; receiving a first request for depth measurement according to the first depth sensing operation mode; operating the depth sensor in the first operation mode in response to the first request by causing the depth sensor to perform the sequence of the first operation steps; receiving a second request for depth measurement according to the second depth sensing operation mode; and operating the depth sensor in the second operation mode in response to the second request, without performing additional configuration operations, by causing the depth sensor to perform the sequence of the second operation steps.

[0152] According to any of these embodiments, the depth sensor may include a time-of-flight camera.

[0153] According to any of these embodiments, a first depth sensing operation mode may correspond to a first distance measurement range, and a second depth sensing operation mode may correspond to a second distance measurement range different from the first distance measurement range.

[0154] According to any of these embodiments, a first depth sensing operation mode may correspond to a first frame rate, and a second depth sensing operation mode may correspond to a second frame rate that is slower than the first frame rate.

[0155] According to any of these embodiments, the computer-readable medium may further include code that enables a computing device to provide depth information from a depth sensor to a virtual reality, augmented reality, or mixed reality display system.

[0156] Other considerations

[0157] For the purpose of summarizing this disclosure, certain aspects, advantages, and features of the invention have been described herein. It should be understood that not all of these advantages may be achieved according to any particular embodiment of the invention. Therefore, the invention may be embodied or practiced in a manner that realizes or optimizes one or more advantages taught herein, without necessarily realizing other advantages that may be taught or suggested herein.

[0158] Embodiments have been described in conjunction with the accompanying drawings. However, it should be understood that the drawings are not drawn to scale. Distances, angles, etc., are merely illustrative and do not necessarily have a precise relationship with the actual size and layout of the illustrated devices. Furthermore, the foregoing embodiments have been described at a level of detail to allow those skilled in the art to make and use the devices, systems, methods, etc., described herein. Various variations are possible. Components, elements, and / or steps may be altered, added, removed, or rearranged.

[0159] The devices and methods described herein can be advantageously implemented, at least in part, using, for example, computer software, hardware, firmware, or any combination of software, hardware, and firmware. Software modules may include computer-executable code stored in computer memory for performing the functions described herein. In some embodiments, the computer-executable code is executed by one or more general-purpose computers. However, based on this disclosure, those skilled in the art will understand that any module that can be implemented using software intended to execute on a general-purpose computer can also be implemented using different combinations of hardware, software, or firmware. For example, such modules may be implemented entirely in hardware using a combination of integrated circuits. Alternatively or additionally, such modules may be implemented entirely or in part using a special-purpose computer designed to perform the specific functions described herein, rather than a general-purpose computer. Furthermore, in cases where methods are described or can be at least partially executed by computer software, it should be understood that such methods may be provided on a non-transitory computer-readable medium (e.g., optical discs (such as CDs or DVDs), hard disks, flash memory, magnetic disks, etc.) that, when read by a computer or other processing device, causes it to execute the method.

[0160] While some embodiments have been clearly described, other embodiments will become apparent to those skilled in the art based on this disclosure.

Claims

1. A method for operating a depth sensor, the method comprising: Receive the first request for a first type of depth measurement; Receive a second request for a second type of depth measurement; Assign the first priority to the first request; Assign a second priority to the second request; as well as If the first priority is higher than the second priority, then the depth sensor is allowed to acquire the first type of depth measurement first; or if the second priority is higher than the first priority, then the depth sensor is allowed to acquire the second type of depth measurement first. in: The first request is for short-range depth measurement. The second request is for long-range depth measurement, and The first priority is higher than the second priority. The short-range depth measurement includes short-range high frame rate depth measurement and short-range low frame rate depth measurement. The long-range depth measurement includes long-range high frame rate depth measurement and long-range low frame rate depth measurement.

2. The method according to claim 1, wherein, The first priority is assigned based on the priority of the first application requesting the first type of depth measurement, and the second priority is assigned based on the priority of the second application requesting the second type of depth measurement.

3. A system for operating a depth sensor, the system comprising: An arbitrator configured to receive a first request for a first type of depth measurement and a second request for a second type of depth measurement, the arbitrator being configured to assign a first priority to the first request and a second priority to the second request; as well as A processor configured to cause the depth sensor to first acquire a depth measurement of the first type if the first priority is higher than the second priority, or to cause the depth sensor to first acquire a depth measurement of the second type if the second priority is higher than the first priority. in: The first request is for short-range depth measurement. The second request is for long-range depth measurement, and The first priority is higher than the second priority. The short-range depth measurement includes short-range high frame rate depth measurement and short-range low frame rate depth measurement. The long-range depth measurement includes long-range high frame rate depth measurement and long-range low frame rate depth measurement.

4. The system according to claim 3, wherein, The arbitrator is configured to assign the first priority based on the priority of a first application requesting the first type of depth measurement, and to assign the second priority based on the priority of a second application requesting the second type of depth measurement.

5. The system according to claim 3, wherein, The system is integrated into a virtual, augmented, or mixed reality display system.

Images