Lightweight and low power cross reality device with high temporal resolution
Patent Information
- Application Number
- JP2024134031
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-02-07
- Filing Date
- 2024-08-09
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-02-07
AI Technical Summary
Existing wearable cross-reality (XR) systems face challenges with weight, power consumption, and accuracy in tracking head pose and object movement due to the need for high-resolution sensors and frequent recalibration, which can lead to unaligned virtual and physical objects, reducing user enjoyment and system usability.
A wearable display system equipped with dynamic vision sensor (DVS) cameras and a processor that asynchronously tracks head pose and object movement using intensity change criteria, selectively activates sensors, and implements a tracking routine to limit image data acquisition to points of interest, enabling high-temporal resolution and low-power operation.
The system achieves accurate and efficient tracking of head pose and object movement with reduced power consumption, maintaining a realistic XR experience by aligning virtual objects with physical surroundings, thus enhancing user enjoyment and system longevity.
Abstract
Description
[Technical field]
[0001] This application generally relates to wearable cross-reality display systems (XR systems) that include dynamic visual sensor (DVS) cameras. [Background technology]
[0002] A computer may control a human user interface and create an X-reality (XR or cross-reality) environment in which some or all of the XR environment is generated by the computer as it is perceived by the user. These XR environments may be virtual reality (VR), augmented reality (AR), or mixed reality (MR) environments in which some or all of the XR environment may be generated by the computer, in part, using data that describes the environment. This data may describe virtual objects, for example, that may be rendered in a way that the user senses or perceives as part of the physical world so that the user may interact with the virtual objects. The user may experience these virtual objects as a result of data being rendered and presented through a user interface device, such as a head-mounted display device. The data may control audio that may be displayed to the user for viewing or played to the user for hearing, or may control a tactile (or haptic) interface, allowing the user to experience touch sensations that the user senses or perceives as they feel the virtual objects.
[0003] XR systems can be useful for many applications ranging from scientific visualization, medical training, engineering design and prototyping, remote manipulation and telepresence, and personal entertainment. AR and MR, in contrast to VR, involve one or more virtual objects in association with real objects in the physical world. The experience of virtual objects interacting with real objects generally enhances the user's enjoyment when using XR systems and also opens up possibilities for a variety of applications that present realistic and easily understandable information about how the physical world can be altered. Summary of the Invention [Means for solving the problem]
[0004] Aspects of the present application relate to a wearable cross-reality display system configured with a DVS camera. The techniques described herein may be used together, separately, or in any suitable combination.
[0005] According to some embodiments, a wearable display system may be provided, the wearable display system comprising: a headset including a first camera and a second camera configurable to output image frames or image data that satisfy an intensity change criterion, the first camera and the second camera positioned to provide overlapping views of a central field of view; and a processor operatively coupled to the first camera and the second camera and configured to create a world model using stereoscopically determined depth information from images output by the first camera and the second camera, and to track head pose using the world model and the image data output by the first camera.
[0006] In some embodiments, the intensity change criteria may include absolute or relative intensity change criteria.
[0007] In some embodiments, the first camera may be configured to output image data asynchronously.
[0008] In some embodiments, the processor may be further configured to track head pose asynchronously.
[0009] In some embodiments, the processor may be further configured to implement a tracking routine and limit image data acquisition to points of interest within the world model.
[0010] In some embodiments, the first camera may be configurable to limit image acquisition to one or more portions of the first camera's field of view, and the tracking routine may include steps of identifying a point of interest in the world model, determining one or more first portions of the first camera's field of view that correspond to the point of interest, and providing instructions to the first camera to limit image acquisition to the one or more first portions of the field of view.
[0011] In some embodiments, the tracking routine may further include estimating one or more second portions of the field of view of the first camera that correspond to the point of interest based on movement of the point of interest relative to the world model or movement of the headset relative to the point of interest, and providing instructions to the first camera to limit image acquisition to the one or more second portions of the field of view.
[0012] In some embodiments, the headset may further include an inertial measurement unit, and performing the tracking routine may include estimating an updated relative position of the object based, at least in part, on an output of the inertial measurement unit.
[0013] In some embodiments, the tracking routine may include repeatedly calculating the position of the point of interest within the world model, and the repeated calculations may be performed with a temporal resolution of greater than 60 Hz.
[0014] In some embodiments, the interval between repeated calculations may be between 1 ms and 15 ms in duration.
[0015] In some embodiments, the processor may be further configured to determine whether the head pose tracking satisfies a quality criterion, and to enable a second camera or modulate the frame rate of the second camera when the head pose tracking does not satisfy the quality criterion.
[0016] In some embodiments, the processor may be mechanically coupled to the headset.
[0017] In some embodiments, the headset may include a display device that is mechanically coupled to the processor.
[0018] In some embodiments, the local data processing module may comprise a processor, the local data processing module is operably coupled to a display device through a communications link, and the headset may comprise a display device.
[0019] In some embodiments, the headset may further include an IR emitter.
[0020] In some embodiments, the processor may be configured to selectively enable IR emitters to enable head pose tracking in low light conditions.
[0021] According to some embodiments, a method of tracking head pose using a wearable display system may be provided, the wearable display system comprising a headset including a first camera and a second camera configurable to output image frames or image data that satisfy an intensity change criterion, the first camera and the second camera positioned to provide overlapping views of a central field of view, and a processor operably coupled to the first camera and the second camera, the method including using the processor to create a world model using stereoscopically determined depth information from images output by the first camera and the second camera, and tracking head pose using the world model and the image data output by the first camera.
[0022] According to some embodiments, a wearable display system may be provided that includes a frame, a first camera mechanically coupled to the frame, the first camera configurable to output image data that satisfies an intensity change criterion within a first field of view for the first camera, and a processor operatively coupled to the first camera and configured to determine whether an object is within the first field of view and track movement of the object using image data received from the first camera for one or more portions of the first field of view.
[0023] According to some embodiments, a method is provided for tracking movement of an object using a wearable display system comprising a frame, a first camera mechanically coupled to the frame, the first camera configurable to output image data that satisfies an intensity change criterion within a first field of view for the first camera, and a processor operably coupled to the first camera, the method including determining, using the processor, whether an object is within the first field of view, and tracking movement of the object using image data received from the first camera for one or more portions of the first field of view.
[0024] According to some embodiments, a wearable display system is provided comprising: a frame; two cameras, a first camera and a second camera, mechanically coupled to the frame, the two cameras being configurable to output image data that satisfies an intensity change criterion, the first camera and the second camera being positioned to provide overlapping views of a central visual field; and a processor operably coupled to the first camera and the second camera.
[0025] The above description is provided by way of illustration and is not intended to be limiting. The present invention provides, for example, the following: (Item 1) 1. A wearable display system, comprising: 1. A headset, comprising: a first camera, the first camera being configurable to output image frames or image data that satisfy an intensity change criterion; and 1 second camera and Including, a headset, the first camera and the second camera positioned to provide overlapping views of a central field of view; a processor operatively coupled to the first camera and the second camera, the processor comprising: creating a world model using stereoscopically determined depth information from images output by the first camera and the second camera; tracking a head pose using the world model and image data output by the first camera; a processor configured to: A wearable display system comprising: (Item 2) 2. The wearable display system of claim 1, wherein the intensity change criteria include absolute or relative intensity change criteria. (Item 3) 2. The wearable display system of claim 1, wherein the first camera is configured to output the image data asynchronously. (Item 4) 4. The wearable display system of item 3, wherein the processor is further configured to asynchronously track head pose. (Item 5) 2. The wearable display system of claim 1, wherein the processor is further configured to perform a tracking routine and limit image data acquisition to points of interest within the world model. (Item 6) the first camera can be configured to limit image capture to one or more portions of a field of view of the first camera; The tracking routine comprises: identifying a point of interest within the world model; determining one or more first portions of a field of view of the first camera that correspond to the point of interest; providing instructions to the first camera to limit image capture to one or more first portions of the field of view; Item 6. The wearable display system of item 5, comprising: (Item 7) The tracking routine further comprises: estimating one or more second portions of the field of view of the first camera that correspond to the point of interest based on a movement of the point of interest relative to the world model or a movement of the headset relative to the point of interest; providing instructions to the first camera to limit image acquisition to one or more second portions of the field of view; 7. The wearable display system of item 6, comprising: (Item 8) the headset further includes an inertial measurement unit; performing the tracking routine includes estimating an updated relative position of the object based at least in part on an output of the inertial measurement unit; Item 6. The wearable display system according to item 5. (Item 9) the tracking routine includes iteratively calculating a position of a point of interest within the world model; The repeated calculations are performed with a temporal resolution of greater than 60 Hz. Item 6. The wearable display system according to item 5. (Item 10) 10. The wearable display system of item 9, wherein the interval between repeated calculations is between 1 ms and 15 ms in duration. (Item 11) The processor further comprises: determining whether the head pose tracking satisfies a quality criterion; and enabling the second camera or modulating the frame rate of the second camera when the head pose tracking does not meet the quality criterion; Item 2. The wearable display system of item 1, configured to perform the following: (Item 12) 2. The wearable display system of claim 1, wherein the processor is mechanically coupled to the headset. (Item 13) Item 1 , the wearable display system of Item 1 , wherein the headset comprises a display device mechanically coupled to the processor. (Item 14) 2. The wearable display system of claim 1, wherein a local data processing module comprises the processor, the local data processing module is operably coupled to a display device via a communication link, and the headset comprises the display device. (Item 15) 2. The wearable display system of claim 1, wherein the headset further includes an IR emitter. (Item 16) Item 16. The wearable display system of item 15, wherein the processor is configured to selectively enable the IR emitters to enable head pose tracking under low light conditions. (Item 17) 1. A method for tracking head pose using a wearable display system, the wearable display system comprising: A headset, a first camera, the first camera being configurable to output image frames or image data that satisfy an intensity change criterion; and 1 second camera and Including, a headset, the first camera and the second camera positioned to provide overlapping views of a central field of view; a processor operatively coupled to the first camera and the second camera; Equipped with The method includes using the processor: creating a world model using stereoscopically determined depth information from images output by the first camera and the second camera; tracking the head pose using the world model and image data output by the first camera; A method comprising: (Item 18) 1. A wearable display system, comprising: A frame, a first camera mechanically coupled to the frame, the first camera configurable to output image data that satisfies an intensity change criterion within a first field of view for the first camera; A processor operatively coupled to the first camera, the processor comprising: determining whether an object is within the first field of view; tracking movement of the object using image data received from the first camera for one or more portions of the first field of view; a processor configured to: A wearable display system comprising: (Item 19) 1. A method for tracking movement of an object using a wearable display system, the wearable display system comprising: A frame, a first camera mechanically coupled to the frame, the first camera configurable to output image data that satisfies an intensity change criterion within a first field of view for the first camera; a processor operatively coupled to the first camera; Equipped with The method includes using the processor: determining whether the object is within the first field of view; tracking movement of the object using image data received from the first camera for one or more portions of the first field of view; A method comprising: (Item 20) 1. A wearable display system, comprising: A frame, Two cameras mechanically coupled to the frame, the two cameras comprising: a first camera configurable to output image data that satisfies an intensity change criterion; 1 second camera and Equipped with two cameras, the first camera and the second camera positioned to provide overlapping views of a central field of view; a processor operatively coupled to the first camera and the second camera; A wearable display system comprising: [Brief description of the drawings]
[0026] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures is represented by a like numeral. For purposes of clarity, not every component is labeled in every figure.
[0027] [Figure 1] FIG. 1 is a sketch illustrating an example of a simplified augmented reality (AR) scene, according to some embodiments.
[0028] [Diagram 2] FIG. 2 is a schematic diagram illustrating an example of an AR display system, according to some embodiments.
[0029] [Figure 3A] FIG. 3A is a schematic diagram illustrating a user wearing an AR display system that renders AR content as the user moves through a physical world environment, according to some embodiments.
[0030] [Figure 3B] FIG. 3B is a schematic diagram illustrating a viewing optics assembly and associated components, according to some embodiments.
[0031] [Figure 4] FIG. 4 is a schematic diagram illustrating an image sensing system, according to some embodiments.
[0032] [Figure 5A] FIG. 5A is a schematic diagram illustrating the pixel cell in FIG. 4 according to some embodiments.
[0033] [Figure 5B] FIG. 5B is a schematic diagram illustrating output events of the pixel cell of FIG. 5A, according to some embodiments.
[0034] [Figure 6] FIG. 6 is a schematic diagram illustrating an image sensor, according to some embodiments.
[0035] [Figure 7] FIG. 7 is a schematic diagram illustrating an image sensor, according to some embodiments.
[0036] [Figure 8] FIG. 8 is a schematic diagram illustrating an image sensor, according to some embodiments.
[0037] [Figure 9] FIG. 9 is a simplified flowchart of a method for image sensing, according to some embodiments.
[0038] [Figure 10] FIG. 10 is a simplified flowchart of the patch identification acts of FIG. 9 according to some embodiments.
[0039] [Figure 11] FIG. 11 is a simplified flowchart of the patch trajectory estimation acts of FIG. 9 according to some embodiments.
[0040] [Figure 12] FIG. 12 is a schematic diagram illustrating the patch trajectory estimation of FIG. 11 for one viewpoint, according to some embodiments.
[0041] [Figure 13] FIG. 13 is a schematic diagram illustrating the patch trajectory estimation of FIG. 11 with respect to viewpoint changes, according to some embodiments.
[0042] [Figure 14] FIG. 14 is a schematic diagram illustrating an image sensing system, according to some embodiments.
[0043] [Figure 15] FIG. 15 is a schematic diagram illustrating the pixel cell in FIG. 14 according to some embodiments.
[0044] [Figure 16] FIG. 16 is a schematic diagram of a pixel sub-array according to some embodiments.
[0045] [Figure 17A] FIG. 17A is a cross-sectional view of a plenoptic device with an angle-of-arrival / intensity converter in the form of two stacked aligned transmissive diffractive masks (TDMs) according to some embodiments.
[0046] [Figure 17B] FIG. 17B is a cross-sectional view of a plenoptic device with an angle-of-arrival / intensity converter in the form of two misaligned stacked TDMs according to some embodiments.
[0047] [Figure 18A] FIG. 18A is a pixel sub-array with color pixel cells and arrival corner pixel cells according to some embodiments.
[0048] [Figure 18B]FIG. 18B is a pixel sub-array with color pixel cells and arrival corner pixel cells according to some embodiments.
[0049] [Figure 18C] FIG. 18C is a pixel sub-array with white pixel cells and arrival corner pixel cells according to some embodiments.
[0050] [Figure 19A] FIG. 19A is a top view of a photodetector array with a single TDM according to some embodiments.
[0051] [Figure 19B] FIG. 19B is a side view of a photodetector array with a single TDM according to some embodiments.
[0052] [Figure 20A] FIG. 20A is a top view of a photodetector array with multiple angle-of-arrival / intensity converters in the form of a TDM according to some embodiments.
[0053] [Figure 20B] FIG. 20B is a side view of a photodetector array with multiple TDMs according to some embodiments.
[0054] [Figure 20C] FIG. 20C is a side view of a photodetector array with multiple TDMs according to some embodiments.
[0055] [Figure 21] FIG. 21 is a schematic diagram of a headset including two cameras and associated components according to some embodiments.
[0056] [Figure 22] FIG. 22 is a flow chart of a calibration routine, according to some embodiments.
[0057] [Figure 23A] 23A-23C depict example field of view diagrams associated with the headset of FIG. 21, according to some embodiments. [Figure 23B] 23A-23C depict example field of view diagrams associated with the headset of FIG. 21, according to some embodiments. [Figure 23C] 23A-23C depict example field of view diagrams associated with the headset of FIG. 21, according to some embodiments.
[0058] [Figure 24] FIG. 24 is a flowchart of a method for creating and updating a passable world model according to some embodiments.
[0059] [Diagram 25] FIG. 25 is a flowchart of a method for head pose tracking, according to some embodiments.
[0060] [Figure 26] FIG. 26 is a flowchart of a method for object tracking according to some embodiments.
[0061] [Figure 27] FIG. 27 is a flowchart of a method for hand tracking, according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0062] The inventors have recognized and appreciated design and operational techniques for wearable XR display systems that improve the enjoyment and usability of such systems. These design and / or operational techniques may enable the use of a limited number of cameras to obtain information for performing multiple functions, including hand tracking, head pose tracking, and world reconstruction, which may be used to realistically render virtual objects as they appear to interact realistically with physical objects. The wearable cross reality display system may be lightweight and consume low power in operation. The system may obtain image information about physical objects in the physical world with low latency using a particular configuration of sensors. The system may perform various routines to improve the accuracy and / or realism of the displayed XR environment. Such routines may include calibration routines to improve the accuracy of stereoscopic depth measurements even when a lightweight frame distorts during use, and routines to detect and address incomplete depth information in a model of the physical world around the user.
[0063] The weight of known XR system headsets may limit user enjoyment. Such XR headsets may weigh more than 340 grams (sometimes even more than 700 grams). Glasses, in contrast, may weigh less than 50 grams. Wearing such relatively heavy headsets for extended periods of time may tire users or distract them, diverting them from the desired immersive XR experience. However, the inventors recognize and understand that some designs that reduce headset weight may also increase headset flexibility, making lightweight headsets more susceptible to changes in sensor position or orientation during use or over time. For example, when a user wears a lightweight headset that includes camera sensors, the relative orientation of these camera sensors may shift. Variations in the spacing of cameras used for stereoscopic imaging may affect the ability of those headsets to obtain accurate stereoscopic information, which relies on cameras having known positional relationships with respect to each other. Thus, a calibration routine, which may be repeated when the headset is worn, may enable a lightweight headset that can use stereoscopic imaging techniques to accurately obtain information about the world around the headset wearer.
[0064] The need to equip XR systems with components to obtain information about objects in the physical world can also limit the usefulness and user enjoyment of these systems. Although the obtained information is used to realistically present computer-generated virtual objects in the appropriate locations and with the appropriate appearance relative to physical objects, the need to obtain the information imposes limitations on the size, power consumption, and realism of the XR system.
[0065] An XR system may obtain information about objects in the physical world around the user, including information about the location of the physical world object in the user's field of view, for example, using sensors worn by the user. Challenges arise because physical objects may move relative to the user's field of view, either as a result of the object moving in the physical world or the user changing its attitude relative to the physical world, such that the position of the physical object in the user's field of view changes, moving into or out of the user's field of view. To present a realistic XR display, the model of the physical objects in the physical world must be updated frequently enough to capture these changes, processed with low enough latency, and accurately predicted into the future to cover the full latency path, including rendering, such that the virtual objects displayed based on that information will have the appropriate position and appearance relative to the physical objects as they are displayed. Otherwise, the virtual objects will appear misaligned with the physical objects, and combined scenes including physical and virtual objects will not appear realistic. For example, the virtual objects may appear as if they are floating in space, rather than resting on the physical objects, or may appear to bounce around relative to the physical objects. Visual tracking errors are amplified when there is significant evolution in the scene, especially when the user is moving at high speed.
[0066] Such problems may be avoided by sensors that obtain new data at a high rate. However, the power consumed by such sensors may lead to the need for larger batteries, increase the weight of the system, or limit the length of use of such systems. Similarly, the processors required to process data generated at a high rate may drain the battery and add additional weight to the wearable system, further limiting the usefulness or enjoyability of such systems. A known approach is to operate a higher frame rate sensor for increased temporal resolution, for example, with a higher resolution to capture sufficient visual detail. An alternative solution may complement this solution with an IR time-of-flight sensor that may directly indicate the position of a physical object relative to the sensor, and simple processing, resulting in low latency, may be performed in displaying virtual objects using this information. However, such sensors consume a substantial amount of power, especially when they operate in sunlight.
[0067] The inventors have recognized and appreciated that an XR system may address changes in sensor position or orientation during use or over time by repeatedly performing a calibration routine. This calibration routine may determine the current relative separation and orientation of the sensors contained within the headset. The wearable XR system may then take this relative separation and orientation of the headset sensors into account when calculating stereoscopic depth information. With such calibration capabilities, the XR system may accurately obtain depth information indicating distance to objects in the physical world without active depth sensors or with only occasional use of active depth sensing. Because active depth sensing may consume substantial power, reducing or eliminating active depth sensing may enable the device to draw less power, which may increase the operating time of the device without recharging the battery, or reduce the size of the device as a result of reducing the size of the battery.
[0068] The inventors have also recognized and appreciated that with the appropriate combination of image sensors and appropriate techniques for processing image information from those sensors, an XR system may obtain information about physical objects with reduced power consumption and low latency by reducing the number of sensors used, eliminating, disabling or selectively activating resource-intensive sensors, and / or reducing overall sensor usage. As a specific example, an XR system may include a headset with two world-facing cameras. A first of the cameras may produce grayscale images and may have a global shutter. These grayscale images may be smaller in size than similar resolution color images, represented in some instances with less than one-third the number of bits. The grayscale camera may require less power than a similar resolution color camera. The grayscale camera may be configured to employ event-based data acquisition and / or patch tracking and limit the amount of data output. A second of the cameras may be an RGB camera. A wearable cross reality display system may be configured to selectively use the present camera to reduce power consumption and extend battery life without compromising the user's XR experience. An RGB camera may be used, for example, in conjunction with a grayscale to build a world model of the user's surrounding environment, while a grayscale camera may then be used to track the user's head pose based on the present world model. Alternatively, or in addition, the grayscale camera may be used primarily to track objects, including the user's hands. Based on detected conditions that indicate poor tracking quality, image information from one or more other sensors may be employed. An RGB camera may be enabled, for example, to obtain color information that aids in distinguishing objects from the background, or output from a light field sensor may be used that passively provides depth information.
[0069] The techniques described herein may be used with many types of devices or separately and for many types of scenarios. Figure 1 illustrates such a scenario. Figures 2, 3A, and 3B illustrate an example AR system including one or more processors, memory, sensors, and a user interface that may operate according to the techniques described herein.
[0070] 1, an AR scene 4 is depicted in which a user of the AR system sees a physical world park-like setting 6 featuring people, trees, buildings in the background, and a concrete platform 8. In addition to these physical objects, a user of the AR technology also perceives as "seeing" a virtual object, here illustrated as a robotic figure 10 standing on the physical world concrete platform 8, and a flying cartoon-like avatar character 2 that appears to be an anthropomorphic bumblebee, although these elements (e.g., the avatar character 2 and the robotic figure 10) do not exist within the physical world. Due to the significant complexity of the human visual perception and nervous system, it is difficult to produce an AR system that facilitates a comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or physical world image elements.
[0071] Such scenes may be presented to the user by presenting image information that represents the real environment around the user and overlaying information that represents virtual objects that are not in the real environment. In an AR system, the user may be able to see objects in the physical world, and the AR system provides information that renders the virtual objects so that they appear in the appropriate location and with the appropriate visual characteristics that make the virtual objects appear to coexist with the objects in the physical world. In an AR system, for example, the user may look through a transparent screen so that the user can see objects in the physical world. The AR system may render virtual objects on its screen so that the user can see both the physical world and the virtual objects. In some embodiments, the screen may be worn by the user, such as a pair of goggles or glasses.
[0072] The scene may be presented to the user via a system that includes multiple components, including a user interface that may stimulate one or more of the user's senses, including vision, hearing, and / or touch. In addition, the system may include one or more sensors that may measure parameters of the physical portion of the scene, including the user's position and / or movement within the physical portion of the scene. Furthermore, the system may include one or more computing devices with associated computer hardware such as memory. These components may be integrated into a single device or even distributed across multiple interconnected devices. In some embodiments, some or all of these components may be integrated into a wearable device.
[0073] In some embodiments, the AR experience may be provided to a user through a wearable display system. FIG. 2 illustrates an example of a wearable display system 80 (hereinafter referred to as "system 80"). The system 80 includes a head-mounted display device 62 (hereinafter referred to as "display device 62") and various mechanical and electronic modules and systems to support the functionality of the display device 62. The display device 62 may be coupled to a frame 64, which is wearable by a user or viewer 60 of the display system (hereinafter referred to as "user 60") and configured to position the display device 62 in front of the user's eyes 60. According to various embodiments, the display device 62 may be a sequential display. The display device 62 may be monocular or binocular.
[0074] In some embodiments, a speaker 66 is coupled to the frame 64 and positioned proximate to the ear canal of the user 60. In some embodiments, another speaker, not shown, is positioned adjacent to another ear canal of the user 60 to provide stereo / adjustable sound control.
[0075] The system 80 may include a local data processing module 70. The local data processing module 70 may be operatively coupled to the display device 62 through a communication link 68, such as by wired leads or wireless connectivity. The local data processing module 70 may be mounted in a variety of configurations, such as fixedly attached to the frame 64, fixedly attached to a helmet or hat worn by the user 60, built into headphones, or otherwise removably attached to the user 60 (e.g., in a backpack configuration, in a belt-coupled configuration). In some embodiments, the local data processing module 70 may not be present, as components of the local data processing module 70 may be integrated into the display device 62 or implemented within a remote server or other component to which the display device 62 is coupled, such as through wireless communication over a wide area network.
[0076] The local data processing module 70 may include a processor and digital memory such as non-volatile memory (e.g., flash memory), both of which may be utilized to aid in processing, caching, and storing data. The data may include a) data captured from sensors (e.g., operably coupled to the frame 64 or otherwise attached to the user 60, such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes), and / or b) data obtained and / or processed using the remote processing module 72 and / or remote data repository 74, possibly for passing to the display device 62 after processing or retrieval. The local data processing module 70 may be operably coupled to the remote processing module 72 and the remote data repository 74 by communication links 76, 78, such as via wired or wireless communication links, respectively, such that these remote modules 72, 74 are operably coupled to each other and available as resources to the local processing and data module 70.
[0077] In some embodiments, the local data processing module 70 may include one or more processors (e.g., a central processing unit and / or one or more graphics processing units (GPUs)) configured to analyze and process the data and / or image information. In some embodiments, the remote data repository 74 may include a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local data processing module 70, allowing for fully autonomous use from the remote modules.
[0078] In some embodiments, the local data processing module 70 is operably coupled to a battery 82. In some embodiments, the battery 82 is a removable power source, such as a commercially available battery. In other embodiments, the battery 82 is a lithium-ion battery. In some embodiments, the battery 82 includes both an internal lithium-ion battery that is rechargeable by the user 60 during periods of non-operation of the system 80, and a removable battery so that the user 60 may operate the system 80 for longer periods of time without having to plug in a power source and recharge the lithium-ion battery or shut off the system 80 and replace the battery.
[0079] FIG. 3A illustrates a user 30 wearing an AR display system that renders AR content as the user 30 moves through a physical world environment 32 (hereafter referred to as “environment 32”). The user 30 positions the AR display system at a location 34, and the AR display system records ambient information of the passable world (e.g., digital representations of real objects in the physical world that can be stored and updated as the real objects in the physical world change) for the location 34. Each location 34 may further be associated with a “pose” and / or mapped features or directional audio inputs related to the environment 32. A user wearing the AR display system on their head may look in a particular direction and tilt their head to create a head pose of the system relative to the environment. At each location and / or pose within the same location, sensors on the AR display system may capture different information about the environment 32. Thus, information collected at the locations 34 may be aggregated to a data input 36 and processed by at least a passable world module 38, which may be implemented, for example, by processing on the remote processing module 72 of FIG. 2.
[0080] The passable world module 38 determines where and how the AR content 40 may be placed relative to the physical world, as determined at least in part from the data input 36. The AR content is "placed" in the physical world by presenting the AR content in a manner such that both the AR content and the physical world are visible to the user. Such an interface may be created, for example, with glasses that the user can see through and view the physical world, and that may be controlled so that virtual objects appear at controlled locations within the user's field of view. The AR content is rendered as if interacting with objects in the physical world. The user interface is such that the user's view of objects in the physical world may be obscured, when appropriate, to create the appearance that the AR content obscures the user's view of those objects. For example, the AR content may be placed by appropriately selecting a portion of an element 42 (e.g., a table) in the environment 32 and displaying and displaying the AR content 40 shaped and positioned as if resting on the element 42 or otherwise interacting with it. AR content may also be placed within structures not already within the field of view 44, or relative to a mapped mesh model 46 of the physical world.
[0081] As depicted, elements 42 are examples of what may be multiple elements in the physical world that may be treated as if they were fixed and stored in the passable world module 38. Once stored in the passable world module 38, information about those fixed elements may be used to present information to the user 30 such that the user 30 may perceive content on the fixed elements 42 without the system having to map it to the fixed elements 42 every time the user 30 sees it. The fixed elements 42 may thus be mesh models mapped from a previous modeling session or determined from a separate user, but stored on the passable world module 38 for future reference by multiple users. Thus, the passable world module 38 may recognize the environment 32 from a previously mapped environment and display the AR content without the user 30's device having to map the environment 32 first, saving computational processes and cycles and avoiding latency of any rendered AR content.
[0082] Similarly, a mapped mesh model 46 of the physical world can be created by the AR display system, and appropriate surfaces and metrics for interacting with and displaying the AR content 40 can be mapped and stored in the passable world module 38 for future retrieval by the user 30 or other users, without the need to remap or model. In some embodiments, the data inputs 36 are inputs such as geographic location, user identification, and current activity to indicate to the passable world module 38 which fixed elements 42 of one or more fixed elements are available, which AR content 40 was last placed on the fixed elements 42, and whether that same content should be displayed (such AR content is "persistent" content regardless of whether the user is viewing a particular passable world model).
[0083] Even in embodiments where objects are considered to be fixed, the passable world module 38 may be updated from time to time to account for possible changes in the physical world. Models of fixed objects may be updated at a very low frequency. Other objects in the physical world may be moving or otherwise not considered to be fixed. To render the AR scene with a sense of realism, the AR system may update the positions of these non-fixed objects at a much higher frequency than that used to update fixed objects. To enable accurate tracking of all of the objects in the physical world, the AR system may draw on information from multiple sensors, including one or more image sensors.
[0084] FIG. 3B is a schematic diagram of the viewing optics assembly 48 and associated optional components. A specific configuration is described in FIG. 21 below. When oriented toward the user's eye 49, in some embodiments, the two eye tracking cameras 50 detect metrics of the user's eye 49, such as eye shape, eyelid occlusion, pupil direction, and phosphenes on the user's eye 49. In some embodiments, one of the sensors may be a depth sensor 51, such as a time-of-flight sensor, that emits signals into the world and detects reflections of those signals from nearby objects to determine the distance to a given object. The depth sensor may, for example, quickly determine whether objects have entered the user's field of view, either as a result of the objects' motion or a change in the user's posture. However, information about the location of objects in the user's field of view may alternatively or additionally be collected using other sensors. In some embodiments, the world camera 52 records a view larger than the peripheral vision, maps the environment 32, and detects inputs that may affect the AR content. In some embodiments, world camera 52 and / or camera 53 may be grayscale and / or color image sensors that may output grayscale and / or color image frames at fixed time intervals. Camera 53 may also capture physical world images within the user's field of view at a specific time. The pixels of a frame-based image sensor may be repeatedly sampled even if their values are immutable. World camera 52, camera 53, and depth sensor 51 each have a separate field of view 54, 55, and 56, and collect and record data from a physical world scene, such as physical world environment 32 depicted in FIG. 3A.
[0085] The inertial measurement unit 57 may determine the movement and / or orientation of the viewing optics assembly 48. In some embodiments, each component is operably coupled to at least one other component. For example, the depth sensor 51 may be operably coupled to the eye tracking camera 50 to ascertain the actual distance of points and / or areas in the physical world that the user's eyes 49 are viewing.
[0086] It should be understood that the viewing optics assembly 48 may include some of the components illustrated in FIG. 3B. For example, the viewing optics assembly 48 may include a different number of components. In some embodiments, for example, the viewing optics assembly 48 may include one world camera 52, two world cameras 52, or more world cameras instead of the four world cameras depicted. Alternatively, or in addition, the cameras 52 and 53 need not capture visible light images of their full field of view. The viewing optics assembly 48 may include other types of components. In some embodiments, the viewing optics assembly 48 may include one or more dynamic visual sensors, the pixels of which may respond asynchronously to relative changes in light intensity that exceed a threshold.
[0087] In some embodiments, the viewing optics assembly 48 may not include a depth sensor 51 based on time-of-flight information. In some embodiments, for example, the viewing optics assembly 48 may include one or more plenoptic cameras, whose pixels may capture not only light intensity but also the angle of incident light. For example, the plenoptic camera may include an image sensor overlaid with a transmissive diffractive mask (TDM). Alternatively or in addition, the plenoptic camera may include an image sensor containing angle-sensing pixels and / or phase-detection autofocus pixels (PDAF) and / or a microlens array (MLA). Such sensors may serve as a source of depth information instead of or in addition to the depth sensor 51.
[0088] It should also be understood that the configuration of components in Figure 3B is illustrated as an example. Viewing optics assembly 48 may include components with any suitable configuration so that a user may have a maximum field of view for a particular set of components. For example, if viewing optics assembly 48 has one world camera 52, the world camera may be located in a central region of the viewing optics assembly instead of on a side.
[0089] Information from these sensors in the viewing optics assembly 48 may be coupled to one or more of the processors in the system. The processor may generate data that may be rendered to cause the user to perceive virtual content interacting with objects in the physical world. The rendering may be implemented in any suitable manner, including generating image data that depicts both physical and virtual objects. In other embodiments, physical and virtual content may be depicted in a scene by modulating the opacity of a display device through which the user views the physical world. The opacity may be controlled to create the appearance of virtual objects and also block the user from seeing objects in the physical world that are occluded by the virtual objects. In some embodiments, the image data may include only the virtual content that may be viewed through a user interface, which may be modified to interact realistically with the physical world (e.g., clipping the content and taking occlusion into account). Regardless of how the content is presented to the user, a model of the physical world may be used so that properties of the virtual objects that may be affected by the physical objects, including the shape, position, motion, and visibility of the virtual objects, may be correctly calculated.
[0090] A model of the physical world may be created from data collected from sensors on a user's wearable device. In some embodiments, the model may be created from data collected by multiple users, which may be aggregated in a computing device remote from all of the users (and which may be "in the cloud").
[0091] In some embodiments, at least one of the sensors may be configured to obtain information about physical objects in a scene, particularly non-fixed objects, at high frequency with low latency using compact and low power components, and the sensor may employ patch tracking to limit the amount of data output.
[0092] 4 depicts an image sensing system 400 according to some embodiments. The image sensing system 400 may include an image sensor 402, which may include an image array 404, which may contain a plurality of pixels, each responsive to light, as in a conventional image sensor. The sensor 402 may further include circuitry for accessing each pixel. Accessing a pixel may involve obtaining information about the incident light generated by that pixel. Alternatively, or in addition, accessing a pixel may involve controlling that pixel, such as by configuring it to only provide an output in response to the detection of an event.
[0093] In the illustrated embodiment, the image array 404 is configured as an array with multiple rows and columns of pixels. In such an embodiment, the access circuitry may be implemented as a row address encoder / decoder 406 and a column address encoder / decoder 408. The image sensor 402 may further contain circuitry that generates inputs to the access circuitry and controls the timing and order in which information is read from the pixels in the image array 404. In the illustrated embodiment, that circuitry is a patch tracking engine 410. In contrast to conventional image sensors that may continuously output image information captured by pixels in each row, the image sensor 402 may be controlled to output image information in defined patches. Furthermore, the location of those patches relative to the image array may change over time. In the illustrated embodiment, the patch tracking engine 410 may output image array access information and control the output of image information from portions of the image array 404 that correspond to the locations of the patches, and the access information may change dynamically based on estimates of the motion of objects in the environment and / or the motion of the image sensor relative to those objects.
[0094] In some embodiments, the image sensor 402 may have the functionality of a dynamic vision sensor (DVS) such that image information is provided by the sensor only when there is a change in an image property (e.g., intensity) for a pixel. For example, the image sensor 402 may apply one or more thresholds that define the on and off states of the pixel. The image sensor may detect that a pixel has changed state and selectively provide an output for only those pixels, or only those pixels in a patch, that have changed state. These outputs may be made asynchronously as they are detected, rather than as part of a read of all pixels in the array. The output may be in the form of an address-event representation (AER) 418, which may include, for example, a pixel address (e.g., row and column) and a type of event (on or off). An on event may indicate that a pixel cell at a particular pixel address senses an increase in light intensity, and an off event may indicate that a pixel cell at a particular pixel address senses a decrease in light intensity. The increase or decrease may be relative to an absolute level, or may be a change relative to a level in the last output from the pixel. The change may be expressed, for example, as a fixed offset or as a percentage of the value in the final output from the pixel.
[0095] The use of DVS techniques in conjunction with patch tracking may enable an image sensor suitable for use in an XR system. When combined in an image sensor, the amount of data generated may be limited to data from pixel cells that are in the patch and detect a change that will trigger the output of an event.
[0096] In some scenarios, high resolution image information is desirable. However, the use of a large sensor with cells of more than a million pixels to generate high resolution image information may generate a large amount of image information when DVS techniques are used. The inventors recognize and understand that DVS sensors may produce a large number of events that reflect changes in the image other than those resulting from movement in the background or the movement of the object being tracked. Currently, the resolution of DVS sensors is limited to below 1 MB, e.g., 128x128, 240x180, and 346x260, to limit the number of events generated. Such sensors sacrifice resolution for tracking objects and may not detect, for example, fine finger movements of a hand. Furthermore, if the image sensor outputs image information in other formats, limiting the resolution of the sensor array to output a manageable number of events may also limit the use of the image sensor to generate high resolution image frames with DVS functionality. Sensors as described herein may have a resolution higher than VGA, including up to 8 or 12 megapixels, in some embodiments. Nevertheless, patch tracking as described herein may be used to limit the number of events output by the image sensor per second. As a result, an image sensor may be enabled that operates in at least two modes. For example, an image sensor with megapixel resolution may operate in a first mode that outputs events within the specific patch being tracked. In a second mode, it may output a high resolution image frame or a portion of an image frame. Such an image sensor may be controlled within an XR system to operate in these different modes based on the capabilities of the system.
[0097] The image array 404 may include a number of pixel cells 500 arranged in an array. FIG. 5A depicts an example of a pixel cell 500, which in this embodiment is configured for use in an imaging array implementing the DVS technique. The pixel cell 500 may include a light sensing circuit 502, a difference circuit 506, and a comparator 508. The light sensing circuit 502 may include a photodiode 504 that converts light striking the photodiode into a measurable electrical signal. In this example, the conversion is to an electrical current I. A transconductance amplifier 510 converts the photocurrent I into a voltage. The conversion may be linear or nonlinear, such as according to a function of logI. Regardless of the specific transfer function, the output of the transconductance amplifier 510 indicates the amount of light detected at the photodiode 504. Although a photodiode is illustrated as an example, it should be understood that other light-sensing components that produce a measurable output in response to incident light may be implemented in the light sensing circuit in place of or in addition to the photodiode.
[0098] In the embodiment of FIG. 5A, circuitry is built into the pixel itself to determine if the pixel's output has changed sufficiently and trigger an output for that pixel cell. In this example, that function is implemented by a difference circuit 506 and a comparator 508. The difference circuit 506 may be configured to reduce DC mismatch between pixel cells, for example, by balancing the output of the difference circuit and resetting the level after the generation of an event. In this example, the difference circuit 506 is configured to produce an output indicative of the change in the output of the photodiode 504 since the last output. The difference circuit may include an amplifier 512 with gain -A, a capacitor 514, which may be implemented as a single circuit element or one or more capacitors connected in a network, and a reset switch 516.
[0099] In operation, the pixel cell will be reset by momentarily closing switch 516. Such a reset may occur at the beginning of operation of the circuit and any time thereafter an event is detected. When pixel 500 is reset, the voltage across capacitor 514 will be such that, when subtracted from the output of transconductance amplifier 510, zero voltage will be present at the input of amplifier 512. When switch 516 is open, the output of transconductance amplifier 510 will be combined with the voltage drop across capacitor 514 such that zero voltage is present at the input of amplifier 512. The output of transconductance amplifier 510 changes as a result of changes in the amount of light striking photodiode 504. As the output of transconductance amplifier 510 increases or decreases, the output of amplifier 512 will swing positive or negative by an amount amplified by the gain of amplifier 512.
[0100] The comparator 508 may determine whether an event is generated and the sign of the event, for example, by comparing the output voltage V of the difference circuit to a predetermined threshold voltage C. In some embodiments, the comparator 508 may include two comparators with transistors, one pair may operate when the output of the amplifier 512 indicates a positive change and may detect an increasing change (on event), and the other comparator may operate when the output of the amplifier 512 indicates a negative change and may detect a decreasing change (off event). However, it should be understood that the amplifier 512 may have a negative gain. In such an embodiment, an increase in the output of the transconductance amplifier 510 may be detected as a negative voltage change at the output of the amplifier 512. Similarly, it should be understood that the positive and negative voltages may be with respect to ground or any suitable reference level. Regardless, the value of the threshold voltage C may be controlled by the characteristics of the transistors (e.g., transistor size, transistor threshold voltage) and / or by the value of a reference voltage that may be applied to the comparator 508.
[0101] 5B depicts an example of an event output (on, off) over time t of pixel cell 500. In the illustrated example, at time t1, the output of the difference circuit has a value of V1, at time t2, the output of the difference circuit has a value of V2, and at time t3, the output of the difference circuit has a value of V3. Between time t1 and time t2, the photodiode senses some increase in light intensity, but the pixel cell does not output an event because the change in V does not exceed the value of the threshold voltage C. At time t2, the pixel cell outputs an on event because V2 is greater than V1 by the value of the threshold voltage C. Between time t2 and time t3, the photodiode senses some decrease in light intensity, but the pixel cell does not output an event because the change in V does not exceed the value of the threshold voltage C. At time t3, the pixel cell outputs an off event because V3 is less than V2 by the value of the threshold voltage C.
[0102] Each event may trigger an output in the AER 418. The output may include, for example, an indication of whether the event is an on or off event and an identification of the pixel, such as its row and column. Other information may alternatively or additionally be included with the output. For example, a timestamp may be included, which may be useful if the event is queued for later transmission or processing. As another example, the current level at the output of the amplifier 510 may be included. Such information may optionally be included, for example, if further processing is to be performed in addition to detecting object motion.
[0103] It should be understood that the frequency of the event output, and therefore the sensitivity of the pixel cell, may be controlled by the value of the threshold voltage C. For example, the frequency of the event output may be reduced by increasing the value of the threshold voltage C, or increased by decreasing the threshold voltage C. It should also be understood that the threshold voltage C may be different for on-events and off-events, for example, by setting different reference voltages for the comparators for detecting on-events and the comparators for detecting off-events. It should also be understood that the pixel cell may also output a value indicative of the size of the light intensity change instead of or in addition to a sign signal indicative of the detection of an event.
[0104] The pixel cell 500 of Figures 5A and 5B is illustrated as an example, according to some embodiments. Other designs may also be suitable for the pixel cell. In some embodiments, the pixel cell includes a light-sensing circuit and a difference circuit, but may share a comparator circuit with one or more other pixel cells. In some embodiments, the pixel cell may include circuitry, such as an active pixel sensor at the pixel level, configured to calculate a change value.
[0105] The ability to configure pixels to output only in response to detection of an event, regardless of the manner in which the event is detected for each pixel cell, may be used to limit the amount of information required to maintain a model of the location of a non-fixed (i.e., movable) object. For example, pixels within a patch may be set such that a threshold voltage C is triggered when a relatively small change occurs. Other pixels outside the patch may have a larger threshold, such as three or five times larger. In some embodiments, the threshold voltage C for any out-of-patch pixel may be set large such that the pixel is effectively disabled and does not produce any output, regardless of the amount of change. In other embodiments, out-of-patch pixels may be disabled in other ways. In such embodiments, the threshold voltage may be fixed for all pixels, but pixels may be selectively enabled or disabled based on whether they are within a patch.
[0106] In yet other embodiments, the threshold voltage for one or more pixels may be adaptively set to modulate the amount of data output from the image array. For example, the AR system may have processing capacity to process a certain number of events per second. The threshold for some or all pixels may be increased when the number of events per second being output exceeds an upper limit. Alternatively, or in addition, the threshold may be lowered when the number of events per second drops below a lower limit to enable more data for more accurate processing. The number of events per second may be 200 to 2,000 events, as a specific example. Such a number of events constitutes a substantial reduction in the number of data to be processed per second, as compared to processing all of the pixel values scanned out of the image sensor, which would constitute, for example, 30 million or more pixel values per second. The number of events is a further reduction, as compared to processing only the pixels in a patch, which, although less, may still amount to tens of thousands of pixel values or more per second.
[0107] The control signals for enabling and / or setting the threshold voltages for each of the pixels may be generated in any suitable manner, however, in the illustrated embodiment, the control signals are set by the patch tracking engine 410 or based on processing within the processing module 72 or other processor.
[0108] Referring back to FIG. 4 , the image sensing system 400 may receive inputs from any suitable components such that the patch tracking engine 410 may dynamically select at least one region of the image array 404 to be enabled and / or disabled based on at least the received inputs to implement a patch. The patch tracking engine 410 may be digital processing circuitry having a memory that stores one or more parameters of the patch. The parameters may be, for example, the boundaries of the patch and may include other information such as information about a scale factor between the movement of the image array and the movement within the image array of the image of the movable object associated with the patch. The patch tracking engine 410 may also include circuitry configured to perform calculations on the stored values and other measured values provided as inputs.
[0109] In the illustrated embodiment, the patch tracking engine 410 receives as input a designation of a current patch. The patch may be designated based on its size and location in the image array 404, such as by defining the range of row and column addresses of the patch. Such a specification may be provided as an output of the processing module 72 (FIG. 2) or other components that process information about the physical world. The processing module 72 may designate a patch to encompass, for example, the current location of each movable object in the physical world or a subset of the movable objects being tracked, in order to render the virtual objects with an appropriate appearance relative to the physical world. For example, if the AR scene is to include as a virtual object a toy doll balanced on a physical object, such as a moving toy car, a patch may be designated to encompass the toy car. A patch may not be designated for another toy car moving in the background, since there may be little need to have up-to-date information about that object to render a realistic AR scene.
[0110] Regardless of how a patch is selected, information about the current location of the patch may be provided to the patch tracking engine 410. In some embodiments, the patch may be rectangular, such that the location of the patch may simply be defined as a start and end row and column. In other embodiments, the patch may have other shapes, such as circular, and the patch may be defined in other ways, such as by a center point and a radius.
[0111] In some embodiments, trajectory information may also be provided for the patch. The trajectory may, for example, define the movement of the patch relative to the coordinates of the image array 404. The processing module 72 may, for example, build a model of the movement of a movable object in the physical world and / or the movement of the image array 404 relative to the physical world. Since the movement of either or both may affect the location in the image array 404 where the image of the object is projected, the trajectory of the patch in the image array 404 may be calculated based on either or both. The trajectory may be defined in any suitable manner, such as parameters of a linear, quadratic, cubic, or other polynomial equation.
[0112] In other embodiments, the patch tracking engine 410 may dynamically calculate the location of the patch based on input from a sensor that provides information about the physical world. The information from the sensor may be provided directly from the sensor. Alternatively, or in addition, the sensor information may be processed to extract information about the physical world before being provided to the patch tracking engine 410. The extracted information may include, for example, the motion of the image array 404 relative to the physical world, the distance between the image array 404 and the object whose image falls within the patch, or other information that may be used to dynamically align the patch in the image array 404 with the image of the object in the physical world as the image array 404 and / or the object move.
[0113] Examples of input components may include an image sensor 412 and an inertial sensor 414. Examples of image sensors 412 may include an eye tracking camera 50, a depth sensor 51, a world camera 52, and / or a camera 52. Examples of inertial sensors 414 may include an inertial measurement unit 57. In some embodiments, the input components may be selected to provide data at a relatively high rate. The inertial measurement unit 57 may have an output rate of 200-2,000 measurements / second, such as, for example, 800-1,200 measurements / second. Patch locations may be updated at a similarly high rate. By using the inertial measurement unit 57 as a source of input to the patch tracking engine 410, the location of the patch may be updated 800-1,200 times / second, as one specific example. In this manner, movable objects may be tracked with high accuracy using relatively small patches, limiting the number of events that need to be processed. Such an approach can lead to very low latency between changes in the relative position of the image sensor and the movable object, with similarly low latency in updating the rendering of the virtual object, providing a desirable user experience.
[0114] In some scenarios, the movable object being tracked with the patch may be a stationary object in the physical world. The AR system may, for example, identify stationary objects from analyzing multiple images taken from the physical world and select features of one or more of the stationary objects as reference points for determining the movement of a wearable device having an image sensor thereon. Frequent and low-latency updates of the location of these reference points relative to the sensor array may be used to provide frequent and low-latency calculations of the head pose of the user of the wearable device. Frequent and low-latency updates of the head pose improve the user experience of the AR system, since the head pose can be used to realistically render virtual objects through a user interface on the wearable. Thus, having an input to the patch tracking engine 410, which controls the position of the patch, may come only from sensors with high output rates, such as one or more inertial measurement units, leading to a desirable user experience of the AR system.
[0115] However, in some embodiments, other information may also be provided to the patch tracking engine 410 to enable it to calculate and / or apply a trajectory to the patch. This other information may include stored information 416, such as the passable world module 38 and / or the mapped mesh model 46. This information may indicate one or more previous positions of the object relative to the physical world, such that a consideration of changes in these previous positions and / or changes in the current position relative to the previous positions may indicate the trajectory of the object in the physical world, which may then be mapped to the trajectory of the patch across the image array 404. Other information in the model of the physical world may be used as an alternative or in addition. For example, other information regarding the size of a movable object and / or its distance or position relative to the image array 404 may be used to calculate either the location or trajectory of a patch across the image array 404 associated with that object.
[0116] Regardless of the manner in which the trajectory is determined, the patch tracking engine 410 may apply the trajectory to calculate updated locations of patches within the image array 404 at a high rate, such as 1 time / second or faster than 800 times / second. The rate may be limited by processing power, such as less than 2,000 times / second in some embodiments.
[0117] It should be understood that the process for tracking changes in movable objects may be a less than complete reconstruction of the physical world. However, reconstructions of the physical world may occur at intervals longer than the interval between updates in the positions of the movable objects, such as every 30 seconds or every 5 seconds. The locations of the objects to be tracked and the locations of patches that would capture information about those objects may be recalculated when reconstructions of the physical world occur.
[0118] 4 illustrates an embodiment in which the processing circuitry for both dynamically generating patches and controlling the selective output of image information from within the patches is configured to directly control the image array 404 such that the image information output from the array is limited to the selected information. Such circuitry may be integrated, for example, within the same semiconductor chip that stores the image array 404, or integrated into a separate controller chip for the image array 404. However, it should be understood that the circuitry generating the control signals for the image array 404 may be distributed throughout the XR system. For example, some or all of the functions may be performed by programming within the processing module 72 or other processors in the system.
[0119] The image sensing system 400 may output image information for each of a plurality of pixels. Each pixel of the image information may correspond to one of the pixel cells of the image array 404. The image information output from the image sensing system 400 may be image information for one or more patches corresponding to at least one region of the image array 404 selected by the patch tracking engine 410. In some embodiments, such as when each pixel of the image array 404 has a different configuration than that illustrated in FIG. 5A, the pixels in the output image information may identify pixels where a change in light intensity is detected by the image sensor 400 in one or more patches.
[0120] In some embodiments, the image information output from the image sensing system 400 may be image information for pixels outside each of one or more patches corresponding to at least one region of the image array selected by the patch tracking engine 410. For example, a deer may be running in a physical world with a flowing river. Details of the waves in the river may not be of interest, but may trigger pixel cells of the image array 402. The patch tracking engine 410 may create a patch that encompasses the river and disable the portion of the image array 402 that corresponds to the patch that encompasses the river.
[0121] Based on the identification of the changed pixels, further processing may be performed. For example, the portion of the world model that corresponds to the portion of the physical world imaged by the changed pixels may be updated. These updates may be performed based on information collected using other sensors. In some embodiments, further processing may be adjusted on or triggered by multiple changed pixels in the patch. For example, an update may be performed once 10% or some other threshold amount of pixels in the patch detect a change.
[0122] In some embodiments, image information in other formats may be output from the image sensor and may be used in combination with the change information to provide updates to the world model. In some embodiments, the format of the image information output from the image sensor may change from time to time during operation of the VR system. In some embodiments, for example, pixel cell 500 may be operated at some times to produce a difference output as produced in comparator 508. The output of amplifier 510 may be switchable at other times to output a magnitude of light incident on photodiode 504. For example, the output of amplifier 510 may be switchably connected to a sense line, which in turn is connected to an A / D converter, which may provide a digital indication of the magnitude of the incident light based on the magnitude of the output of amplifier 510.
[0123] The image sensor, in this configuration, is operated as part of the AR system and may output differentially in most cases, outputting only events for pixels where a change above a threshold is detected, or only events for pixels in a patch where a change above a threshold is detected. Periodically, such as every 5-30 seconds, a full image frame may be output, with information magnitude for all pixels in the image array. Low latency and accurate processing may be achieved in this way, and the differential information may be used to quickly update selected parts of the world model where changes have occurred that are most likely to affect the user perception, while the full image may be used to more frequently update larger parts of the world model. Full updates to the world model occur only at a slower rate, but any delay in updating the model may not meaningfully affect the user's perception of the AR scene.
[0124] The output mode of the image sensor may be changed at any time throughout operation of the image sensor such that the sensor outputs one or more of intensity information for some or all of the pixels and an indication of change for some or all of the pixels in the array.
[0125] It is not a requirement that image information from the patch be selectively output from the image sensor by limiting the information output from the image array. In some embodiments, image information may be output by all pixels in the image array, or only information about specific regions of the array may be output from the image sensor. FIG. 6 depicts an image sensor 600, according to some embodiments. The image sensor 600 may include an image array 602. In this embodiment, the image array 602 may resemble a conventional image array that scans out rows and columns of pixel values. The operation of such an image array may be accommodated by other components. The image sensor 600 may further include a patch tracking engine 604 and / or a comparator 606. The image sensor 600 may provide an output 610 to an image processor 608. The processor 608 may be, for example, part of the processing module 72 (FIG. 2).
[0126] The patch tracking engine 604 may have a structure and functionality similar to the patch tracking engine 410. It may be configured to receive a signal defining at least one selected region of the image array 602 and then generate a control signal defining a dynamic location of the region within the image array 602 of an image of an object represented by the region based on a calculated trajectory. In some embodiments, the patch tracking engine 604 may receive a signal defining at least one selected region of the image array 602, which may include trajectory information for the region or regions. The patch tracking engine 604 may be configured to perform a calculation that dynamically identifies pixel cells within the at least one selected region based on the trajectory information. Variations in the implementation of the patch tracking engine 604 are also possible. For example, the patch tracking engine may update the location of the patch based on a sensor indicating the movement of the image array 602 and / or the projected movement of an object associated with the patch.
[0127] In the embodiment illustrated in FIG. 6, the image sensor 600 is configured to output difference information for pixels within an identified patch. The comparator 606 may be configured to receive a control signal from the patch tracking engine 604 that identifies a pixel within the patch. The comparator 606 may selectively act on pixels being output from the image array 602 that have an address within the patch as indicated by the patch tracking engine 604. The comparator 606 may act on the pixel cells to generate a signal indicative of a change in the sensed light detected by at least one region of the image array 602. As one example of implementation, the comparator 606 may contain a memory element that stores reset values for pixel cells within the array. As the current values of those pixels are scanned out of the image array 602, circuitry within the comparator 606 may compare the stored values to the current values and output an indication if the difference exceeds a threshold. Digital circuitry may be used, for example, to store values and perform such comparisons. In this embodiment, the output of image sensor 600 may be processed like the output of image sensor 400 .
[0128] In some embodiments, the image array 602, the patch tracking engine 604, and the comparator 606 may be implemented in a single integrated circuit, such as a CMOS integrated circuit. In some embodiments, the image array 602 may be implemented in a single integrated circuit. The patch tracking engine 604 and the comparator 606 may be implemented in a second single integrated circuit, for example, configured as a driver for the image array 602. Alternatively or in addition, some or all of the functionality of the patch tracking engine and / or the comparator 606 may be distributed to other digital processors in the AR system.
[0129] Other configurations or processing circuitry are also possible. Figure 7 depicts an image sensor 700, according to some embodiments. The image sensor 700 may include an image array 702. In this embodiment, the image array 702 may have pixel cells with a differential configuration as shown for the pixel 500 in Figure 5A. However, embodiments herein are not limited to differential pixel cells, and patch tracking may be implemented using an image sensor that outputs intensity information.
[0130] In the illustrated embodiment of Figure 7, the patch tracking engine 704 produces a control signal indicating the addresses of pixel cells in one or more patches being tracked. The patch tracking engine 704 may be constructed and operated similarly to the patch tracking engine 604. Here, the patch tracking engine 704 provides the control signal to a pixel filter 706, which passes image information from only those pixels in the patch to an output 710. As shown, the output 710 is coupled to an image processor 708, which may further process the image information for the pixels in the patch using techniques as described herein or in other suitable manners.
[0131] A further variation is illustrated in FIG. 8, which depicts an image sensor 800 according to some embodiments. The image sensor 800 may include an image array 802, which may be a conventional image array that scans out intensity values for pixels. The image array may be adapted to provide difference image information as described herein through the use of a comparator 806. The comparator 806, similar to the comparator 606, may calculate difference information based on stored values for the pixels. Selected ones of those difference values may be passed by a pixel filter 808 to an output 812. Similar to the pixel filter 706, the pixel filter 808 may receive a control input from a patch tracking engine 804. The patch tracking engine 804 may be similar to the patch tracking engine 704. The output 812 may be coupled to an image processor 810. Some or all of the above-described components of the image sensor 800 may be implemented within a single integrated circuit. Alternatively, the components may be distributed across one or more integrated circuits or other components.
[0132] Image sensors as described herein may be operated as part of an augmented reality system to maintain information about movable objects or other information about the physical world that is useful in realistically rendering images of virtual objects in combination with information about the physical environment. Figure 9 depicts a method 900 for image sensing, according to some embodiments.
[0133] At least a portion of method 900 may be implemented to operate an image sensor, including, for example, image sensor 400, 600, 700, or 800. Method 900 may begin with receiving image information from one or more inputs (act 902), including, for example, image sensor 412, inertial sensor 414, and stored information 416. Method 900 may include identifying, at least in part, one or more patches on an image output of an image sensing system based on the received information (act 904). An example of act 904 is illustrated in FIG. 10. In some embodiments, method 900 may include calculating a moving trajectory for the one or more patches (act 906). An example of act 906 is illustrated in FIG.
[0134] Method 900 may also include configuring the image sensing system (act 908) based at least in part on the identified one or more patches and / or their estimated moving trajectories. The configuring may be accomplished by enabling a portion of pixel cells of the image sensing system based at least in part on the identified one or more patches and / or their estimated moving trajectories, for example, through comparator 606, pixel filter 706, etc. In some embodiments, comparator 606 may receive a first reference voltage value for pixel cells corresponding to selected patches on the image and a second reference voltage value for pixel cells that do not correspond to any selected patches on the image. Comparator 606 may set the second reference voltage to be much higher than the first reference voltage such that a reasonable light intensity change sensed by a pixel cell having a comparator cell with the second reference voltage may not result in an output by the pixel cell. In some embodiments, pixel filter 706 may disable output from pixel cells with addresses (e.g., rows and columns) that do not correspond to any selected patches on the image.
[0135] 10 depicts patch identification 904, according to some embodiments. Patch identification 904 may include segmenting one or more images from one or more inputs based, at least in part, on color, light intensity, angle of arrival, depth, and semantics (act 1002).
[0136] Patch identification 904 may also include a step of recognizing one or more objects in one or more images (act 1004). In some embodiments, object recognition 1004 may be based at least in part on predefined features of the object, including, for example, hands, eyes, facial features. In some embodiments, object recognition 1004 may be based on one or more virtual objects. For example, a virtual animal character is walking on a physical pencil. Object recognition 1004 may target the virtual animal character as an object. In some embodiments, object recognition 1004 may be based at least in part on artificial intelligence (AI) training received by the image sensing system. For example, the image sensing system may be trained by reading images of cats in different types and colors, and thus the learned characteristics of cats, enabling it to identify cats within the physical world.
[0137] Patch identification 904 may include generating a patch based on one or more objects (act 1006). In some embodiments, object patching 1006 may generate the patch by calculating a convex hull or bounding box for the one or more objects.
[0138] 11 depicts patch trajectory estimation 906, according to some embodiments. Patch trajectory estimation 906 may include predicting movement over time for one or more patches (act 1102). Movement for one or more patches may occur for a number of reasons, including, for example, a moving object and / or a moving user. Motion prediction 1102 may include deriving a movement speed for the moving object and / or the moving user based on received images and / or received AI training.
[0139] Patch trajectory estimation 906 may include calculating a trajectory over time for one or more patches based, at least in part, on the predicted movement (act 1104). In some embodiments, the trajectory may be calculated by modeling a first-order linear equation, assuming that an object under motion will continue to move at the same speed and in the same direction. In some embodiments, the trajectory may be calculated by curve fitting or using heuristics, including pattern detection.
[0140] 12 and 13 illustrate coefficients that may be applied in the calculation of the patch trajectory. FIG. 12 depicts an example of a movable object, which in this example is a moving object 1202 (e.g., a hand) that moves relative to a user of the AR system. In this example, the user wears an image sensor as part of a head mounted display 62. In this example, the user's eye 49 is looking straight ahead such that the image array 1200 captures the field of view (FOV) for the eye 49 relative to one viewpoint 1204. The object 1202 is within the FOV and therefore appears by creating intensity variations in the corresponding pixels in the array 1200.
[0141] The array 1200 has a number of pixels 1208 arranged in the array. For a system tracking a hand 1202, a patch 1206 in the array that contains the object 1202 at time t0 may include a portion of the pixels. If the object 1202 is moving, the location of the patch capturing the object will change over time. That change can be captured in the patch trajectory from patch 1206 to patches X and Y used at a later time.
[0142] A patch trajectory may be estimated, such as in act 906, by identifying features 1210 associated with an object within the patch, e.g., a fingertip in the illustrated example. A motion vector 1212 may be calculated for the features. In this example, the trajectory is modeled as a first-order linear equation, and the prediction is based on the assumption that the object 1202 continues on its same patch trajectory 1214 over time, leading to patch locations X and Y at each of two successive times.
[0143] As the patch location changes, the image of the moving object 1202 remains within the patch. Although the image information is limited to that gathered with the pixels within the patch, the image information is sufficient to represent the motion of the moving object 1202. This would be true whether the image information is intensity information or differential information as produced by a differential circuit. In the case of a differential circuit, for example, an event indicating an increase in intensity may occur as the image of the moving object 1202 moves across the pixels. Conversely, an event indicating a decrease in intensity may occur as the image of the moving object 1202 passes from a pixel. The pattern of pixels with the increase and decrease events may be used as a reliable indication of the motion of the moving object 1202, which may be updated quickly with low latency due to the relatively small amount of data indicating the events. As a specific example, such a system may lead to a realistic XR system that tracks a user's hands and alters the rendering of the virtual object, creating the sensation for the user that the user is interacting with the virtual object.
[0144] The location of the patch may change for other reasons as well, any or all of which may be reflected in the trajectory calculation. One such other change is the movement of the user while the user is wearing the image sensor. FIG. 13 depicts an example of a moving user creating a changing viewpoint for the user and the image sensor. In FIG. 13, the user may initially be looking straight at an object with viewpoint 1302. In this configuration, the pixel array 1300 of the image array will capture an object in front of the user. The object in front of the user may be within patch 1312.
[0145] The user may then change the viewpoint, such as by turning their head. The viewpoint may change to viewpoint 1304. An object that was previously directly in front of the user will have a different location in the user's field of view at viewpoint 1304, even if it has not moved. It will also be at a different point in the field of view of the image sensor worn by the user, and therefore at a different location in image array 1300. The object may be contained within a patch, for example, at location 1314.
[0146] If the user further changes their viewpoint to viewpoint 1306 and the image sensor moves with the user, the location of the object that was previously directly in front of the user will be imaged at a different point in the field of view of the image sensor worn by the user, and therefore at a different location in image array 1300. The object may be contained within a patch at location 1316, for example.
[0147] As can be seen, as the user further changes his / her viewpoint, the location of the patch in the image array needed to capture the object moves further. The trajectory of this movement from location 1312 to location 1314 to location 1316 may be estimated and used to track the future position of the patch.
[0148] The trajectory may be estimated in other ways. For example, measurements with inertial sensors may indicate the acceleration and velocity of the user's head when the user has a gaze point 1302. This information may be used to predict the trajectory of the patches in the image array based on the movement of the user's head.
[0149] Based at least in part on these inertial measurements, patch trajectory estimate 906 may predict that the user will have viewpoint 1304 at time t1 and viewpoint 1306 at time t2. Thus, patch trajectory estimate 906 may predict that patch 1308 may move to patch 1310 at time t1 and to patch 1312 at time t2.
[0150] As an example of such an approach, it may be used to provide accurate and low latency estimation of head pose in an AR system. The patch may be positioned to encompass an image of a stationary object in the user's environment. As a specific example, processing of the image information may identify a corner of a picture frame hanging on a wall as a recognizable and stationary object to track. The processing may center the patch on the object. As in the case of the moving object 1202 described above in connection with FIG. 12, the relative movement between the object and the user's head will produce an event that can be used to calculate the relative motion between the user and the tracked object. In this example, since the tracked object is stationary, the relative motion is indicative of the motion of the imaging array worn by the user. That motion therefore indicates a change in the user's head pose relative to the physical world and can be used to maintain an accurate calculation of the user's head pose, which can be used in realistically rendering virtual objects. Because imaging arrays as described herein can provide fast updates using a relatively small amount of data per update, calculations for rendering virtual objects remain accurate (they can be performed quickly and updated frequently).
[0151] Referring back to FIG. 11, patch trajectory estimation 906 may include adjusting the size of at least one of the patches based at least in part on the calculated patch trajectory (act 1106). For example, the size of the patch may be set large enough to include pixels onto which an image of the movable object or at least a portion of the object for which image information is to be generated will be projected. The patch may be set slightly larger than the projected size of the image of the portion of the object of interest so that if there is any error in estimating the trajectory of the patch, the patch may still include the relevant portion of the image. As the object moves relative to the image sensor, the size of the image of that object in pixels may change based on distance, angle of incidence, object orientation, or other factors. A processor defining the patch associated with the object may set the size of the patch, such as by measuring the size of the patch associated with the object based on other sensor data or by calculating the size of the patch based on a world model. Other parameters of the patch, such as its shape, may be set or updated as well.
[0152] 14 depicts an image sensing system 1400 configured for use in an XR system, according to some embodiments. Like image sensing system 400 (FIG. 4), image sensing system 1400 includes circuitry for selectively outputting values within a patch, and may also be configured to output events for pixels within the patch, as described above. In addition, image sensing system 1400 is configured to selectively output measured intensity values, which may be output for a complete image frame.
[0153] In the illustrated embodiment, separate outputs are shown for events and intensity values generated using the DVS techniques described above. The output generated using the DVS techniques may be output as AER 1418, using the representation as described above in connection with AER 418. The output representing the intensity values may be output through an output, here designated as APS 1420. These intensity outputs may be for a patch or for the entire image frame. The AER and APS outputs may be active simultaneously. However, in the illustrated embodiment, the image sensor 1400 operates at any given time in a mode for outputting events or in a mode in which intensity information is output. The system in which such an image sensor is used may selectively use the event output and / or the intensity information.
[0154] The image sensing system 1400 may include an image sensor 1402, which may include an image array 1404, which may contain a plurality of pixels 1500, each responsive to light. The sensor 1402 may further include circuitry for accessing the pixel cells. The sensor 1402 may further include circuitry that generates inputs to the access circuitry to control the mode in which information is read from the pixel cells in the image array 1404.
[0155] In the illustrated embodiment, the image array 1404 is configured as an array with multiple rows and columns of pixel cells that are both accessible in read mode. In such an embodiment, the access circuitry may include a row address encoder / decoder 1406, a column address encoder / decoder 1408 that controls a column select switch 1422, and / or a register 1424 that may temporarily hold information about incident light sensed by one or more corresponding pixel cells. The patch tracking engine 1410 may generate inputs to the access circuitry to control the pixel cells providing image information at any time.
[0156] In some embodiments, the image sensor 1402 may be configured to operate in a rolling shutter mode, a global shutter mode, or both. For example, the patch tracking engine 1410 may generate inputs to the access circuitry and control the read mode of the image array 1402.
[0157] When the sensor 1402 operates in a rolling shutter readout mode, a single column of pixel cells is selected during each system clock, for example, by closing a single column switch 1422 of the multiple column switches. During that system clock, the selected column of pixel cells is exposed and read out to the APS 1420. To generate an image frame by the rolling shutter mode, the columns of pixel cells in the sensor 1402 are read out, column by column, and then processed by an image processor to generate the image frame.
[0158] When the sensor 1402 operates in a global shutter mode, columns of pixel cells are exposed simultaneously, e.g., within a single system clock, such that information captured by pixel cells in multiple columns can be read out simultaneously to the APS 1420b, storing the information in registers 1424. Such a readout mode allows for direct output of an image frame without the need for further data processing. In the illustrated embodiment, information about the incident light sensed by the pixel cells is stored in individual registers 1424. It should be understood that multiple pixel cells may share one register 1424.
[0159] In some embodiments, the sensor 1402 may be implemented in a single integrated circuit, such as a CMOS integrated circuit. In some embodiments, the image array 1404 may be implemented in a single integrated circuit. The patch tracking engine 1410, the row address encoder / decoder 1406, the column address encoder / decoder 1408, the column select switch 1422, and / or the register 1424 may be implemented in a second single integrated circuit, for example, configured as a driver for the image array 1404. Alternatively or in addition, some or all of the functionality of the patch tracking engine 1410, the row address encoder / decoder 1406, the column address encoder / decoder 1408, the column select switch 1422, and / or the register 1424 may be distributed to other digital processors in the AR system.
[0160] 15 illustrates an example pixel cell 1500. In the illustrated embodiment, each pixel cell may be configured to output either event or intensity information. However, it should be understood that in some embodiments, the image sensor may be configured to output both types of information in parallel.
[0161] Both the event information and the intensity information are based on the output of the photodetector 504, as described above in connection with FIG. 5A. The pixel cell 1500 includes circuitry for generating the event information. The circuitry includes a light sensing circuit 502, a difference circuit 506, and a comparator 508, also as described above. A switch 1520, when in a first state, connects the photodetector 504 to the event generating circuitry. The switch 1520 or other control circuitry may be controlled by a processor, which controls the AR system such that a relatively small amount of image information is provided for a substantial period of time when the AR system is in operation.
[0162] The switches 1520 or other control circuitry may also be controlled to configure the pixel cells 1500 to output intensity information. In the illustrated embodiment, the intensity information is provided as a complete image frame, represented continuously as a stream of pixel intensity values for each pixel in the image array. To operate in this mode, the switches 1520 in each pixel cell may be set to a second position, exposing the output of the photodetector 504 so that it may be connected to an output line after passing through the amplifier 510.
[0163] In the illustrated embodiment, the output lines are illustrated as column lines 1510. There may be one such column line for each column in the image array. Each pixel cell in a column may be coupled to a column line 1510, but the pixel array may be controlled such that one pixel cell at a time is coupled to the column line 1510. There is one such switch in each pixel cell, switch 1530, which controls when the pixel cell 1500 is connected to its respective column line 1510. Access circuitry, such as row address decoder 410, may close switch 1530, ensuring that only one pixel cell at a time is connected to each column line. Switches 1520 and 1530 may be implemented using one or more transistors or similar components that are part of the image array.
[0164] 15 illustrates additional components that may be included within each pixel cell, according to some embodiments. A sample and hold circuit (S / H) 1532 may be connected between the photodetector 504 and the column line 1510. When present, the S / H 1532 may enable the image sensor 1402 to operate in a global shutter mode. In global shutter mode, a trigger signal is sent to each pixel cell in the array in parallel. Within each pixel cell, the S / H 1532 captures a value that indicates the intensity at the time of the trigger signal. The S / H 1532 stores that value and generates an output based on that value until the next value is captured.
[0165] As shown in FIG. 15, a signal representing a value stored by S / H 1532 may be coupled to column line 1510 when switch 1530 is closed. The signal coupled to the column line may be processed to produce an output of the image array. The signal may be buffered and / or amplified, for example, in amplifier 1512 upon termination of column line 1510, and then applied to an analog-to-digital converter (A / D) 1514. The output of A / D 1514 may be passed through other read circuitry 1516 to output 1420. Read circuitry 1516 may include, for example, column switch 1422. Other components in read circuitry 1516 may perform other functions, such as serializing the multi-bit output of A / D 1514.
[0166] Those skilled in the art will understand how to implement circuits to perform the functions described herein. S / H 1532 may be implemented, for example, as one or more capacitors and one or more switches. However, it should be understood that S / H 1532 may be implemented using other components or in circuit configurations other than those illustrated in FIG. 15A. It should be understood that other components other than those illustrated may also be implemented. For example, FIG. 15 shows one amplifier and one A / D converter per column. In other embodiments, there may be one A / D converter shared across multiple columns.
[0167] In a pixel array configured for a global shutter, each S / H 1532 may store intensity values reflecting image information at the same moment in time. These values may be stored as the values stored in each pixel are successively read during a readout phase. Successive readout may be accomplished, for example, by connecting the S / H 1532 of each pixel cell in a row to its respective column line. The values on the column lines may then be passed to the APS output 1420, one at a time. Such information flow may be controlled by sequencing the opening and closing of column switches 1422, whose operation may be controlled, for example, by the column address decoder 1408. Once the values for each pixel in a row have been read, the pixel cells in the next row may be connected to column lines in their place. Their values may be read, one column at a time. The process of reading out the values for one row at a time may be repeated until the intensity values for all pixels in the image array have been read. In embodiments where intensity values are read for one or more patches, the process may be complete once the values for the pixel cells within the patch have been read.
[0168] The pixel cells may be read in any suitable order. The rows may be interleaved, for example, so that every third row is read in sequence. The AR system may still process the image data as a frame of image data by deinterleaving the data.
[0169] In embodiments in which S / H 1532 is not present, values may still be read from each pixel cell sequentially as rows and columns of values are scanned out. However, the value read from each pixel cell may represent the intensity of light detected at the cell's photodetector at the time the value in that cell was captured as part of the readout process, such as when that value is applied to A / D 1514. As a result, with a rolling shutter, the pixels of an image frame may represent images incident on the image array at slightly different times. For an image sensor that outputs complete frames at a rate of 30 Hz, the difference in time between when the first pixel value for a frame is captured and when the last pixel value for a frame is captured may vary by as much as 1 / 30 of a second, which is imperceptible for many applications.
[0170] For some XR functions, such as tracking objects, the XR system may perform calculations on image information collected using an image sensor using a rolling shutter. Such calculations may interpolate between successive image frames and, for each pixel, calculate an interpolated value that represents the estimated value of the pixel at a point in time between successive frames. The same time may be used for all pixels such that, via the calculation, the interpolated image frame contains pixels that represent the same point in time as may be produced using an image sensor with a global shutter. Alternatively, a global shutter image array may be used for one or more image sensors in a wearable device that forms part of the XR system. A global shutter for full or partial image frames may avoid the interpolation of other processes that may be performed to compensate for variations in capture time in image information captured using a rolling shutter. Interpolation calculations may therefore be avoided even when the image information is used to track a hand or other movable object, or to determine the head pose of a user of a wearable device in an AR system, or even to track the movement of an object, such as may occur using a camera on a wearable device that may be moving when the image information is collected, for processing to construct an accurate representation of the physical environment.
[0171] Differentiated Pixel Cells
[0172] In some embodiments, each pixel cell in the sensor array may be identical. Each pixel cell may be responsive to a broad spectrum of visible light, for example. Each photodetector may thus provide image information indicative of the intensity of visible light. In this scenario, the output of the image array may be a "grayscale" output indicative of the amount of visible light incident on the image array.
[0173] In other embodiments, pixel cells may be differentiated. For example, different pixel cells in a sensor array may output image information indicative of the intensity of light in a particular portion of the spectrum. A suitable technique for differentiating pixel cells is to position a filter element in the light path leading to the photodetector in the pixel cell. The filter element may be, for example, bandpass, allowing visible light of a particular color to pass. Applying such a color filter over a pixel cell configures that pixel cell to provide image information indicative of the intensity of light of the color corresponding to the filter.
[0174] Filters may be applied across pixel cells regardless of the structure of the pixel cells. They may be applied across pixel cells in a sensor array, for example with a global shutter or a rolling shutter. Similarly, filters may be applied to pixel cells that are configured to output an intensity or a change in intensity using DVS techniques.
[0175] In some embodiments, a filter element that selectively passes light of a primary color may be mounted across the photodetectors in each pixel cell in the sensor array. For example, a filter that selectively passes red, green, or blue light may be used. The sensor array may have multiple subarrays, each subarray having one or more pixels configured to sense light of a respective primary color. In this manner, the pixel cells in each subarray provide both intensity and color information about the object being imaged by the image sensor.
[0176] The inventors have recognized and appreciated that in an XR system, some functions require color information, while some functions can be implemented with grayscale information. A wearable device equipped with an image sensor and providing image information for the operation of an XR system may have multiple cameras, some of which may be formed with an image sensor that may provide color information. Others of the cameras may be grayscale cameras. The inventors have recognized and appreciated that a grayscale camera may consume less power, be more sensitive under low light conditions, output data faster, and / or output less data to represent the same range of the physical world with the same resolution as a camera formed with a comparable image sensor configured to sense color. However, a grayscale camera may output sufficient image information for many functions implemented within the XR system. Thus, an XR system may be configured with both grayscale and color cameras, using primarily grayscale cameras or multiple cameras, selectively using color cameras.
[0177] For example, an XR system may collect and process image information to create a passable world model. That processing may use color information, which may improve the effectiveness of certain functions, such as distinguishing between objects, identifying surfaces associated with the same object, and / or recognizing objects. Such processing may be performed or updated at any time, for example, when a user first turns on the system, moves into a new environment by walking into another room, etc., or when a change in the user's environment is otherwise detected.
[0178] Other functions are not significantly improved through the use of color information. For example, once a passable world model has been created, the XR system may use images from one or more cameras to determine the orientation of the wearable device relative to features within the passable world model. Such functions may be performed, for example, as part of head pose tracking. Some or all of the cameras used for such functions may be grayscale. Because head pose tracking is performed frequently as the XR system operates, in some embodiments, persistent use of one or more grayscale cameras for this function may provide significant power savings, reduced computation, or other advantages.
[0179] Similarly, at times during operation of the XR system, the system may use stereoscopic information from two or more cameras to determine distances to movable objects. Such a function may require high-rate processing of image information as part of tracking a user's hand or other movable object. Using one or more grayscale cameras for this function may provide lower latency or other advantages associated with processing high-resolution image information.
[0180] In some embodiments of the XR system, the XR system may have both a color and at least one grayscale camera and may selectively enable the grayscale and / or color cameras based on the function for which the image information from those cameras is to be used.
[0181] Pixel cells in an image sensor may be distinguished in ways other than based on the spectrum of light to which they are sensitive. In some embodiments, some or all of the pixel cells may produce an output having an intensity indicative of the angle of arrival of light incident on the pixel cell. The angle of arrival information may be processed to calculate the distance to the object being imaged.
[0182] In such an embodiment, the image sensor may obtain depth information passively. Passive depth information may be obtained by placing a component in the light path to a pixel cell in the array such that the pixel cell outputs information indicative of the angle of arrival of light striking that pixel cell. An example of such a component is a transmissive diffractive mask (TDM) filter.
[0183] The angle of arrival information may be converted, through calculations, into distance information indicating the distance to the object from which the light is being reflected. In some embodiments, pixel cells configured to provide angle of arrival information may be interspersed with pixel cells that capture light intensities of one or more colors. As a result, the angle of arrival information, and therefore the distance information, may be combined with other image information about the object.
[0184] In some embodiments, one or more of the sensors may be configured to obtain information about physical objects in a scene at high frequency with low latency using compact and low power components. The image sensor may, for example, draw less than 50 mW, allowing the device to be powered by a battery small enough to be used as part of a wearable system. The sensor may be an image sensor configured to passively obtain depth information instead of or in addition to image information indicating changes in intensity and / or intensity information of one or more color information. Such a sensor may also be configured to provide small amounts of data by using patch tracking or by using DVS techniques to provide a differential output.
[0185] Passive depth information may be obtained by configuring an image array, such as an image array incorporating any one or more of the techniques described herein, with a component that adapts one or more of the pixel cells in the array to output information indicative of the light field emanating from the object being imaged. The information may be based on the angle of arrival of the light striking that pixel. In some embodiments, a pixel cell, such as those described above, may be configured to output an indication of the angle of arrival by placing a plenoptic component in the light path to the pixel cell. An example of a plenoptic component is a transmissive diffractive mask (TDM). The angle of arrival information may be converted, through calculations, into distance information indicative of the distance to the object from which the light is being reflected, forming the image being captured. In some embodiments, pixel cells configured to provide angle of arrival information may be interspersed with pixel cells that capture light intensity in grayscale or in one or more colors. As a result, the angle of arrival information may also be combined with other image information about the object.
[0186] FIG. 16 illustrates a pixel subarray 100 according to some embodiments. In the illustrated embodiment, the subarray has two pixel cells, but the number of pixel cells in the subarray is not a limitation on the present invention. Here, a first pixel cell 121 and a second pixel cell 122 are shown, one of which is configured to capture angle of arrival information (first pixel cell 121), but it should be understood that the number and location within the array of pixel cells configured to measure angle of arrival information can be varied. In this example, the other pixel cell (second pixel cell 122) is configured to measure the intensity of one color of light, but other configurations are possible, including pixel cells that are sensitive to different colors of light, or one or more pixel cells that are sensitive to a broad spectrum of light such as in a grayscale camera.
[0187] The first pixel cell 121 of the pixel sub-array 100 of FIG. 16 includes an angle of arrival / intensity converter 101, a photodetector 105, and differential readout circuitry 107. The second pixel cell 122 of the pixel sub-array 100 includes a color filter 102, a photodetector 106, and differential readout circuitry 108. It should be understood that not all of the components illustrated in FIG. 16 need be included in all embodiments. For example, some embodiments may not include differential readout circuitry 107 and / or 108, and some embodiments may not include color filter 102. Additionally, additional components not shown in FIG. 16 may be included. For example, some embodiments may include a polarizer arranged to allow light of a particular polarization to reach the photodetector. As another example, some embodiments may include scan output circuitry instead of or in addition to the differential readout circuitry 107. As another example, the first pixel cell 121 may also include a color filter such that the first pixel 121 measures both the angle of arrival and the intensity of a particular color of light incident on the first pixel 121.
[0188] The angle of arrival / intensity converter 101 of the first pixel 121 is an optical component that converts the angle θ of the incident light 111 to an intensity that can be measured by a photodetector. In some embodiments, the angle of arrival / intensity converter 101 may include refractive optics. For example, one or more lenses may be used to convert the angle of incidence of the light to a position on the image plane, the amount of which is detected by one or more pixel cells. In some embodiments, the angle of arrival / position intensity converter 101 may include diffractive optics. For example, one or more diffraction gratings (e.g., TDMs) may convert the angle of incidence of the light to an intensity that can be measured by a photodetector below the TDMs.
[0189] The photodetector 105 of the first pixel cell 121 receives the incident light 110, which passes through the angle of arrival to intensity converter 101, and generates an electrical signal based on the intensity of the light incident on the photodetector 105. The photodetector 105 is located at an image plane associated with the angle of arrival to intensity converter 101. In some embodiments, the photodetector 105 may be a single pixel of an image sensor, such as a CMOS image sensor.
[0190] The differential readout circuitry 107 of the first pixel 121 receives a signal from the photodetector 105 and outputs an event only when the amplitude of the electrical signal from the photodetector differs from the amplitude of the previous signal from the photodetector 105, implementing the DVS technique as described above.
[0191] The second pixel cell 122 includes a color filter 102 for filtering the incident light 112 such that only light within a particular wavelength range passes through the color filter 102 and is incident on the photodetector 106. The color filter 102 may be a bandpass filter, for example, allowing one of red, green, or blue light to pass through it and rejecting other wavelengths of light, and / or may limit the IR light that reaches the photodetector 106 to only a particular portion of the spectrum.
[0192] In this embodiment, the second pixel cell 122 also includes a photodetector 106 and a differential readout circuitry 108, which may function in a similar manner to the photodetector 105 and the differential readout circuitry 107 of the first pixel cell 121.
[0193] As mentioned above, in some embodiments, an image sensor may include an array of pixels, with each pixel associated with a photodetector and readout circuitry. A subset of the pixels may be associated with an angle of arrival / intensity converter that is used to determine the angle of detected light incident on the pixel. Another subset of the pixels may be associated with color filters that are used to determine color information about the scene being viewed, or may selectively pass or block light based on other characteristics.
[0194] In some embodiments, the angle of arrival of light may be determined using a single photodetector and a diffraction grating at two different depths. For example, light may be incident on a first TDM, which converts the angle of arrival to a position, and a second TDM may be used to selectively pass light incident at a particular angle. Such an arrangement may utilize the Talbot effect, a near-field diffraction effect in which when a plane wave is incident on a diffraction grating, an image of the diffraction grating is created at a distance from the diffraction grating. If a second diffraction grating is placed at an image plane, where an image of the first diffraction grating is formed, the angle of arrival may be determined from the intensity of the light measured by a single photodetector positioned after the second grating.
[0195] 17A illustrates a first array of pixel cells 140 including a first TDM 141 and a second TDM 143 aligned with each other such that the ridges and / or regions of increased refractive index for the two gratings are aligned horizontally (Δs=0), where Δs is the horizontal offset between the first TDM 141 and the second TDM 143. Both the first TDM 141 and the second TDM 143 may have the same grating period d, and the two gratings may be separated by a distance / depth z. The depth z, known as the Talbot length, at which the second TDM 143 is located relative to the first TDM 141 may be determined by the grating period d and the wavelength λ of the light being analyzed, and is given by the following equation: [ka]
[0196] As illustrated in FIG. 17A, incident light 142 with a zero degree arrival angle is diffracted by the first TDM 141. The second TDM 143 is located at a depth equal to the Talbot length such that an image of the first TDM 141 is created, resulting in most of the incident light 142 passing through the second TDM 143. An optional dielectric layer 145 may separate the second TDM 143 from the photodetector 147. As the light passes through the dielectric layer 145, the photodetector 147 detects the light and generates an electrical signal with a property (e.g., voltage or current) proportional to the intensity of the light incident on the photodetector. On the other hand, incident light 144 with a non-zero arrival angle θ is also diffracted by the first TDM 141, but the second TDM 143 prevents at least a portion of the incident light 144 from reaching the photodetector 147. The amount of incident light that reaches photodetector 147 depends on the arrival angle θ, with larger angles resulting in less light reaching the photodetector. The dashed line emanating from light 144 illustrates that the amount of light that reaches photodetector 147 is attenuated. In some cases, light 144 may be completely blocked by diffraction grating 143. Thus, information about the arrival angle of the incident light may be obtained using a single photodetector 147 using two TDMs.
[0197] In some embodiments, information obtained by adjacent pixel cells without the angle-of-arrival-to-intensity converter may provide an indication of the intensity of the incident light and may be used to determine the portion of the incident light that passes through the angle-of-arrival-to-intensity converter. From this image information, the angle of arrival of the light detected by photodetector 147 may be calculated, as described in more detail below.
[0198] 17B illustrates a second arrangement of pixel cells 150 including a first TDM 151 and a second TDM 153 that are misaligned with each other such that the ridges and / or regions of increased refractive index for the two gratings are not aligned horizontally (Δs≠0), where Δs is the horizontal offset between the first TDM 151 and the second TDM 153. Both the first TDM 151 and the second TDM 153 may have the same grating period d, and the two gratings may be separated by a distance / depth z. Unlike the situation discussed in connection with FIG. 17A, where the two TDMs are aligned, the misalignment results in incident light at angles different from zero passing through the second TDM 153.
[0199] As illustrated in FIG. 17B, incident light 152 with a zero degree arrival angle is diffracted by the first TDM 151. The second TDM 153 is located at a depth equal to the Talbot length, but due to the horizontal offset of the two gratings, at least a portion of the light 152 is blocked by the second TDM 153. The dashed line originating from the light 152 illustrates that the amount of light reaching the photodetector 157 is attenuated. In some cases, the light 152 may be completely blocked by the diffraction grating 153. On the other hand, incident light 154 with a non-zero arrival angle θ is diffracted by the first TDM 151 but passes through the second TDM 153. After traversing the optional dielectric layer 155, the photodetector 157 detects the light incident on the photodetector 157 and generates an electrical signal with a property (e.g., voltage or current) proportional to the intensity of the light incident on the photodetector.
[0200] Pixel cells 140 and 150 have different output functions, with different intensities of light detected for different angles of incidence. In either case, however, the relationship may be fixed and based on the design of the pixel cell, or determined by measurements as part of a calibration process. Regardless of the precise transfer function, the measured intensity may be converted to an angle of arrival, which may in turn be used to determine the distance to the object being imaged.
[0201] In some embodiments, different pixel cells of the image sensor may have different arrangements of TDMs. For example, a first subset of pixel cells may include a first horizontal offset between the grids of two TDMs associated with each pixel, while a second subset of pixel cells may include a second horizontal offset between the grids of two TDMs associated with each pixel cell, the first offset being different from the second offset. Each subset of pixel cells with a different offset may be used to measure a different angle of arrival or a range of different angles of arrival. For example, a first subset of pixels may include an arrangement of TDMs similar to pixel cell 140 of FIG. 17A, and a second subset of pixels may include an arrangement of TDMs similar to pixel cell 150 of FIG. 17B.
[0202] In some embodiments, not all pixel cells of the image sensor include a TDM. For example, a subset of the pixel cells may include a color filter while a different subset of the pixel cells may include a TDM for determining angle of arrival information. In other embodiments, the color filter is not used such that a first subset of pixel cells simply measures the overall intensity of the incident light and a second subset of pixel cells measures the angle of arrival information. In some embodiments, information about the intensity of light from neighboring pixel cells without a TDM may be used to determine the angle of arrival for light incident on a pixel cell with one or more TDMs. For example, with two TDMs arranged to exploit the Talbot effect, the intensity of light incident on the photodetector after the second TDM is a sinusoidal function of the angle of arrival of the light incident on the first TDM. Thus, if the total intensity of light incident on the first TDM is known, the angle of arrival of the light may be determined from the intensity of the light detected by the photodetector.
[0203] In some embodiments, the configuration of pixel cells in a subarray may be selected to provide different types of image information with appropriate resolution. Figures 18A-C illustrate an example arrangement of pixel cells in a pixel subarray of an image sensor. It should be understood that the illustrated example is non-limiting and alternative pixel arrangements are also contemplated by the inventors. This arrangement may be repeated across an image array, which may contain millions of pixels. The subarray may include one or more pixel cells that provide angle of arrival information for the incident light and one or more other pixel cells (with or without color filters) that provide intensity information for the incident light.
[0204] 18A is an example of a pixel sub-array 160 including a first set of pixel cells 161 and a second set of pixel cells 163 that are different from each other and are rectangular rather than square. The pixel cells labeled "R" are pixel cells with a red filter such that red incident light passes through the filter to the associated photodetector, the pixel cells labeled "B" are pixel cells with a blue filter such that blue incident light passes through the filter to the associated photodetector, and the pixel cells labeled "G" are pixel cells with a green filter such that green incident light passes through the filter to the associated photodetector. In the example sub-array 160, there are more green pixel cells than red or blue pixel cells to illustrate that the various types of pixel cells need not be present in the same proportions.
[0205] The pixel cells labeled A1 and A2 are pixels that provide angle of arrival information. For example, pixel cells A1 and A2 may include one or more grids for determining angle of arrival information. The pixel cells that provide angle of arrival information may be configured similarly or differently, such as to be sensitive to different ranges of angles of arrival or angles of arrival relative to different axes. In some embodiments, the pixels labeled A1 and A2 include two TDMs, and the TDMs of pixel cells A1 and A2 may be oriented in different directions, for example perpendicular to each other. In other embodiments, the TDMs of pixel cells A1 and A2 may be oriented parallel to each other.
[0206] In an embodiment using pixel sub-array 160, both color image data and arrival angle information may be obtained. To determine the arrival angle of light incident on set 161 of pixel cells, the total light intensity incident on set 161 is estimated using electrical signals from the RGB pixel cells. Using the fact that the intensity of light detected by the A1 / A2 pixels varies in a predictable manner as a function of arrival angle, the arrival angle may be determined by comparing the total intensity (estimated from the RGB pixel cells in the group of pixels) with the intensity measured by the A1 and / or A2 pixel cells. For example, the intensity of light incident on the A1 and / or A2 pixels may vary sinusoidally with the arrival angle of the incident light. The arrival angle of light incident on set 163 of pixel cells is determined in a similar manner using electrical signals generated by the pixels of set 163.
[0207] It should be understood that Figure 18A illustrates a specific embodiment of a subarray and that other configurations are possible. In some embodiments, for example, a subarray may be only a set of pixel cells 161 or 163.
[0208] FIG. 18B is an alternative pixel sub-array 170 including a first set of pixel cells 171, a second set of pixel cells 172, a third set of pixel cells 173, and a fourth set of pixel cells 174. Each set of pixel cells 171-174 is square and has identical arrangements of pixel cells therein, but may have pixel cells for determining angle-of-arrival information spanning a range of angles or for different planes (e.g., the TDMs of pixels A1 and A2 may be oriented perpendicular to each other). Each set of pixels 171-174 includes one red pixel cell (R), one blue pixel cell (B), one green pixel cell (G), and one angle-of-arrival pixel cell (A1 or A2). Note that in the exemplary pixel sub-array 170, there are equal numbers of red / green / blue pixel cells in each set. It should further be understood that pixel sub-arrays may be repeated in one or more directions to form a larger array of pixels.
[0209] In embodiments using pixel sub-array 170, both color image data and angle of arrival information may be obtained. To determine the angle of arrival of light incident on set 171 of pixel cells, the total light intensity incident on set 171 may be estimated using signals from the RGB pixel cells. Using the fact that the intensity of light detected by the angle of arrival pixel cells has a sinusoidal or other predictable response to the angle of arrival, the angle of arrival may be determined by comparing the total intensity (estimated from the RGB pixel cells) with the intensity measured by the A1 pixel. The angle of arrival of light incident on set 172-174 of pixel cells may be determined in a similar manner using electrical signals generated by the pixel cells of each individual pixel set.
[0210] FIG. 18C is an alternative pixel sub-array 180 including a first set of pixel cells 181, a second set of pixel cells 182, a third set of pixel cells 183, and a fourth set of pixel cells 184. Each set of pixel cells 181-184 is square with identical arrangements of pixel cells therein, and no color filters are used. Each set of pixel cells 181-184 includes two "white" pixels (e.g., no color filters, so that red, blue, and green light are detected to form a grayscale image), one arriving corner pixel cell (A1) with a TDM oriented in a first direction, and one arriving corner pixel cell (A2) with a TDM oriented with a second spacing or in a second direction (e.g., perpendicular) relative to the first direction. Note that in the exemplary pixel sub-array 170, no color information is present. The resulting image is grayscale, illustrating that passive depth information may be obtained using techniques as described herein in color or grayscale image arrays. As with the other subarray configurations described herein, the pixel subarray arrangement may be repeated in one or more directions to form a larger array of pixels.
[0211] In an embodiment using pixel sub-array 180, both grayscale image data and arrival angle information may be obtained. To determine the arrival angle of light incident on set 181 of pixel cells, the total light intensity incident on set 181 is estimated using electrical signals from two white color pixels. Using the fact that the intensity of light detected by the A1 and A2 pixels has a sinusoidal or other predictable response to arrival angle, the arrival angle may be determined by comparing the total intensity (estimated from the white pixels) with the intensity measured by the A1 and / or A2 pixel cells. The arrival angle of light incident on set 182-184 of pixel cells may be determined in a similar manner using electrical signals generated by the pixels of each individual pixel set.
[0212] In the above examples, the pixel cells are illustrated as squares and are arranged in a square grid. The embodiments are not so limited. For example, in some embodiments, the pixel cells may be rectangular shaped. Furthermore, the subarrays may be triangular, or arranged diagonally, or have other geometric shapes.
[0213] In some embodiments, the angle of arrival information is obtained using the image processor 708 or a processor associated with the local data processing module 70, which may further determine the distance of the object based on the angle of arrival. For example, the angle of arrival information may be combined with one or more other types of information to obtain the distance of the object. In some embodiments, the object in the mesh model 46 may be associated with the angle of arrival information from the pixel array. The mesh model 46 may include the location of the object, including the distance from the user, which may be updated to a new distance value based on the angle of arrival information.
[0214] Using angle of arrival information to determine distance values can be particularly useful in scenarios where objects are close to the user. This is because a change in distance from the image sensor results in a change in the angle of arrival of light for nearby objects that is greater than a similar magnitude change in distance for objects located farther away from the user. Thus, a processing module that utilizes passive distance information based on angle of arrival may selectively use that information based on the estimated distance of the object and may utilize one or more other techniques to determine a distance to the object that exceeds a threshold distance, such as up to 1 meter, up to 3 meters, or up to 5 meters, in some embodiments. As a specific example, the processing module of the AR system may be programmed to use passive distance measurements using angle of arrival information for objects within 3 meters of the user of the wearable device, but for objects outside that range, may use stereoscopic image processing using images captured by the two cameras.
[0215] Similarly, pixels configured to detect angle of arrival information may be most sensitive to changes in distance within a certain range of angles from the normal to the image array. The processing module similarly uses distance information derived from angle of arrival measurements within that range of angles, but may be configured to use other sensors and / or other techniques to determine distances outside that range.
[0216] One exemplary application for determining the distance of an object from an image sensor is hand tracking. Hand tracking may be used in an AR system, for example, to provide a gesture-based user interface for the system 80 and / or to allow a user to move a virtual object in an environment in an AR experience provided by the system 80. The combination of an image sensor providing angle of arrival information for accurate depth determination and differential readout circuitry to reduce the amount of data to the process for determining the user's hand movement provides an efficient interface by which the user can interact with virtual objects and / or provide input to the system 80. A processing module for determining the location of the user's hand may use distance information obtained using different techniques depending on the location of the user's hand within the field of view of the image sensor of the wearable device. Hand tracking may be implemented as a form of patch tracking during the image sensing process, according to some embodiments.
[0217] Another application in which depth information may be useful is occlusion processing. Occlusion processing uses depth information to determine that certain portions of a model of the physical world do not need or can not be updated based on image information being captured by one or more image sensors that collect image information about the physical environment around the user. For example, if a first object is determined to be present at a first distance from the sensor, the system 80 may determine not to update the model of the physical world over distances that exceed the first distance. For example, even if the model includes a second object at a second distance from the sensor, where the second distance exceeds the first distance, model information about that object may not be updated if it is behind the first object. In some embodiments, the system 80 may generate an occlusion mask based on the location of the first object and update only the portions of the model that are not masked by the occlusion mask. In some embodiments, the system 80 may generate more than one occlusion mask for more than one object. Each occlusion mask may be associated with a distinct distance from the sensor. For each occlusion mask, model information associated with objects at a distance from the sensor greater than the distance associated with the respective occlusion mask will not be updated. By limiting the portion of the model that is updated at any given time, the speed at which the AR environment is generated and the amount of computational resources required to generate the AR environment are reduced.
[0218] Although not shown in FIGS. 18A-C, some embodiments of the image sensor may include pixels with IR filters in addition to or instead of color filters. For example, the IR filter may allow light of a wavelength, such as approximately equal to 940 nm, to pass through and be detected by an associated photodetector. Some embodiments of the wearable may include an IR light source (e.g., an IR LED) that emits light of the same wavelength as that associated with the IR filter (e.g., 940 nm). The IR light source and IR pixels may be used as an alternative method of determining the distance of an object from the sensor. By way of example, and not limitation, the IR light source may be pulsed and a time-of-flight measurement may be used to determine the object distance from the sensor.
[0219] In some embodiments, the system 80 may be capable of operating in one or more operating modes. A first mode may be a mode in which the depth determination is made using passive depth measurements, for example based on the angle of arrival of light determined using a pixel with an angle of arrival / intensity converter. A second mode may be a mode in which the depth determination is made using active depth measurements, for example based on the time of flight of IR light measured using an IR pixel of an image sensor. A third mode may use stereoscopic measurements from two separate image sensors to determine the distance of an object. Such stereoscopic measurements may be more accurate than using the angle of arrival of light determined using a pixel with an angle of arrival / intensity converter when the object is very far from the sensor. Other suitable methods of determining depth may also be used for one or more additional operating modes for depth determination.
[0220] In some embodiments, it may be preferable to use passive depth determination because such techniques utilize less power. However, the system may determine that it should operate in an active mode under certain conditions. For example, when the intensity of visible light being detected by the sensor is below a threshold, it may be too dark to perform passive depth determination accurately. In some embodiments, IR illumination may be used to counteract the effects of low light. IR illumination may be selectively enabled in response to detected low light conditions that prevent the device from obtaining sufficient images for the task it is performing, such as head tracking or hand tracking. As another example, an object may be too far away for passive depth determination to be accurate. Thus, the system may be programmed to select to operate in a third mode, where depth is determined based on a stereoscopic measurement of the scene using two spatially separated image sensors. As another example, determining the depth of an object based on the angle of arrival of light determined using a pixel with an angle of arrival / intensity converter may be inaccurate at the periphery of the image sensor. Thus, if an object is detected by pixels near the periphery of the image sensor, the system may choose to operate in the second mode, using active depth determination.
[0221] While the image sensor embodiments described above used individual pixel cells with stacked TDMs to determine the angle of arrival of light incident on the pixel cell, other embodiments may use a group of multiple pixel cells with a single TDM across all pixels of the group to determine angle of arrival information. The TDM may project a pattern of light across the sensor array, where the pattern depends on the angle of arrival of the incident light. Multiple photodetectors associated with one TDM may detect the pattern more accurately because each photodetector of the multiple photodetectors is located at a different position in the image plane (the image plane that includes the photodetectors that sense light). The relative intensity sensed by each photodetector may indicate the angle of arrival of the incident light.
[0222] FIG. 19A is a top plan view example of a plurality of photodetectors (in the form of a photodetector array 120, which may be a subarray of pixel cells of an image sensor) associated with a single transmissive diffractive mask (TDM), according to some embodiments. FIG. 19B is a cross-sectional view of the same photodetector array as FIG. 19A, taken along line A in FIG. 19A. The photodetector array 120 includes 16 separate photodetectors 121, in the example shown, which may be within pixel cells of the image sensor. The photodetector array 120 includes a TDM 123, which is disposed above the photodetectors. It should be understood that each group of pixel cells is illustrated with four pixels (e.g., forming a grid of four pixels by four pixels) for clarity and simplicity. Some embodiments may include more than four pixel cells. For example, 16 pixel cells, 64 pixel cells, or any other number of pixels may be included within each group.
[0223] The TDM 123 is located a distance x from the photodetector 121. In some embodiments, the TDM 123 is formed on the top surface of the dielectric layer 125, as illustrated in FIG. 19B. For example, the TDM 123 may be formed from ridges or by valleys etched into the surface of the dielectric layer 125, as illustrated. In other embodiments, the TDM 123 may be formed within the dielectric layer. For example, portions of the dielectric layer may be modified to have a higher or lower refractive index relative to other portions of the dielectric layer, resulting in a holographic phase grating. Light incident on the photodetector array 120 from above is diffracted by the TDM, resulting in an arrival angle of the incident light as translated to a position in an image plane a distance x from the TDM 123 where the photodetector 121 is located. The intensity of the incident light measured at each photodetector 121 of the photodetector array may be used to determine the arrival angle of the incident light.
[0224] FIG. 20A illustrates an example of multiple photodetectors (in the form of a photodetector array 130) associated with multiple TDMs, according to some embodiments. FIG. 20B is a cross-sectional view of the same photodetector array as FIG. 20A through line B of FIG. 20A. FIG. 20C is a cross-sectional view of the same photodetector array as FIG. 20A through line C of FIG. 20A. The photodetector array 130 includes 16 separate photodetectors, in the example shown, which may be within pixel cells of an image sensor. As shown, there are four groups 131a, 131b, 131c, 131d of four pixel cells. The photodetector array 130 includes four separate TDMs 133a, 133b, 133c, 133d, each TDM provided above an associated group of pixel cells. It should be understood that each group of pixel cells is illustrated with four pixel cells for clarity and simplicity. Some embodiments may include more than four pixel cells. For example, 16 pixel cells, 64 pixel cells, or any other number of pixel cells may be included within each group.
[0225] Each TDM 133a-d is located a distance x from the photodetectors 131a-d. In some embodiments, the TDMs 133a-d are formed on the top surface of the dielectric layer 135, as illustrated in FIG. 20B. For example, the TDMs 123a-d may be formed from ridges or by valleys etched into the surface of the dielectric layer 135, as illustrated. In other embodiments, the TDMs 133a-d may be formed within the dielectric layer. For example, portions of the dielectric layer may be modified to have a higher or lower refractive index relative to other portions of the dielectric layer, resulting in a holographic phase grating. Light incident on the photodetector array 130 from above is diffracted by the TDMs, resulting in an arrival angle of the incident light that translates to a position in an image plane at a distance x from the TDMs 133a-d where the photodetectors 131a-d are located. The intensity of the incident light measured at each photodetector 131a-d of the photodetector array may be used to determine the arrival angle of the incident light.
[0226] TDMs 133a-d may be oriented in different directions from each other. For example, TDM 133a is perpendicular to TDM 133b. Thus, the intensity of light detected using photodetector group 131a may be used to determine the angle of arrival of the incident light in a plane perpendicular to TDM 133a, and the intensity of light detected using photodetector group 131b may be used to determine the angle of arrival of the incident light in a plane perpendicular to TDM 133b. Similarly, the intensity of light detected using photodetector group 131c may be used to determine the angle of arrival of the incident light in a plane perpendicular to TDM 133c, and the intensity of light detected using photodetector group 131d may be used to determine the angle of arrival of the incident light in a plane perpendicular to TDM 133d.
[0227] Pixel cells configured to passively obtain depth information may be integrated into an image array with features as described herein to support operations useful in cross-reality systems. According to some embodiments, pixel cells configured to obtain depth information may be implemented as part of an image sensor used to implement a camera with a global shutter. Such a configuration may provide, for example, a complete frame output. A complete frame may include image information for different pixels simultaneously indicating depth and intensity. With an image sensor of this configuration, a processor may obtain depth information for the complete scene at once.
[0228] In other embodiments, the pixel cells of the image sensor that provide the depth information may be configured to operate according to the DVS technique as described above. In such a scenario, the event may indicate a change in the depth of the object as indicated by the pixel cell. The event output by the image array may indicate the pixel cell for which a change in depth was detected. Alternatively, or in addition, the event may include a value of the depth information for that pixel cell. With an image sensor of this configuration, the processor may obtain depth information updates at a very high rate to provide high temporal resolution. In some embodiments, high temporal resolution may involve updating the depth information more frequently than 1 Hz or 5 Hz, or may instead involve updating the depth information hundreds or thousands of times per second, such as every millisecond.
[0229] In yet other embodiments, the image sensor may be configured to operate in either full frame or DVS mode. In such embodiments, a processor, which processes image information from the image sensor, may programmatically control the image sensor's mode of operation based on the function being performed by the processor. For example, while performing a function involving tracking an object, the processor may configure the image sensor to output image information as DVS events. On the other hand, while processing to update the world reconstruction, the processor may configure the image sensor to output full frame depth information. In some embodiments, in full frame and / or DVS mode, active illumination may be used to illuminate the scene being imaged. For example, IR illumination may be provided in instances where the intensity of light being detected by the sensor is below a threshold that may be too dark to accurately perform passive depth determination.
[0230] Wearable Configuration
[0231] Multiple image sensors may be used within an XR system. The image sensors may be combined with optical components such as lenses and control circuitry to create a camera. The image sensors may obtain imaging information using one or more of the techniques described above, such as grayscale imaging, color imaging, global shutter, DVS techniques, plenoptic pixel cells, and / or dynamic patches. Regardless of the imaging technique used, the resulting camera may be mounted to a support member to form a headset, which may include or be connected to a processor.
[0232] FIG. 21 is a schematic diagram illustrating a headset 2100 of a wearable display system consistent with disclosed embodiments. As shown in FIG. 21, the headset 2100 may include a display device with monocular 2110a and monocular 2110b, which may be optical eyepieces or displays configured to transmit and / or display visual information to a user's eye. The headset 2100 may also include a frame 2101, which may be similar to the frame 64 described above with respect to FIG. 3B. The headset 2100 may further include two cameras (DVS camera 2120 and camera 2140) and additional components such as an emitter 2130a, an emitter 2130b, an inertial measurement unit 2170a (IMU 2170a), and an inertial measurement unit 2170b (IMU 2170b).
[0233] Camera 2120 and camera 2140 are world cameras that are oriented to image the physical world as seen by a user wearing headset 2100. In some embodiments, the two cameras may be sufficient to obtain image information about the physical world, and the two cameras may be the only world-facing cameras. Headset 2100 may also include additional components, such as an eye-tracking camera, as discussed above with respect to FIG. 3B.
[0234] The monocular 2110a and the monocular 2110b may be mechanically coupled to a support member such as the frame 2101 using techniques such as adhesives, fasteners, pressure fitting, etc. Similarly, the two cameras and ancillary components (e.g., emitters, inertial measurement units, eye tracking cameras, etc.) may be mechanically coupled to the frame 2101 using techniques such as adhesives, fasteners, pressure fitting, etc. These mechanical couplings may be direct or indirect. For example, one or more of the cameras and / or one or more of the ancillary components may be directly attached to the frame 2101. As an additional example, one or more of the cameras and / or one or more of the ancillary components may be directly attached to the monocular, which may then be attached to the frame 2101. The mechanism of attachment is not intended to be limiting.
[0235] Alternatively, monocular subassemblies may be formed and then mounted to the frame 2101. Each subassembly may include, for example, a support member to which the monocular 2110a or 2110b is mounted. The IMU and one or more cameras may be mounted to the support member as well. Mounting both the camera and the IMU to the same support member may allow inertial information about the camera to be obtained based on the output of the IMU. Similarly, mounting the monocular to the same support member as the camera may allow image information about the world to be spatially correlated with information rendered on the monocular.
[0236] The headset 2100 may be lightweight. For example, the headset 2100 may weigh between 30 and 300 grams. The headset 2100 may be made of a material that flexes when in use, such as plastic or thin metal. Such materials may allow for a lightweight and comfortable headset that may be worn by a user for extended periods of time. An XR system with such a lightweight headset may still support high accuracy stereoscopic image analysis (requiring that the separation between the cameras is accounted for) using a calibration routine that may be repeated as the headset is worn and compensate for any inaccuracies that may result from flexing of the headset when in use. In some embodiments, the lightweight headset may include a battery pack. The battery pack may include one or more batteries, which may be rechargeable or non-rechargeable. The battery pack may be built into the lightweight frame or may be removable. The battery pack and the lightweight frame may be formed as a single unit, or the battery pack may be formed as a unit separate from the lightweight frame.
[0237] The DVS camera 2120 may include an image sensor and a lens. The image sensor may be configured to produce a grayscale image. The image sensor may be configured to output an image with a size between 1 megapixel and 4 megapixels. For example, the image sensor may be configured to output an image with a horizontal resolution of 1,016 lines by a vertical resolution of 1,016 lines. In some aspects, the image sensor may be a CMOS image sensor.
[0238] The DVS camera 2120 may support dynamic visual sensing, as disclosed above with respect to FIGS. 4, 5A, and 5B. In one mode of operation, the DVS camera 2120 may be configured to output image information in response to a detected change in an image property, such as a pixel-level change in light intensity. In some embodiments, the detected change may satisfy an intensity change criterion. For example, the image sensor may be configured to have one threshold for incremental increases in light intensity and another threshold for incremental decreases in light intensity. Image information may be provided asynchronously as such changes are detected. In another mode of operation, the DVS camera 120 may be configured to repeatedly or periodically output image frames in response to a detected change in an image property. For example, the DVS camera 2120 may be configured to output images at a frequency between 30 Hz and 120 Hz, such as 60 Hz.
[0239] The DVS camera 2120 may support patch tracking, as disclosed above with respect to FIGS. 6-15. For example, the DVS camera 2120, in various aspects, may be configured to provide image information for a subset of pixels in an image sensor (e.g., a patch of pixels in an image sensor). The DVS camera 2120 may be configured to combine dynamic vision sensing and patch tracking and provide image information for those pixels in a patch that are subject to changes in image properties. In various aspects, the image sensor can be configured with a global shutter. As discussed above with respect to FIGS. 14 and 15, the global shutter may enable each pixel to obtain an intensity measurement simultaneously.
[0240] The DVS camera 2120 may be a plenoptic camera, as disclosed above with respect to FIG. 3B and FIG. 15-20C. For example, a component may be placed in the light path to one or more pixel cells of the image sensor of the DVS camera 2120 such that these pixel cells produce an output having an intensity indicative of the angle of arrival of light incident on the pixel cell. In such an embodiment, the image sensor may passively obtain depth information. An example of a component suitable for placement in the light path is a TDM filter. The processor may be configured to use the angle of arrival information to calculate a distance to the object being imaged. For example, the angle of arrival information may be converted to distance information indicative of the distance to the object from which the light is being reflected. In some embodiments, pixel cells configured to provide angle of arrival information may be interspersed with pixel cells that capture light intensities of one or more colors. As a result, the angle of arrival information, and therefore the distance information, may be combined with other image information about the object.
[0241] The DVS camera 2120 can be configured to have a wide field of view, consistent with disclosed embodiments. For example, the DVS camera 2120 may include an equidistant lens (e.g., a fisheye lens). The DVS camera 2120 may be angled toward the camera 2140 to create an area directly in front of the user of the headset 2100 that is imaged by both cameras. For example, a vertical plane passing through the center of the field of view 2121, the field of view associated with the DVS camera 2120, may intersect and form an angle with a vertical plane passing through the midline of the headset 2100. In some embodiments, the field of view 2121 may have a horizontal field of view and a vertical field of view. The horizontal field of view may range from 90 degrees to 175 degrees, while the vertical field of view may range from 70 degrees to 125 degrees. In some embodiments, the DVS camera 2120 can be configured to possess an angular pixel resolution of 1 to 5 minutes per pixel.
[0242] Emitters 2130a and 2130b may enable imaging and / or active depth sensing by headset 2100 under low light conditions. Emitters 2130a and 2130b may be configured to emit light at specific wavelengths. This light can be reflected by physical objects in the physical world around the user. Headset 2100 may be configured with sensors to detect this reflected light, including image sensors as described herein. In some embodiments, these sensors may be incorporated into at least one of cameras 2120 or cameras 2140. For example, as described above with respect to FIGS. 18A-18C, these cameras may be configured with detectors corresponding to emitters 2130a and / or emitters 2130b. For example, these cameras may include pixels configured to detect light emitted by emitters 2130a and / or emitters 2130b.
[0243] Emitter 2130a and emitter 2130b may be configured to emit IR light consistent with disclosed embodiments. The IR light may have a wavelength between 900 nanometers and 1 micrometer. The IR light may be a 940 nm light source, for example, the emitted light energy is centered at about 940 nm. Emitters emitting light at other wavelengths may alternatively or additionally be used. For systems intended for indoor use only, for example, an emitter emitting light centered at about 850 nm may be used. At least one of DVS camera 2120 or camera 2140 may include one or more IR filters positioned across at least a subset of pixels in the camera's image sensor. The filters may pass light at wavelengths emitted by emitter 2130a and / or emitter 2130b while attenuating light at certain other wavelengths. For example, the IR filters may be notch filters that pass IR light with wavelengths matching those of the emitters. The notch filter may substantially attenuate other IR light. In some embodiments, the notch filter may be an IR notch filter that blocks IR light and allows light from the emitter to pass. The IR notch filter may also allow light outside the IR band to pass. Such a notch filter may allow the image sensor to receive both visible light and light from the emitter reflected from objects within the field of view of the image sensor. In this manner, a subset of pixels may act as a detector for IR light emitted by emitter 2130a and / or emitter 2130b.
[0244] In some embodiments, a processor of the XR system may selectively enable emitters, such as to enable imaging under low light conditions. The processor may process image information generated by one or more image sensors and detect whether images output by those image sensors would provide adequate information about objects in the physical world without the emitters enabled. The processor may enable an emitter in response to detecting that the images do not provide adequate image information as a result of low ambient light conditions. For example, an emitter may be turned on when stereoscopic information is being used to track an object and a lack of ambient light results in an image with insufficient contrast between features of the tracked object to accurately determine distances using stereoscopic imaging techniques.
[0245] Alternatively or in addition, emitter 2130a and / or emitter 2130b may be configured for use in making active depth measurements, such as by emitting light in short pulses. The wearable display system may be configured to perform time-of-flight measurements by detecting reflections of such pulses from objects within the illumination field 2131a of emitter 2130a and / or the illumination field 2131b of emitter 2130b. These time-of-flight measurements may provide additional depth information for tracking objects or updating a passable world model. In other embodiments, one or more emitters may be configured to emit patterned light, and the XR system may be configured to process images of objects illuminated by the patterned light. Such processing may detect variations in the pattern, which may reveal distances to the objects.
[0246] In some embodiments, the extent of the illumination field associated with an emitter may be sufficient to illuminate at least the field of view of a camera used to obtain image information about the object. For example, the emitters may collectively illuminate the central field of view 2150. In the illustrated embodiment, emitter 2130a and emitter 2130b may be positioned to collectively illuminate illumination fields 2131a and 2131b, which span a range in which active illumination may be provided. In this exemplary embodiment, two emitters are shown, but it should be understood that more or fewer emitters may be used to span a desired range.
[0247] In some embodiments, emitters such as emitters 2130a and 2130b may be turned off by default, but may be enabled when additional illumination is desired to obtain more information than can be obtained using passive imaging. The wearable display system may be configured to enable emitter 2130a and / or emitter 2130b when additional depth information is required. For example, when the wearable display system detects that it cannot obtain adequate depth information for tracking hand or head pose using stereoscopic image information, the wearable display system may be configured to enable emitter 2130a and / or emitter 2130b. The wearable display system may be configured to disable emitter 2130a and / or emitter 2130b when additional depth information is not required, thereby reducing power consumption and improving battery life.
[0248] Furthermore, even if the headset is configured with an image sensor configured to detect IR light, it is not a requirement that the IR emitter be mounted on the headset 2100 or only on the headset 2100. In some embodiments, the IR emitter may be an external device disposed in a space, such as a room, in which the headset 2100 may be used. Such an emitter may project IR light, such as in an ArUco pattern, at 940 nm, which is invisible to the human eye. Light with such a pattern may facilitate "instrumentation / assisted tracking," in which the headset 2100 does not need to provide power to provide the IR pattern, but may still provide IR image information as a result of which the pattern is presented so that processing performed on that image information may determine the distance to or location of an object in the space. A system with an external illumination source may also allow more devices to operate in the space. If multiple headsets are operating in the same space, each moving around the space without a fixed positional relationship, there is a risk that light emitted by one headset will be projected onto the image sensor of another headset and thus interfere with its operation. Such a risk of interference between headsets may limit the number of headsets that may operate in a space to, for example, three or four. Using one or more IR emitters in the space that illuminate objects that may be imaged by an image sensor on the headset, in some embodiments many more than ten headsets may operate in the same space without interference.
[0249] As disclosed above with respect to FIG. 3B, the camera 2140 may be configured to capture images of the physical world within the field of view 2141. The camera 2140 may include an image sensor and a lens. The image sensor may be configured to produce color images. The image sensor may be configured to output images of a size between 4 megapixels and 16 megapixels. For example, the image sensor may output a 12 megapixel image. The image sensor may be configured to output images repeatedly or periodically. For example, the image sensor may be configured to output images at a frequency between 30 Hz and 120 Hz, such as 60 Hz, when enabled. In some embodiments, the image sensor may be configured to selectively output images based on the task being performed. The image sensor may be configured with a rolling shutter. As discussed above with respect to FIG. 14 and FIG. 15, the rolling shutter may repeatedly read subsets of pixels in the image sensor such that pixels in different subsets reflect light intensity data collected at different times. For example, the image sensor may be configured to read a first row of pixels in the image sensor at a first time, and a second row of pixels in the image sensor at some later time.
[0250] In some embodiments, the camera 2140 can be configured as a plenoptic camera. For example, as discussed above with respect to FIG. 3B and FIG. 15-20C, a component can be placed in the light path to one or more pixel cells of the image sensor such that these pixel cells produce an output having an intensity indicative of the angle of arrival of light incident on the pixel cell. In such an embodiment, the image sensor may passively obtain depth information. An example of a component suitable for placement in the light path is a transmissive diffractive mask (TDM) filter. The processor can be configured to use this angle of arrival information to calculate a distance to the object being imaged. For example, the angle of arrival information can be converted into distance information indicative of the distance to the object from which the light is being reflected. In some embodiments, pixel cells configured to provide angle of arrival information may be interspersed with pixel cells that capture light intensities of one or more colors. As a result, the angle of arrival information, and therefore the distance information, may be combined with other image information about the object. In some embodiments, similar to the DVS camera 2120, the camera 2140 may be configured to provide event detection and patch tracking functionality. The processor of the headset 2100 can be configured to provide instructions to the camera 2140 to limit image capture to a subset of pixels. In some embodiments, the sensor may be a CMOS sensor.
[0251] The camera 2140 can be positioned on the side of the headset 2100 opposite the DVS camera 2120. For example, as shown in FIG. 21, when the camera 2140 is on the same side of the headset 2100 as the monocular 2110a, the DVS camera 2120 can be on the same side of the headset as the monocular 2110b. The camera 2140 can be angled inward on the headset 2100. For example, a vertical plane passing through the center of the field of view 2141, the field of view associated with the camera 2140, can intersect and form an angle with a vertical plane passing through the midline of the headset 2100. The field of view 2141 of the camera 2140 may have a horizontal field of view and a vertical field of view. The horizontal field of view may range from 75 to 125 degrees, while the vertical field of view may range from 60 to 125 degrees.
[0252] The camera 2140 and the DVS camera 2120 can be asymmetrically angled inward toward the midline of the headset 2100. The angle of the camera 2140 can be 1 to 20 degrees inward toward the midline of the headset 2100. The angle of the DVS camera 2120 can be 1 to 40 degrees inward toward the midline of the headset 2100 and can be different than the angle of the camera 2140. The angular range of the field of view 2121 can exceed the angular range of the field of view 2141.
[0253] The DVS camera 2120 and the camera 2140 may be configured to provide overlapping views of a central field of view 2150. The angular range of the central field of view 2150 may be between 40 and 120 degrees. For example, the angular range of the central field of view 2150 may be about 70 degrees (e.g., 70±7 degrees). The central field of view 2150 may be asymmetric. For example, the central field of view 2150 may extend further toward the side of the headset 2100 that includes the camera 2140, as shown in FIG. 21. In addition to the central field of view 2150, the DVS camera 2120 and the camera 2140 may be positioned to provide at least two peripheral fields of view. The peripheral field of view 2160a may be associated with the DVS camera 2120 and may include that portion of the field of view 2121 that does not overlap with the field of view 2141. In some embodiments, the horizontal angular range of the peripheral field of view 2160a may be in the range of 20 to 80 degrees. For example, the angular range of the peripheral field of view 2160a may be approximately 40 degrees (e.g., 40±4 degrees). The peripheral field of view 2160b (not depicted in FIG. 21) may be associated with the camera 2140 and may include that portion of the field of view 2141 that does not overlap with the field of view 2121. In some embodiments, the horizontal angular range of the peripheral field of view 2160b may range from 10 to 40 degrees. For example, the angular range of the peripheral field of view 2160a may be approximately 20 degrees (e.g., 20±2 degrees). The location of the peripheral field of view may vary. For example, for a certain configuration of the headset 2100, the peripheral field of view 2160b may not extend within 0.25 meters of the headset 2100 because, within that distance, the field of view 2141 may fall entirely within the field of view 2121. In contrast, the peripheral field of view 2160a may extend within 0.25 meters of the headset 2100. In such a configuration, the wider field of view and larger inward angle of the DVS camera 2120 may ensure that even within 0.25 meters of the headset 2100, the field of view 2121 falls, at least in part, outside the field of view 2141, as shown in FIG. 21.
[0254] IMU 2170a and / or IMU 2170b may be configured to provide acceleration and / or velocity and / or tilt information to the wearable display system. For example, as a user wearing headset 2100 moves, IMU 2170a and / or IMU 2170b may provide information describing the acceleration and / or velocity of the user's head.
[0255] The XR system may be coupled to a processor, which may be configured to process image data output by the camera and / or render virtual objects on a display device. The processor may be mechanically coupled to the frame 2101. Alternatively, the processor may be mechanically coupled to a display device, such as a display device including the monocular 2110a or monocular 2110b. As a further alternative, the processor may be operably coupled to the headset 2100 and / or the display device through a communication link. For example, the XR system may include a local data processing module. This local data processing module may include a processor and may be connected to the headset 2100 or the display device through a physical connection (e.g., a wire or cable) or a wireless (e.g., Bluetooth, Wi-Fi, Zigbee, or equivalent) connection.
[0256] The processor may be configured to perform world reconstruction, head pose tracking, and object tracking operations. The processor may be configured to create a passable world model using the DVS camera 2120 and the camera 2140. In creating the passable world model, the processor may be configured to stereoscopically determine depth information using multiple images of the same physical object acquired by the DVS camera 2120 and the camera 2140. The processor may be configured to update an existing passable world model using the DVS camera 2120 but without the camera 2140. As mentioned above, the DVS camera 2120 may be a grayscale camera with a relatively lower resolution than the color camera 2140. Furthermore, the DVS camera 2120 may output image information asynchronously (e.g., in response to a detected event), allowing the processor to asynchronously update the passable world model, head pose, and / or object location only when a change is detected. In some embodiments, the processor may provide instructions to the DVS camera 2120 to restrict image data acquisition to one or more patches of pixels within the image sensor of the DVS camera 2120. The DVS camera 2120 may then restrict image data acquisition to these patches of pixels. As a result, updating the passable world model using image information output by the DVS camera 2120 rather than the camera 2140 may be performed quickly with reduced power consumption and improved battery life. In some embodiments, the processor may be configured to update the passable world model using image information output by the DVS camera 2120 on an occasional or periodic basis.
[0257] While the processor may preferentially update the passable world model using the DVS camera 2120, the processor may in some embodiments update the passable world model occasionally or periodically using both the DVS camera 2120 and the camera 2140. For example, the processor may be configured to determine that a passable world quality criterion is no longer satisfied, that a predetermined time interval has elapsed since the last acquisition and / or use of the image output by the camera 2140, and / or that changes have occurred to objects within the portion of the physical world currently within the field of view of the DVS camera 2120 and the camera 2140.
[0258] After creating the passable world model, the processor may preferentially track the position of the head, objects, and / or hands using image information output by the DVS camera 2120. As described above, the DVS camera 2120 may be configurable for event-based image acquisition. The acquired image information may be specific to one or more patches within the image sensor of the DVS camera 2120. Alternatively, the processor may track the position of the head, objects, and / or hands using monocular images output by the DVS camera 2120 or the camera 2140. These monocular images may be full frame images or may be output by the DVS camera 2120 on a repetitive or cyclical basis. For example, the DVS camera may be configured to output images at a frequency of 30 Hz to 120 Hz, such as 60 Hz, while changes may be output at a much higher, effective rate, such as hundreds or thousands of times per second. In some embodiments, the processor may track the position of the head, objects, and / or hands using stereoscopic projection images output by both the DVS camera 2120 and the camera 2140. These stereoscopic projection images may be output repeatedly or periodically.
[0259] The processor may be configured to use a plenoptic camera to obtain light field information, such as angle-of-arrival information regarding light incident on an image sensor. In some embodiments, the plenoptic camera can be at least one of the DVS camera 2120 or the camera 2140. Consistent with disclosed embodiments, such depth information may be determined from or supplemented by light field information acquired by the plenoptic camera, when depth information may be described herein or enhance processing. For example, when the camera 2140 includes a TDM filter, the processor may be configured to use images acquired from the DVS cameras 2120 and 2140, along with light field information acquired from the camera 2140, to create a passable world model. Alternatively, or in addition, when the DVS camera 2120 includes a TDM filter, the processor may use light field information acquired from the DVS camera 2120.
[0260] The processor may be configured to detect conditions in which one or more types of image information are unavailable or insufficient to provide resolution for a particular function, such as world reconstruction, object tracking, or head pose determination. The processor may be configured to select additional or alternative sources of image information to provide sufficient image information for the function. These sources may be selected in an order that results in suitable image information available with low processing burden in each situation. For example, the DVS camera 2120 may be configured to output image data that satisfies an intensity change criterion, but pixel intensity may not change enough to trigger image data acquisition when an object of uniform visual characteristics fills the field of view 2121. Such an example may be true, for example, when a user's hand fills the field of view even as the user's hand moves. In some embodiments, even partial filling of the field of view 2121 may inhibit or prevent acquisition of suitable image data. For example, when patches tracked by the processor are filled with images of objects of uniform appearance, the image data for those patches may be inappropriate for tracking the object. Similarly, if a nearby object fills the camera's field of view or at least the patch used to track points of interest, such as points of interest tracked to track head pose, the image information from that camera may be inappropriate for the head tracking function.
[0261] However, given the geometry of the headset 2100 shown in Figure 21, the camera 2140 may be positioned to output images of the object that fill a relevant portion of the field of view. These images may alternatively or additionally be used to determine the depth of the object. The camera 2140 may also be positioned to image a point of interest that is tracked within the patch for head pose tracking.
[0262] Thus, the processor can be configured to determine whether the object satisfies a filling criterion for the field of view 2121. Based on this determination, the processor can be configured to enable the camera 2140 or increase the frame rate of the camera 2140. The processor can then receive at least image data from the camera 2140. In some embodiments, the image data can include light field information. For object tracking, the processor can be configured to use the received image data to determine depth information. The processor can use the determined depth information to track an occluded object. For head pose tracking, the processor can be configured to use the received image data to determine a change in head pose. When the DVS camera 2120 is configured for patch tracking of a point of interest, the processor can be configured to track the point of interest using the camera 2120. The processor may be configured to disable the camera 2140 or reduce its frame rate based on determining that the filling criteria is no longer satisfied, and revert to using data that is acquired and / or processed more quickly with lower latency, but that may be adequate for the function being performed, such as object tracking or head pose tracking. This may enable the DVS camera 2120 to track images at very high temporal resolution. For example, the DVS camera 2120 may output information indicating the current position of a moving object at a frequency higher than 60 Hz, such as at a frequency of hundreds or thousands of times per second.
[0263] It should be appreciated that the processor may enable or disable the cameras to dynamically provide different sources of image information for one or more functions based on operating conditions detected in any one or more ways. The processor may send control signals to the underlying image sensor hardware to modify the operation of the hardware. Alternatively or in addition, the processor may enable the camera by reading the image information generated by the camera, or disable the camera by not accessing or using the image information generated by the camera. These techniques may be used to enable or suppress image information, in whole or in part. For example, the processor may be configured to perform a size reduction routine and adjust the image output using the camera 2140. The camera 2140 may generate larger images than the DVS camera 2120. For example, the camera 2140 may generate 12 megapixel images, while the DVS camera 2120 may generate 1 megapixel images. The images generated by the camera 2140 may contain more information than is necessary to perform passable world creation, head tracking, or object tracking operations. Processing this additional information may require additional power, reduce battery life, or increase latency. Thus, the processor can be configured to discard or combine pixels in the image output by the camera 2140. For example, the processor can be configured to output an image with 1 / 16th the number of pixels as the originally generated image. Each pixel in this output image can have a value based on a corresponding 4x4 set of pixels in the originally generated image (e.g., the average of the values of these 16 pixels).
[0264] The XR system may include a hardware accelerator, according to some embodiments. The hardware accelerator may be implemented as an application specific integrated circuit (ASIC) or other semiconductor device, and may be integrated within or otherwise coupled to the headset 2100 to receive image information from the camera 2120 and the camera 2140. This hardware accelerator may assist in the stereoscopic determination of depth information using images output by these two world cameras. The images from the camera 2120 may be grayscale images, and the images from the camera 2140 may be color images. Using hardware acceleration may accelerate the determination of depth information and reduce power consumption, thus increasing battery life.
[0265] Exemplary Calibration Process
[0266] FIG. 22 depicts a simplified flowchart of a calibration routine (method 2200) according to some embodiments. The processor may be configured to perform the calibration routine while the wearable display system is worn. The calibration routine may address distortion resulting from the lightweight structure of the headset 2100. For example, the processor may perform the calibration routine repeatedly such that the calibration routine compensates for distortion in the frame 2101 during use of the wearable display system. The compensation routine may be performed automatically or in response to a manual input (e.g., a user request to perform the calibration routine). The calibration routine may include determining the relative positions and orientations of the cameras 2120 and 2140. The processor may be configured to perform the calibration routine using images output by the DVS cameras 2120 and 2140. In some embodiments, the processor may be further configured to use the output of the IMUs 2170a and 2170b.
[0267] After starting at block 2201, the method 2200 may proceed to block 2210. In block 2210, the processor may identify corresponding features in images output from the DVS camera 2120 and the camera 2140. The corresponding features may be part of an object in the physical world. In some embodiments, the object may have features easily identifiable in the image that may be placed in the central field of view 2150 by the user for calibration purposes and may have a predetermined relative position. However, the calibration techniques described herein may be implemented to allow calibration to be repeated during use of the headset 2100 based on features on objects present in the central field of view 2150 at the time of calibration. In various embodiments, the processor may be configured to automatically select features detected in both the field of view 2121 and the field of view 2141. In some embodiments, the processor may be configured to use the estimated locations of the features in the field of view 2121 and the field of view 2141 to determine a correspondence between the features. Such an estimation may be based on a passable world model constructed for the object containing these features or other information about the features.
[0268] The method 2200 may proceed to block 2230. In block 2230, the processor may receive inertial measurement data. The inertial measurement data may be received from the IMU 2170a and / or IMU 2170b. The inertial measurement data may include tilt and / or acceleration and / or velocity measurements. In some embodiments, the IMUs 2170a and 2170b may be mechanically coupled, directly or indirectly, to the camera 2140 and the DVS camera 2120, respectively. In such embodiments, differences in the inertial measurements, such as tilt, made by the IMUs 2170a and 2170b may indicate differences in the positions and / or orientations of the camera 2140 and the camera 2120. Thus, the outputs of the IMUs 2170a and 2170b may provide a basis for making an initial estimate of the relative positions of the camera 2140 and the DVS camera 2120.
[0269] After block 2230, the method 2200 may proceed to block 2250. In block 2250, the processor may calculate an initial estimate of the relative position and orientation of the DVS camera 2120 and the camera 2140. This initial estimate may be calculated using measurements received from IMU 2170b and / or IMU 2170a. In some embodiments, for example, the headset may be designed with a nominal relative position and orientation of the DVS camera 2120 and the camera 2140. The processor may be configured to attribute differences in the measurements received between IMU 2170a and IMU 2170b to distortions in the frame 2101 that may alter the position and / or orientation of the DVS camera 2120 and the camera 2140. For example, IMU 2170a and IMU 2170b may be mechanically coupled to the frame 2101, directly or indirectly, such that tilt and / or acceleration and / or velocity measurements by these sensors have a predetermined relationship. This relationship may be affected when the frame 2101 becomes distorted. As a non-limiting example, IMU 2170a and IMU 2170b may be mechanically coupled to the frame 2101 such that when there is no distortion of the frame 2101, these sensors measure similar tilt, acceleration, or velocity vectors during movement of the headset. In this non-limiting example, a twist or bend that rotates IMU 2170a relative to IMU 2170b may result in a corresponding rotation of the tilt, acceleration, or velocity vector measurement relative to IMU 2170a relative to the corresponding vector measurement relative to IMU 2170b. The processor may therefore adjust the nominal relative positions and orientations for the DVS camera 2120 and camera 2140 to match the measured relationship between IMU 2170a and IMU 2170b, since IMU 2170a and IMU 2170b are mechanically coupled to the DVS camera 2140 and camera 2120, respectively.
[0270] Other techniques may alternatively or additionally be used to make the initial estimate. In embodiments in which the calibration method 2200 is performed repeatedly during operation of the XR system, the initial estimate may be, for example, the most recently calculated estimate.
[0271] After block 2250, a sub-process begins in which further estimates of the relative positions and orientations of the cameras 2120 and 2140 are made. One of the estimates is selected as the relative positions and orientations of the cameras 2120 and 2140 for computing stereoscopic depth information from images output by the cameras 2120 and 2140. The sub-process may be performed iteratively such that further estimates are made in each iteration until an acceptable estimate is identified. In the example of FIG. 22, the sub-process includes blocks 2270, 2272, 2274, and 2290.
[0272] In block 2270, the processor may calculate an error for the estimated relative orientation of the cameras and the features being compared. In calculating this error, the processor may be configured to estimate how an identified feature should appear or where it should be located in a corresponding image output using the cameras 2120 and 2140 based on the estimated relative orientation of the DVS camera 2120 and the camera 2140 and the estimated location feature being used for calibration. In some embodiments, this estimate may be compared to the appearance or apparent location of the corresponding feature in the image output using each of the two cameras to generate an error for each estimated relative orientation. Such an error may be calculated using linear algebra techniques. For example, the mean squared deviation between the calculated location and the actual location of each of the features in the image may be used as a metric for the error.
[0273] After block 2270, the method 2200 may proceed to block 2272, where a check may be made as to whether the error meets an acceptance criterion. The criterion may be, for example, the overall magnitude of the error or the change in the error between iterations. If the error meets the acceptance criterion, the method 2200 proceeds to block 2290.
[0274] At block 2290, the processor may select one of the estimated relative orientations based on the error calculated at block 2272. The selected estimated relative position and orientation may be the estimated relative position and orientation having the lowest error. In some embodiments, the processor may be configured to select the estimated relative position and orientation associated with this lowest error as the current relative position and orientation of the DVS camera 2120 and the camera 2140. After block 2290, the method 2200 may proceed to block 2299. The method 2200 ends at block 2299, and the selected positions and orientations of the DVS camera 2120 and the camera 2140 may be used to calculate stereoscopic image information based on images formed with the cameras.
[0275] If the error does not meet the acceptance criteria at block 2272, the method 2200 may proceed to block 2274. At block 2274, the estimates used in calculating the error at block 2270 may be updated. The updates may be to the estimated relative positions and / or orientations of the cameras 2120 and 2140. In embodiments where the relative positions of a set of features being used for calibration are estimated, the updated estimates selected at block 2274 may alternatively or additionally include updates to the positions of the feature locations within the set. Such updates may be made according to linear algebra techniques used to solve sets of equations involving multiple variables. As a specific example, one or more of the estimated positions or orientations may be increased or decreased. If the change reduces the calculated error in one iteration of the sub-process, in a subsequent iteration, the same estimated positions or orientations may be further changed in the same direction. Conversely, if the change increases the error, in a subsequent iteration, those estimated positions or orientations may be changed in the opposite direction. The estimated positions and orientations of the cameras and features used in the calibration process may be varied in this manner, either sequentially or in combination.
[0276] Once an updated estimate has been calculated, the subprocess returns to block 2270. A further iteration of the subprocess then begins with the calculation of an error for the estimated relative position. In this manner, the estimated position and orientation are updated until an updated relative position and orientation is selected that provides an acceptable error. However, it should be understood that the processing at block 2272 may apply other criteria for terminating the iterative subprocess, such as completing a certain number of iterations without finding an acceptable error.
[0277] Although method 2200 is described with reference to DVS camera 2120 and camera 2140, similar calibration may be performed for any pair of cameras used for stereoscopic imaging, or for any set of multiple cameras for which relative positions and orientations are desired.
[0278] Example Camera Configurations
[0279] The headset 2100 incorporates components that provide a field of view and lighting field to support multiple functions of the XR system. Figures 23A-23C are exemplary diagrams of the field of view or lighting associated with the headset 2100 of Figure 21, according to some embodiments. The exemplary diagrams each depict the field of view or lighting from different orientations and distances from the headset. Figure 23A depicts the field of view or lighting from an elevated off-axis line of sight at a distance of one meter from the headset. Figure 23A depicts the overlap between the fields of view for the DVS camera 2120 and the camera 2140, and in particular how the DVS camera 2120 and the camera 2140 are angled such that the fields of view 2121 and the fields of view 2141 cross the midline of the headset 2100. In the depicted configuration, the field of view 2141 extends beyond the field of view 2121, forming a peripheral field of view 2160b. As depicted, the illumination fields for emitter 2130a and emitter 2130b largely overlap. In this manner, emitter 2130a and emitter 2130b may be configured to support imaging or depth measurement for objects within central field of view 2150 under low ambient light conditions. FIG. 23B depicts the field of view or illumination from an up-down eye line at a distance of 0.3 meters from the headset. FIG. 23B depicts that overlap of field of view 2121 and field of view 2141 exists at 0.3 meters from the headset. However, in the depicted configuration, field of view 2141 does not extend very far beyond field of view 2121, limiting the extent of peripheral field of view 2160b and demonstrating the asymmetry between peripheral field of view 2160a and peripheral field of view 2160b. FIG. 23C depicts the field of view or illumination from a front-on eye line at a distance of 0.25 meters from the headset. 23C depicts that overlap of field of view 2121 and field of view 2141 exists at 0.25 meters from the headset. However, field of view 2141 is entirely contained within field of view 2121, and thus, in the depicted configuration, peripheral field of view 2160b does not exist at this distance from the headset 2100.
[0280] As can be seen from FIGS. 23A-23C, the overlap of field of view 2121 and field of view 2141 creates a central field of view where stereoscopic imaging techniques can be employed using grayscale images output with camera 2120 and camera 2140, with or without IR illumination from emitters 2130a and emitters 2130b. In this central field of view, color information from camera 2140 may be combined with grayscale image information from camera 2120. In addition, there is a peripheral field of view where there is no overlap, but monocular grayscale image information or color image information is available from one of cameras 2120 or 2140, respectively. Different operations may be performed on the image information obtained for the central and peripheral fields of view, as described herein.
[0281] World model generation
[0282] In some embodiments, image data output by the DVS camera 2120 and / or the camera 2140 may be used to build or update a world model. FIG. 24 is a simplified flowchart of a method 2400 for creating or updating a passable world model, according to some embodiments. As disclosed above with respect to FIG. 21, the XR system may be configured to determine and update the passable world model using a processor. In some embodiments, the DVS camera 2120 may be configured to operate to output image information representative of a detected intensity level at each of a plurality of pixels, or to output image information indicative of pixels for which a change in intensity exceeding a threshold has been detected. In some embodiments, the processor may determine and update this passable world model based on the output of the DVS camera 2120 representative of the detected intensity, which may be used in conjunction with the image information output from the camera 2140 to stereoscopically determine the location of an object within the passable world. In some embodiments, the output of the DVS camera 2120 representative of the detected change in intensity may be used to identify areas of the world model to update based on the change in image information from those locations.
[0283] In some embodiments, the DVS camera 2120 may be configured with a global shutter to output image information reflecting intensity, while the camera 2140 may be configured with a rolling shutter. The processor may therefore implement a compensation routine to compensate for rolling shutter distortion in the images output by the camera 2140. In various embodiments, the processor may determine and update this passable world model without using the emitters 2130a and 2130b. However, in some embodiments, the passable world model may be incomplete. For example, the processor may incompletely determine depths for walls or other flat surfaces. As an additional example, the passable world model may incompletely represent objects with many corners, curved surfaces, transparent surfaces, or large surfaces, such as windows, doors, balls, tables, and the like. The processor may be configured to identify such incomplete information, obtain additional information, and use the additional depth information to update the world model. In some embodiments, emitters 2130a and 2130b may be selectively enabled to collect additional image information from which to build or update a passable world model. In some scenarios, the processor may be configured to perform object recognition within the acquired images, select templates for the recognized objects, and add information to the passable world model based on the templates. In this manner, the wearable display system may improve the passable world model with little or no utilization of power-intensive components such as emitters 2130a and 2130b, thereby extending battery life.
[0284] Method 2400 may be initiated one or more times during operation of the wearable display system. The processor may be configured to create a passable world model when the user first moves to a new environment by turning on the system, walking into another room, etc., or generally when the processor detects a change in the user's physical environment. Alternatively, or in addition, method 2400 may be performed periodically during operation of the wearable display system, or when a significant change in the physical world is detected, or in response to a user input, such as an input indicating that the world model is out of sync with the physical world.
[0285] In some embodiments, all or a portion of the passable world model may be stored, provided by other users of the XR system, or otherwise obtained. Thus, while creation of a world model is described, it should be understood that method 2400 may be used for portions of the world model, along with other portions of the world model that are derived from other sources.
[0286] In block 2405, the processor can implement a compensation routine to compensate for rolling shutter distortion in images output by the camera 2140. As explained above, images acquired by an image sensor with a global shutter, such as the image sensor in the DVS camera 2120, include pixel values acquired at the same time. In contrast, images acquired by an image sensor with a rolling shutter include pixel values acquired at different times. Relative movement of the headset and the environment during acquisition of images by the camera 2140 can introduce spatial distortions into the images. These spatial distortions can affect the accuracy of methods that rely on comparison of images acquired by the camera 2140 and the DVS camera 2120.
[0287] Implementing the compensation routine may include using a processor to compare the image output by the DVS camera 2120 and the image output by the camera 2140. The processor performs this comparison to identify any distortions in the image output by the camera 2140. Such distortions may include distortions in at least a portion of the image. For example, if the image sensor in the camera 2140 is acquiring pixel values row by row from the top of the image sensor to the bottom of the image sensor while the headset 2100 is translating laterally, the appearance of the object or part of the object may be offset in successive rows of pixels by an amount that depends on the speed of the translation and the time difference between acquiring the values of each row. Similar distortions may also occur when the headset is rotated. These distortions may result in an overall distortion in the location and / or orientation of the object or part of the object in the image. The processor may be configured to perform a line-by-line comparison between the image output by the camera 2120 and the image output by the camera 2140 to determine the amount of distortion. The image output by the camera 2140 may then be transformed to remove the distortion (eg, to remove the detected distortion).
[0288] At block 2410, a passable world model may be created. In the illustrated embodiment, the processor may create the passable world model using the DVS camera 2120 and the camera 2140. As described above, in generating the passable world model, the processor may be configured to stereoscopically determine depth information about objects in the physical world using images output from the DVS camera 2120 and the camera 2140 when building the passable world model. In some embodiments, the processor may receive color information from the camera 2140. This color information may be used to distinguish objects or to identify surfaces associated with the same object. The color information may also be used to recognize objects. As disclosed above with respect to FIG. 3A, the processor can create the passable world model by correlating information about the physical world with information about the location and orientation of the headset 2100. As a non-limiting example, the processor can be configured to determine a distance from the headset 2100 to features in a view (e.g., the field of view 2121 and / or the field of view 2141). The processor can be configured to estimate the current location and orientation of the view. The processor can be configured to accumulate such distances along with location and orientation information. By triangulating distances to features obtained from multiple locations and orientations, the location and orientation of the feature in the environment can be determined. In various embodiments, the passable world model can be a combination of raster images, points and descriptors, and polygonal / geometric definitions that describe the location and orientation of such features in the environment. In some embodiments, the distance from the headset 2100 to features in the central field of view 2150 can be determined stereoscopically using a compensation image generated using image data output by the DVS camera 2120 and image data output by the camera 2140. In various embodiments, light field information can be used to supplement or refine this determination.For example, angle of arrival information may be converted, through calculations, into distance information indicating the distance to the object from which the light is being reflected.
[0289] In block 2415, the processor may disable the camera 2140 or reduce the frame rate of the camera 2140. For example, the frame rate of the camera 2140 may be reduced from 30 Hz to 1 Hz. As disclosed above, the color camera 2140 may consume more power than the DVS camera 2120 or a grayscale camera. By disabling the camera 2140 or reducing the frame rate thereof, the processor may reduce power consumption and extend the battery life of the wearable display system. Thus, the processor may disable the camera 2140 or reduce the frame rate of the camera 2140 to conserve power. This lower power state may be maintained until a condition is detected indicating that an update in the world model may be required. Such a condition may be detected based on input, such as over time or from sensors gathering information about the user's surrounding environment or from the user.
[0290] Alternatively, or in addition, once the passable world model is sufficiently complete, the location and orientation of features in the physical environment may be sufficiently determinable using images or image patches acquired using the camera 2120. As a non-limiting example, the passable world model may be identified as sufficiently complete based on the percentage of the space around the user's location that is represented in the model, or based on the amount of new image information that matches the passable world model. With respect to this latter approach, newly acquired images may be associated with locations in the passable world. If features in those images have features that match features identified as landmarks in the passable world model, the world model may be considered complete. The coverage or match rate need not be 100% complete. Rather, a suitable threshold may be applied, such as greater than 95% or greater than 90% coverage of features that match previously identified landmarks, for each criterion. Regardless of how the passable world model is determined to be complete, once completed, the processor may use existing passable world information to refine estimates of the location and orientation of features in the physical world. This process may reflect the assumption that features in the physical world, if applicable, change position and / or orientation slowly compared to the rate at which the processor processes the images output by the DVS camera 2120.
[0291] In block 2420, after creating the passable world model, the processor may identify surfaces and / or objects to update the passable world model with. In some embodiments, the processor may identify such surfaces or objects using grayscale images or image patches output by the DVS camera 2120. For example, once a world model showing a surface at a particular location in the passable world is created in block 2410, the grayscale images or image patches output by the DVS camera 2120 may be used to detect a surface of approximately identical characteristics and determine that the passable world model should be updated by updating the position of that surface in the passable world model. A surface at approximately the same location with approximately the same shape as a surface in the passable world model may be considered to be comparable to that surface in the passable world model, for example, and the passable world model may be updated accordingly. As another example, the position of an object represented in the passable world model may be updated based on the grayscale images or image patches output by the DVS camera 2120. As described herein, the DVS camera 2120 can detect events associated with grayscale images or even image patches. In response to detecting such events, the DVS camera 2120 can be configured to update a passable world model using the grayscale images or image patches.
[0292] In some embodiments, the processor may use light field information obtained from the camera 2120 to determine depth information for objects in the physical world. For example, angle of arrival information may be used to determine the depth information. This determination may be more accurate for objects in the physical world that are closer to the headset 2100. Thus, in some embodiments, the processor may be configured to use the light field information to update only those portions of the passable world model that satisfy a depth criterion. The depth criterion may be based on a maximum distinguishable distance. For example, the processor may be unable to distinguish objects at different distances from the headset 2100 when their distances exceed a threshold distance. The depth criterion may be based on a maximum error threshold. For example, an error in the estimated distance may increase with increasing distance, with a particular distance corresponding to the maximum error threshold. In some embodiments, the depth criterion may be based on a minimum distance. For example, the processor may be unable to accurately determine distance information for objects within a minimum distance from the headset 2100, such as 15 cm. Thus, portions of the world model that are more than 16 cm from the headset may satisfy the depth criterion. In some embodiments, the passable world model may be composed of three-dimensional "bricks" of voxels. In such embodiments, updating the passable world model may include identifying a brick of voxels for updating. In some embodiments, the processor may be configured to determine a viewing frustum. The viewing frustum may have a maximum depth, such as 1.5m. The processor may be configured to identify bricks within the viewing frustum. The processor may then update passable world information for voxels within the identified bricks. In some embodiments, the processor may be configured to use the light field information obtained in step 2450 to update the passable world information for the voxels as described herein.
[0293] The process for updating the world model may differ based on whether the object is in the central or peripheral field of view. For example, updates may be performed for detected surfaces in the central field of view. In the peripheral field of view, updates may be performed only for objects for which the processor has a model such that the processor can verify that any updates to the passable world model are consistent with that object. Alternatively, or in addition, new objects or surfaces may be recognized based on processing of the grayscale images. Even if such processing leads to a less accurate representation of the object or surface than the processing in block 2410, trading off accuracy for faster and lower power processing may lead to a better overall system in some scenarios. Furthermore, lower accuracy information may be periodically replaced by higher accuracy information by periodically repeating the method 2400 to replace portions of the world model generated using only monocular grayscale images with portions generated stereoscopically through the use of the color camera 2140 in combination with the DVS camera 2120.
[0294] In some embodiments, the processor may be configured to determine whether the updated world model satisfies the quality criteria. When the world model satisfies the quality criteria, the processor may continue updating the world model with the camera 2140 disabled or with a reduced frame rate. When the updated world model does not satisfy the quality criteria, the method 2400 may enable the camera 2140 or increase the frame rate of the camera 2140. The method 2400 may also return to step 2410 and recreate the passable world model.
[0295] In block 2425, after updating the passable world model, the processor may identify whether the passable world model contains incomplete depth information. Incomplete depth information can arise in any of a number of ways. For example, some objects do not result in detectable structures in the image. For example, areas in the physical world that are very dark cannot be imaged with sufficient resolution to extract depth information from images obtained with ambient lighting. As another example, the surface of a window or glass table may not appear or be recognized by computerized processing in the visible image. As yet another example, a large uniform surface such as a table surface or a wall may lack sufficient features that can be correlated in two stereoscopic images to enable stereoscopic image processing. As a result, the processor may be unable to determine the location of such an object using stereoscopic processing. In these scenarios, a "hole" exists in the world model, and a process that attempts to use the passable world model to determine the distance to a surface in a particular direction passing through the "hole" will not be able to obtain any depth information.
[0296] When the passable world model does not include incomplete depth information, the method 2400 may return to updating the passable world model using grayscale images or image patches obtained from the DVS camera 2120.
[0297] Following identification of incomplete depth information, the processor controlling method 2400 may take one or more actions to obtain additional depth information. Method 2400 may proceed to block 2431, block 2433, and / or block 2435. In block 2431, the processor may enable emitter 2130a and / or emitter 2130b. As disclosed above, one or more of camera 2120 and camera 2140 may be configured to detect light emitted by emitter 2130a and / or emitter 2130b. The processor may then obtain depth information by causing emitter 2130a and / or 2130b to emit light that may enhance the obtained image of an object in the physical world. When DVS camera 2120 and camera 2140 are sensitive to the emitted light, for example, images output by DVS camera 2120 and camera 2140 may be processed to extract stereoscopic information. Other analysis techniques may alternatively or additionally be used to obtain depth information when emitter 2130a and / or emitter 2130b are enabled. Time-of-flight measurements and / or structured light techniques may alternatively or additionally be used in some embodiments.
[0298] In block 2433, the processor may determine additional depth information from previously obtained depth information. In some embodiments, for example, the processor may be configured to identify an object in an image formed using the DVS camera 2120 and / or the camera 2140 and fill any holes in the passable world model based on a model of the identified object. For example, the process may detect a planar surface in the physical world. The planar surface may be detected using existing depth information obtained using the DVS camera 2120 and / or the camera 2140 or depth information stored in the passable world model. The planar surface may be detected in response to determining that a portion of the world model includes incomplete depth information. The processor may be configured to estimate additional depth information based on the detected planar surface. For example, the processor may be configured to extend the identified planar surface through an area of incomplete depth information. In some embodiments, the processor may be configured to interpolate the missing depth information based on the surrounding portion of the passable world model when extending the planar surface.
[0299] In some embodiments, as an additional example, the processor may be configured to detect an object in a portion of the world model that includes incomplete depth information. In some embodiments, the detection may involve recognizing the object using a neural network or other machine learning tool. In some embodiments, the processor may be configured to access a database of stored templates and select an object template that corresponds to the identified object. For example, when the identified object is a window, the processor may be configured to access a database of stored templates and select a corresponding window template. As a non-limiting example, the template may be a three-dimensional model that represents a class of object, such as a type of window, door, ball, or the like. The processor may construct an instance of the object template based on an image of the object in the updated world model. For example, the processor may scale, rotate, and translate the template to match the detected location of the object in the updated world model. Additional depth information may then be estimated based on a boundary of the constructed template that represents a surface of the recognized object.
[0300] In block 2435, the processor may obtain light field information. In some embodiments, this light field information may be obtained with the image and may include angle of arrival information. In some embodiments, the camera 2120 is configured as a plenoptic camera and may obtain this light field information.
[0301] After blocks 2431, 2433, and / or 2435, the method 2400 may proceed to block 2440. In block 2490, the processor may update the passable world model using the additional depth information obtained in blocks 2431 and / or 2473. For example, the processor may be configured to blend additional depth information obtained from measurements made with active IR illumination into the existing passable world model. Similarly, additional depth information determined from light field information using triangulation, for example, based on angle of arrival information, can be blended into the existing passable world model. As additional examples, the processor may be configured to blend interpolated depth information obtained by extending a detected planar surface into the existing passable world model, or blend additional depth information estimated from the boundary of a configured template into the existing passable world model.
[0302] The information may be blended in one or more ways depending on the nature of the additional depth information and / or the information in the passable world model. Blending may be performed, for example, by adding to the passable world model additional depth information collected for locations where holes exist in the passable world model. Alternatively, the additional depth information may overwrite information at corresponding locations in the passable world model. As yet another alternative, blending may involve selecting between information already in the passable world model and the additional depth information. Such a selection may be based, for example, on selecting depth information that is either already in the passable world model or that in the additional depth information that represents a surface closest to the camera being used to gather the additional depth information.
[0303] In some embodiments, the passable world model may be represented by a mesh of connected points. Updating the world model may be done by computing a mesh representation of the object or surface to be added to the world model and then combining that mesh representation with the mesh representation of the world model. The inventors have recognized and appreciated that performing the operations in this order may require less processing than adding the object or surface to the world model and then computing a mesh for the updated model.
[0304] 24 shows that the world model may be updated at both blocks 2420 and 2440. The processing at each block may be performed in the same manner or in different manners, for example, by generating a mesh representation of the object or surface to be added to the world model and combining the generated mesh with a mesh of the world model. In some embodiments, this merging operation may be performed once for both the object or surface identified at block 2420 and block 2440. Such combined processing may be performed, for example, as described in connection with block 2440.
[0305] In some embodiments, method 2400 may return to block 2420 to repeat the process of updating the world model based on information obtained using DVS camera 2120. The process at block 2420 may be performed on fewer images and smaller images than the process at block 2410 and may therefore be repeated at a higher rate. This process may be performed at a rate less than 10 times per second, such as 3-7 times per second.
[0306] Method 2400 may be repeated in this manner until an exit condition is detected. For example, method 2400 may be repeated for a predetermined period of time, until user input is received, or until a particular type or size change in a portion of the physical world model within the field of view of the camera of headset 2100 is detected. Method 2400 may then end at block 2499. Method 2400 may be started again as new information for the world model is captured at block 2405, including that obtained using a higher resolution color camera. Method 2400 may be terminated and restarted to repeat the process at block 2405 to create a portion of the world model using the color camera at a slower average rate than the rate at which the world model is updated based solely on grayscale image information. The process using the color camera may be repeated, for example, once per second or at a slower average rate.
[0307] Head Pose Tracking
[0308] The XR system may track the position and orientation of the head of a user wearing the XR display system. Determining the user's head pose allows information in the passable world model to be transformed into the frame of reference of the user's wearable display device so that the information in the passable world model may be used in rendering objects on the wearable display device. As head poses are updated frequently, performing head pose tracking using only the DVS camera 2120 may provide power savings, reduced computation, or other advantages. Such tracking may be event-based, as described above with respect to FIGS. 4-16, and may obtain full image and / or image patch data. The XR system may therefore be configured to disable the color camera 2140 or reduce its frame rate as necessary to balance head tracking accuracy against power consumption and computation requirements.
[0309] FIG. 25 is a simplified flowchart of a method 2500 for head pose tracking, according to some embodiments. The method 2500 may include creating a world model, selecting a tracking methodology, tracking head pose using the selected methodology, and evaluating tracking quality. According to the method 2500, the processor may preferentially track head pose using event-based acquisition of image patch data by the DVS camera 2120. If this preferred approach proves inadequate, the processor may track head pose using full frame images output periodically by the DVS camera 2120. If this secondary approach proves inadequate, the processor may stereoscopically track head pose using images output by the DVS camera 2120 and by the camera 2140.
[0310] In block 2510, the processor may create a passable world model. In some embodiments, the processor may be configured to create the passable world model as described above with respect to blocks 2405-2415 of method 2400. For example, the processor may be configured to obtain images from the DVS camera 2120 and the camera 2140. In some implementations, the processor may compensate for rolling shutter distortion in the camera 2140. The processor may then use the images from the DVS camera 2120 and the compensated images from the camera 2140 to stereoscopically determine depths for features in the physical world. Using these depths, the processor may create the passable world model. After creating the passable world model, in some embodiments, the processor may be configured to disable the camera 2140 or reduce its frame rate. By disabling the camera 2140 or reducing its frame rate after generating the passable world model, the XR system may reduce power consumption and computing requirements.
[0311] The processor may select features from the world model that correspond to the constant features, for example as described above in connection with FIG. 13. Image information indicating the location of the constant features relative to a camera mounted on a device worn on the user's head may be used to calculate the change in position of the user's head relative to the world model. According to some embodiments, the processor may select a tracking methodology for tracking the relative positions of the constant features that both meets a quality criterion and requires a low amount of computation relative to other tracking methods.
[0312] In block 2520, the processor may select a tracking methodology. The processor may preferentially select head pose tracking using events detected by the DVS camera 2120. The processor may continue to use the preferred approach while the head pose tracking satisfies a tracking quality criterion. For example, the difficulty of tracking a head pose may depend on the location and orientation of the user's head and the contents of the passable world model. Thus, in some instances, the processor may be unable or unable to track the head pose using only the asynchronously obtained image data output by the DVS camera 2120.
[0313] If the processor determines that the head pose tracking provided by the preferred approach does not satisfy the tracking quality criteria, the processor may select a secondary approach to track the head pose. For example, the processor may select to track the head pose using indications of events output by the DVS camera 2120 in combination with color information acquired with the camera 2140. If the processor determines that this secondary approach does not satisfy the tracking quality criteria, the processor may select a tertiary approach to track the head pose. For example, the processor may select to track the head pose stereoscopically using images output by the DVS camera 2120 and by the camera 2140. The processor may continue to use the selected approach while the head pose tracking satisfies the tracking quality criteria. Alternatively, the processor may revert to the more preferred approach after a predetermined duration, time, or number of head pose updates, or in response to satisfaction of the criteria.
[0314] In each approach, image information may be obtained for the entire field of view for each camera used, however, as described above in connection with Figure 13, image information may be collected for only a patch of the image that corresponds to the portion containing the feature being tracked.
[0315] FIG. 25 illustrates a first tracking methodology implemented in blocks 2530a and 2540a. In block 2530a, the processor may enable DVS functionality of the DVS camera 2120 when the functionality is not already enabled. The functionality may be enabled by setting a threshold change in intensity associated with the movement of stationary features selected from the world model. In some embodiments, a patch incorporating those features may also be set. In block 2540a, the processor may track head pose using patch data obtained in response to events detected by the DVS camera 2120. In some embodiments, the processor may be configured to calculate real-time or near real-time user head pose from the patch data.
[0316] A secondary tracking methodology is illustrated in blocks 2530b and 2540b. In this example, color image information may be used in combination with event information to track the relative positions of features. In block 2530b, the processor may enable the camera 2140 if the camera 2140 is not already enabled. The camera 2140 may be enabled to provide images at a rate that is the same as or slower than the rate at which head pose updates are provided. For example, head pose updates may be provided through the use of asynchronous event data at an average rate of 30-60 Hz. The camera 2140 may be enabled to provide frames at a rate less than 30 Hz, such as 5-10 Hz.
[0317] Color information may be used to increase the accuracy with which stationary features are tracked. Color information may be used, for example, to calculate an updated location of the tracked feature, which may be determined more accurately than if grayscale events were used alone. As a result, the location of the tracked patch may be updated or changed to include other features. Alternatively, or in addition, information from the camera 2140 may be used to identify alternative features for tracking. As yet a further alternative, color information may allow the relative position of the camera to be calculated based on analyzing surfaces, edges, or features larger than those traced using the DVS camera 2120.
[0318] The tertiary tracking methodology is illustrated in blocks 2530c and 2540c. In this example, the tertiary tracking may be based on stereoscopic information. In block 2530b, the processor may disable the DVS functionality of the DVS camera 2120 such that the camera 2120 outputs intensity information rather than event information representing changes in intensity. In block 2540b, the processor may track head pose using images periodically output by the DVS camera 2120, which may be full frame images or images within the specific patch being tracked. In block 2540c, the processor may track head pose using stereoscopic image data obtained from images output by the DVS camera 2120 and the camera 2140. For example, the processor may be configured to determine depth information from the stereoscopic image data. In some embodiments, the processor may be configured to calculate real-time or near real-time user head pose from these images.
[0319] In block 2550, the processor may evaluate the tracking quality according to a tracking criterion. The tracking criterion may depend on the stability of the estimated head pose, the noisiness of the estimated head pose, the consistency of the estimated head pose with a world model, or similar factors. As a specific example, the calculated head pose may be compared to other information that may indicate inaccuracies, such as the output of an inertial measurement unit or a model of the range of motion of the human head, so that errors in the head pose may be identified. The specific tracking criterion may vary based on the tracking methodology used. For example, in a methodology that uses event-based information, the correspondence between the location of features as indicated by the event-based output compared to the location of the corresponding feature in a full frame image may be used. Alternatively or in addition, the visual distinctiveness of a feature relative to its surroundings may be used as a tracking criterion. For example, a tracking criterion for an event-based methodology may indicate poor tracking when the field of view is filled with one or more objects that make it difficult to identify the movement of the specific feature. The percentage of the field of view that is occluded is an example of a criterion that may be used. For example, a threshold such as above 40% may be used as an indication to switch from using an image-based methodology for head pose tracking. As a further example, reprojection error may be used as a measure of head pose tracking quality. Such a criterion may be calculated by matching features in the image being acquired with a previously determined passable world model. The location of the feature in the image may be related to a location in the passable world model using a geometric transformation calculation based on the head pose. For example, a deviation, expressed as a mean square error between the calculated location and the feature in the passable world model, may indicate an error in the head pose, such that the deviation may then be used as a tracking criterion.
[0320] In some embodiments, the processor may be configured to calculate an error for the estimated head pose based on the world model. In calculating this error, the processor may be configured to estimate how the world model (or features in the world model) should appear based on the estimated head pose. In some embodiments, this estimate may be compared to the world model (or features in the world model) to generate an error for the estimated head pose. Such an error may be calculated using linear algebra techniques. For example, the mean squared deviation between the calculated location and the actual location of each of the features in the image may be used as a metric for the error. This metric may in turn be used as a measure of head pose tracking quality.
[0321] After evaluating the head pose tracking quality, method 2500 may return to block 2520, where the processor may use the measured head pose tracking quality to select a tracking methodology. In scenarios where the tracking quality is poor, such as below a threshold, an alternative tracking methodology may be selected.
[0322] Method 2500 may be repeated in this manner until an exit condition is detected. For example, method 2500 may be repeated for a predetermined period of time until user input is received or until a particular type or magnitude of change in the portion of the physical world model within the field of view of the camera of headset 2100 is detected. Method 2500 may then end at block 2599.
[0323] Method 2500 may be started again so that new information for the world model, including that obtained using a higher resolution color camera, is captured at block 2510. Method 2500 may be terminated and restarted to repeat the processing at block 2510 using the color camera to create a portion of the world model at a slower average rate than the rate at which head pose tracking is performed at blocks 2520-2550. The processing using the color camera may be repeated, for example, once per second or at a slower average rate.
[0324] Other tracking methodologies may be used instead of or in addition to the tracking methodologies described above as examples. In some embodiments, the processor may be configured to calculate real-time or near real-time user head pose from image information, which may include grayscale images and / or light field information such as angle of arrival information. Alternatively, or in addition, in scenarios where an image-based head pose tracking methodology has an unacceptable quality metric, a "dead reckoning" methodology may be selected, in which movement of the user's head as measured by an inertial measurement unit may be used to calculate the head pose.
[0325] Object Tracking
[0326] As described above, the processor of the XR system may track objects in the physical world to support realistic rendering of virtual objects relative to the physical objects. Tracking has been described in connection with movable objects, such as, for example, the hands of a user of the XR system. For example, the XR system may track objects in the central field of view 2150, the peripheral field of view 2160a, and / or the peripheral field of view 2160b. Rapidly updating the position of the movable objects allows the virtual objects to be rendered realistically, since the rendering may reflect occlusion of the physical objects by the virtual objects or vice versa or interactions between virtual objects within the physical objects. In some embodiments, for example, updates regarding the location of the physical objects may be calculated at an average rate of at least 10 times / second, such as about 30 times / second, and in some embodiments at least 20 times / second. When the tracked object is the user's hand, tracking may enable gestural control by the user. For example, certain gestures may correspond to commands to the XR system.
[0327] In some embodiments, the XR system may be configured to track objects that have features that provide high contrast when imaged with an image sensor that is sensitive to IR light. In some embodiments, objects with high contrast features may be created by adding markers to the object. For example, a physical object may be equipped with one or more markers that appear as high contrast areas when imaged with IR light. The markers may be passive markers that are highly reflective or highly absorptive of IR light. In some embodiments, at least 25% of the light over a frequency range of interest may be absorbed or reflected. Alternatively or additionally, the markers may be active markers that emit IR light, such as IR LEDs. By tracking such features, for example with a DVS camera, information that precisely represents the location of the physical object may be rapidly determined.
[0328] Similar to head pose tracking, tracked object positions are updated frequently, and therefore performing object tracking using only the DVS camera 2120 may provide power savings, reduce computational requirements, or provide other advantages. The XR system may therefore be configured to disable or reduce the frame rate of the color camera 2140 as necessary to balance object tracking accuracy against power consumption and computational requirements. Additionally, the XR system may be configured to track objects asynchronously in response to events generated by the DVS camera 2120.
[0329] FIG. 26 is a simplified flowchart of a method 2600 for object tracking, according to some embodiments. According to the method 2600, the processor can perform object tracking differently depending on the field of view containing the object and the value of the tracking quality criterion. Furthermore, the processor may or may not use light field information depending on whether the tracked object satisfies the depth criterion. Other criteria, such as available battery power, or the operation of the XR system being performed and the need for those operations to track object location or to track object location with high accuracy, may alternatively or additionally be applied by the processor to dynamically select the object tracking methodology.
[0330] Method 2600 may begin at block 2601. In some embodiments, the camera 2140 may be disabled or have a reduced frame rate. The processor may disable the camera 2140 or reduce the frame rate of the camera 2140 to reduce power consumption and improve battery life. In various embodiments, the processor may track an object in the physical world (e.g., a user's hand). The processor may be configured to predict a next location of the object or a trajectory of the object based on one or more previous locations of the object.
[0331] After starting at block 2601, the method 2600 may proceed to block 2610. In block 2610, the processor may determine a field of view (e.g., field of view 2121, field of view 2141, peripheral field of view 2160a, peripheral field of view 2160b, or central field of view 2150) that contains the object. In some embodiments, the processor may base this determination on a current location of the object (e.g., whether the object is currently within the central field of view 2150). In various embodiments, the processor may base this determination on an estimate of the object's location. For example, the processor may determine that an object that is away from the central field of view 2150 may enter the peripheral field of view 2160a or the peripheral field of view 2160b.
[0332] In block 2620, the processor may select an object tracking methodology. According to the method 2600, when an object is within the field of view 2121 (e.g., within the peripheral field of view 2160a or the central field of view 2150), the processor may preferentially select object tracking using the DVS camera 2120. Additionally, the processor may preferentially select event-based asynchronous object tracking.
[0333] In some embodiments, patch tracking as described above may be used, with one or more patches established to encompass features of the object being tracked, as described above. Patch tracking may be used for some or all of the object tracking methodologies and for some or all of the cameras. Patches may be selected to encompass the estimated location of the tracked object within the field of view.
[0334] As described above in connection with FIG. 25 and head pose tracking, the processor may dynamically select an appropriate tracking methodology for object tracking. A methodology may be selected to provide a suitable tracking quality that requires low processing overhead compared to other methodologies. Thus, if event-based asynchronous object tracking does not satisfy the object tracking criteria, the processor may use another methodology. In the example of FIG. 26, four methodologies are illustrated, shown in blocks 2640a, 2640b, 2640c, and 2640d. The methodologies are ordered, and the method 2600 will select the first methodology in the order that satisfies the tracking quality criteria. The methodologies may be ordered to reflect tradeoffs in accuracy, latency, and power consumption, for example. The methodology with the shortest latency may be ordered first, for example, and methodologies with longer latency or more power consumption will be lower in the ordering.
[0335] In block 2630a, the processor may enable DVS functionality of the DVS camera 2120 if the functionality is not already enabled. In block 2640a, the processor may track an object using an event detected by the DVS camera 2120. In some embodiments, the DVS camera 2120 may be configured to restrict image acquisition to a patch in the image sensor that encompasses the location of the object in the field of view 2121. A change in image data in the patch (e.g., caused by movement of the object) may trigger an event. In response to the event, the DVS camera 2120 may acquire image data for the patch and update the position and / or orientation of the object based on the acquired patch image data.
[0336] In some embodiments, the event may indicate a change in intensity. For example, as described above in conjunction with FIGS. 12 and 13, the change in intensity may be tracked to track the movement of the object. In some embodiments, the DVS camera 2120 may be a plenoptic camera or may be configurable to operate as such. When the object satisfies a depth criterion, the processor may be configured to obtain angle of arrival information in addition or alternatively. The depth criterion may be the same as or similar to the depth criterion described above with respect to block 2420 of method 2400. For example, the depth criterion may relate to a maximum error rate or a maximum distance beyond which the processor cannot distinguish different distances between objects. The depth information may thus be used to determine the location of the object and / or a change in the location of the object. The use of such plenoptic image information may be a further methodology that may be part of the methodology described herein or may be used in conjunction with other methodologies.
[0337] A second methodology is shown in blocks 2630b and 2640b. In block 2630b, the processor may enable the camera 2140 when the camera 2140 is not already enabled. As described above in connection with block 2530b, the color camera may be operated to obtain color images at a relatively low average rate. Color information may be used primarily in block 2640b in conjunction with events output by the DVS camera 2120 and information from the color images to better identify features or their locations for tracking.
[0338] The tertiary methodology is illustrated in blocks 2630c and 2640c. In block 2630c, the processor may enable the camera 2140 when the camera 2140 is not already enabled. The camera 2140 is described as being capable of outputting color image information. For the tertiary methodology, color image information may be used, but in some embodiments or scenarios, only intensity information may be obtained from the camera 2140 or only intensity information may be processed. The processor may also increase the frame rate of the camera 2140 to a rate sufficient for object tracking (e.g., a frame rate of 40 Hz to 120 Hz). In some embodiments, this rate may match the sampling frequency of the DVS camera 2120. The processor may also disable the DVS functionality of the DVS camera 2120 when this functionality is not already disabled. In this configuration, the output of the DVS camera 2120 may represent intensity information. In block 2640c, the processor may track the object using stereoscopic image data obtained from images output by the DVS camera 2120 and the camera 2140.
[0339] A fourth methodology is illustrated in blocks 2630d and 2640d. In block 2630d, the processor may enable a camera when not already enabled. The other camera may be disabled. The enabled camera may be the DVS camera 2120 or the camera 2140. The DVS camera 2120, if used, may be configured to output image intensity information. The camera 2140, when enabled, may be enabled to output color information or simply grayscale intensity information. The processor may also increase the frame rate of the enabled camera to a rate sufficient for object tracking (e.g., a frame rate between 40 Hz and 120 Hz). In block 2640d, the processor may track objects using images output by the enabled camera.
[0340] In block 2650, the processor may evaluate the tracking quality. One or more of the metrics described above in connection with block 2550 may be used in block 2650. However, in block 2650, those metrics would be applied to tracking features on the object rather than tracking stationary features in the environment.
[0341] After evaluating the object tracking quality, method 2600 may return to blocks 2610 and 2620, where the processor may again determine a field of view that encompasses the object (block 2610) and then use the determined field of view and the measured object tracking quality to select an object tracking methodology (block 2620). In the illustrated embodiment, the first methodology in the order that provides a quality that exceeds a threshold associated with suitable performance is selected.
[0342] The method 2600 may be repeated in this manner until an exit condition is detected. For example, the method 2600 may be repeated for a predetermined period of time until a user input is received or until the tracked object exits the field of view of the XR device. The method 2600 may then end at block 2699.
[0343] The method 2600 may be used to track any object, including a user's hand. However, in some embodiments, different or additional actions may be performed in tracking a user's hand. FIG. 27 is a simplified flowchart of a method 2700 of hand tracking, according to some embodiments. The object tracked in the method 2700 may be a user's hand. In various embodiments, the XR system may perform hand tracking using image data or images acquired from the DVS camera 2120 and / or images acquired from the camera 2140. These cameras may be configurable to operate in one of a number of modes. For example, one or both may be configured to obtain image patch data. Alternatively or in addition, the DVS camera 2140 may be configured to output events, as described above with respect to FIGS. 4-16. Alternatively or in addition, the camera 2140 may be configured to output color information or simply grayscale intensity information. Furthermore, the camera 2140 may be configured to output plenoptic information, as described above with respect to FIGS. 16-20. The DVS camera 2120 may similarly be configured to output plenoptic information in some embodiments. Any combination of these cameras and capabilities may be selected to generate information regarding hand tracking.
[0344] Similar to the object tracking described in connection with FIG. 26, the processor may be configured to select a hand tracking approach based on the determined location of the hand and an assessment of the quality of hand tracking provided by the selected approach. For example, stereoscopic depth information may be acquired when the hand is within the central field of view 2150. As another example, the plenoptic information may have sufficient resolution only for tracking of the hand within a certain angular range (e.g., + / - 20 degrees) relative to the center of the field of view of the plenoptic camera. Thus, techniques that rely on stereoscopic or plenoptic image information may be used only when the hand is detected to be within an appropriate field of view.
[0345] If necessary, the XR system can enable camera 2140 or increase its frame rate to enable tracking within field of view 2160b. In this manner, the wearable display system may be configured in this configuration to provide adequate hand tracking using a reduced number of available cameras, allowing for reduced power consumption and increased battery life.
[0346] The method 2700 may be implemented under the control of a processor of the XR system. The method may be initiated in response to detection of an object to be tracked, such as a hand, as a result of analysis of an image acquired with any one of the cameras on the headset 2100. The analysis may involve recognizing the object as a hand based on a region of the image having photometric properties that are characteristic of a hand. Alternatively, or in addition, depth information acquired based on stereoscopic image analysis may be used to detect the hand. As a specific example, the depth information may indicate that an object is present having a shape that matches a 3D model of a hand. Detecting the presence of the hand in this manner may also involve setting parameters of the hand model to match the orientation of the hand. In some embodiments, such a model may also be used for high-speed hand tracking by using photometric information from one or more grayscale cameras to determine how the hand has moved from its original position.
[0347] Other trigger conditions may initiate method 2700, such as the XR system performing an action that involves tracking an object, such as rendering a virtual button that the user is likely to attempt to press with the user's hand, such that the user's hand is expected to enter the field of view of one or more cameras. Method 2700 may be repeated at a relatively high rate, such as 30-100 times per second, e.g., 40-60 times per second. As a result, updated position information about the tracked object may be made available with low latency for processing to render a virtual object that interacts with the physical object.
[0348] After starting at block 2701, the method 2700 may proceed to block 2710. In block 2710, the processor may determine potential hand locations. In some embodiments, the potential hand locations may be locations of detected objects in the acquired image. In embodiments where hands are detected based on matching depth information to a 3D model of a hand, that same information may be used as an initial hand position in block 2710.
[0349] In some embodiments, the image information used to construct the hand model may be dynamically selected. After block 2710, method 2700 may proceed to block 2720. In block 2720, the processor may select a processing approach for constructing the hand model. The selection may depend on the potential locations of the hand and whether a hand model has already been constructed. In some embodiments, the selection may further depend on the desired accuracy and / or quality of the hand model and / or available processing computer power given other tasks being performed by the processor. As an example, when the hand model is initially constructed in response to detection of the hand, more extensive, but more accurate, image information may be used at the expense of additional processing. For example, stereoscopic information may be used to initially construct the model. Thereafter, the model itself provides information about the position of the hand since there are limitations to the manner in which a human hand may move. Thus, as the XR system operates, processing to reconstruct the hand model and account for hand motion may be optionally performed using less comprehensive, but more quickly processed, image information, or when more comprehensive image information is unavailable. For example, when an object is within the central field of view 2150 and robust hand tracking or fine hand detail is required, or if other approaches prove inadequate, the processor may perform hand tracking stereoscopically using images output by the DVS camera 2120 and by the camera 2140. Alternatively, when an object is not within the central field of view 2150 or robust hand tracking or fine hand detail is not required, the processor may be configured to preferentially construct a hand model using other image information, such as monocular color or grayscale images. The processor may be configured to construct a hand model using images obtained from the DVS camera 2120 when an object is within the peripheral field of view 2160a. The processor may be configured to construct a hand model using images obtained from the camera 2140 when an object is within the peripheral field of view 2160b.In some embodiments, the processor may be configured to use the lowest power consuming method, for example, that still meets quality standards.
[0350] 27 illustrates four approaches for gathering information to construct a hand model, shown as four parallel paths through the flowchart, including paths through blocks 2740a, 2740b, 2740c, and 2740d.
[0351] In the first path, at block 2730a, when a potential hand is within the central field of view 2150 and hand tracking robustness or fine hand details are required, the processor may enable the camera 2140 if this camera is not already enabled. This path may also be selected for the initial construction of the hand model at block 2720. At block 2730a, the processor may also increase the frame rate of the camera 2140 to a rate sufficient for hand tracking (e.g., a frame rate of 40 Hz to 120 Hz). In some embodiments, this rate may match the sampling frequency of the DVS camera 2120. The processor may also disable the DVS functionality of the DVS camera 2120 if this functionality has not already been disabled. As a result, both cameras may provide image information representing intensity. This image information may be grayscale or, alternatively, may include color information for cameras that support color imaging.
[0352] At block 2740a, the processor may obtain depth information for the potential hand. The depth information may be obtained based on stereoscopic image analysis, in which a distance between a camera collecting image information from and a segment or feature of the potential hand may be calculated. The processor may, for example, select a feature within the central field of view and determine depth information for the selected feature.
[0353] In some embodiments, the selected features may represent different segments of the human hand as defined by bones and joints. Feature selection may be based on matching image information to a model of the human hand. Such matching may be done heuristically, for example. The human hand may be represented by a finite number of segments, e.g., 16, and points in the image of the hand may be mapped to one of those segments such that features on each segment may be selected. Alternatively, or in addition, such matching may use deep neural nets or classification / decision forests to apply a series of yes / no decisions in the analysis to identify different parts of the hand and select features that represent different parts of the hand. The matching may, for example, identify whether a particular point in the image belongs to the palm area, the back of the hand, the non-thumb finger, the thumb, the fingertip, and / or the knuckles. Any suitable classifier may be used for this analysis stage. For example, a deep learning module or neural network mechanism may be used instead of or in addition to a classification forest. In addition, a regression forest (e.g., using a Hough transform, etc.) may be used in addition to a classification forest.
[0354] Regardless of the particular number of features selected and the technique used to select those features, after block 2740a, the method 2700 may proceed to block 2750a. In block 2750a, the processor may construct a hand model based on the depth information. In some embodiments, the hand model may reflect structural information about a human hand, for example, representing each of the bones in the hand as segments within the hand and each joint to define a range of likely angles between adjacent segments. By assigning a location to each of the segments in the hand model based on the depth information of the selected features, information about the position of the hand may be provided for subsequent processing by the XR system.
[0355] Regardless of how the 3D hand model is updated, the updated model may be refined based on photometric image information. The model may be used, for example, to generate a projection of the hand to represent how an image of the hand is expected to appear. The expected image may be compared to photometric image information acquired using an image sensor. The 3D model may then be adjusted to reduce errors between the expected and acquired photometric information. The adjusted 3D model then provides an indication of the position of the hand. As this update process is repeated, the 3D model provides an indication of the position of the hand as it moves.
[0356] In some embodiments, the processing at blocks 2740a and 2750a may be performed iteratively, where the selection of features for which depth information is collected is refined based on the configuration of the hand model. The hand model may include shape and motion constraints that the processor may be configured to use to refine the selection of features representing the components of the hand. For example, when a feature selected to represent a segment of the hand indicates a position or motion of that section that violates the constraints of the hand model, a different feature may be selected to represent that segment.
[0357] In block 2750a, the processor may select features in the patch or image output by the DVS camera 2120 or camera 2140 that represent the structure of the human hand. Such features may be identified heuristically or using AI techniques. For example, features may be selected heuristically by representing the human hand by a finite number of segments and mapping points in the image to individual ones of those segments so that a feature on each segment may be selected. Alternatively, or in addition, such matching may use a deep neural network or a classification / decision forest to apply a series of yes / no decisions in the analysis to identify different parts of the hand and select features that represent the different parts of the hand. Any suitable classifier may be used for this analysis stage. For example, a deep learning module or neural network mechanism may be used instead of or in addition to a classification forest. Additionally, a regression forest (e.g., using a Hough transform, etc.) may be used in addition to a classification forest. The processor may attempt to match the selected features and the motion of those selected features per image to a model of the hand without the benefit of depth information. This match may result in information that is less robust or may be less accurate than that generated in block 2750a, but the information identified based on the monocular information may still provide useful information regarding the operation of the XR system.
[0358] In block 2750a, the processor may also evaluate the hand model. This evaluation may depend on the completeness of the match between the selected features and the hand model, the stability of the match between the selected features and the hand model, the noisiness of the estimated hand location, the consistency between the location and orientation of the detected features and the constraints imposed by the hand model, or similar factors. In some embodiments, the processor may determine where the selected features should appear based on the hand model. As a specific example, the processor may parameterize a generic hand model and then check the photometric consistency of the model's edges and compare those edges with the edges as detected in the acquired image of the hand. In some embodiments, this estimate may be compared to the estimated locations and orientations of the selected features to generate an error for the hand model. Such an error may be calculated using linear algebraic techniques. For example, the mean squared deviation between the calculated location and the actual location of each of the multiple selected features in the image may be used as a metric for the error. This metric may then be used as a measure of the hand model quality.
[0359] After matching the image portion to the portion of the hand model in block 2750a, the model may be used in one or more operations performed by the XR system. As an example, the model may be used as an indication of the location of an object in the physical world to directly render a virtual object. In such an embodiment, the processing in block 2760 may be optionally omitted. Alternatively, or in addition, the model may be used to determine whether the user has made a gesture with the hand, such as a gesture indicating a command or interaction with a virtual object. Thus, the processor may recognize a hand gesture in block 2760 using the determined hand model information. This gesture recognition may be performed using the hand tracking method described in U.S. Patent Publication No. 2016 / 0026253 (which teaches related to tracking hands and using information about hands obtained from image information in an XR system, and is incorporated herein by reference in its entirety).
[0360] In some embodiments, gestures may be recognized without stereoscopic depth information. Gestures may be recognized, for example, based on a hand model constructed based on monocular image information. Thus, in some embodiments, gesture recognition may even be performed with respect to the hand in peripheral vision 2160a and 2160b when stereoscopic depth information is unavailable. Alternatively, or in addition, monocular image information may be used when hand tracking is performed for gesture recognition rather than other functions (e.g., rendering virtual objects to appear to interact realistically with the user's hands) that may require a more precise determination of the hand location. In such embodiments, different cameras may be enabled and / or used to collect image information in blocks 2730a and 2740b.
[0361] In some embodiments, successive iterations of the hand tracking process may be performed as the system operates. Gestures may be identified, for example, by successive determinations of hand position. Alternatively, or in addition, successive iterations may be performed such that a constructed model of the user's hand matches the actual hand position as the hand position changes. The same or different imaging techniques may be used in each iteration.
[0362] In some embodiments, a source of image information may be selected for iteration at block 2720 based on one or more factors, including the application consisting of a hand model and / or the quality of tracking performed using a particular tracking methodology. The quality of tracking may be determined, for example, using an approach such as described above in connection with block 2650 (FIG. 26), and the selection of a technique may also be made to require a minimum amount of processing, power consumption, and other computing resources, while achieving a desired quality metric, as also described above.
[0363] Thus, Figure 27 shows that once an iteration has been performed, the method 2700 returns to block 2720 where a processing approach may be selected for further iterations. These alternative approaches may use image information instead of or in addition to stereoscopic depth information. At each iteration, the 3D model of the hand may be updated to reflect the hand's movements.
[0364] An approach that requires less processing is to track the hand position based on event information. Such an approach may be selected in block 2720 in a scenario where a suitable accuracy of the hand model has already been calculated, such as through the use of stereoscopic image information in a previous iteration. Such an approach may be implemented by branching to block 2730b. In block 2730b, the processor may enable DVS functionality of the DVS camera 2120 when the functionality is not already enabled. In block 2740b, the processor may obtain image data in response to an event detected by the DVS camera 2120. In some embodiments, the DVS camera 2120 may be configured to obtain information about a patch in the image sensor that encompasses the location of a potential hand in the field of view 2121. A change in the image data in the patch (e.g., caused by a movement of the potential hand) may trigger an event. In response to the event, the DVS camera 2120 may obtain image data. This information about the user's hand movement may then be used to update the hand model in block 2750b. The updates may be performed using techniques such as those described above with respect to block 2750a. Such updates may take into account other information, including previously calculated positions of the hand as indicated by the model and constraints on human hand motion.
[0365] Once the hand model has been updated, in block 2760 it may be used by the system in the same manner as the initial hand model calculated in block 2750a, including to determine interactions between virtual objects and the user's hands and / or recognize gestures.
[0366] In some scenarios, event-based image information may be inadequate even to update the hand model. Such a scenario may occur, for example, when the user's hand fills the field of view of the DVS camera 2120. In such a scenario, rather than event information, the hand model may be updated based on color information, such as using color information from the camera 2140. Such an approach may be implemented by branching to block 2730d when the processing in block 2720 detects that the images in the field of view of the DVS camera 2120 have intensity variation below a predetermined threshold or other characteristics indicating a lack of features distinct enough for event-based tracking. In block 2730d, the processor may disable the DVS functionality of the DVS camera 2120 if this functionality is not already disabled. The camera 2140 may be enabled to obtain color information.
[0367] In block 2740d, the processor may use the color image information to identify the user's hand movements. As described above, this information about the user's hand movements may then be used to update the hand model in block 2750b.
[0368] In some scenarios, color information may be unavailable or unnecessary for tracking hand movements. For example, when the hand is within the peripheral field of view 2160a, color information may be unavailable, but intensity image information acquired using a DVS camera 2120 operating in a mode in which DVS functionality is disabled may be available and preferred. Alternatively, in some scenarios where event-based tracking produces a quality metric below a threshold, tracking using grayscale image information may produce a quality metric above a threshold of suitability. Such an approach may be implemented by branching to block 2730c. In block 2730c, the processor may enable the camera 2140 when it is not already enabled. The processor may also increase the frame rate of the camera 2140 to a rate sufficient for hand tracking (e.g., a frame rate of 40 Hz to 120 Hz).
[0369] In block 2740d, the processor may obtain image data using the camera 2140. In other scenarios, the DVS camera 2120 may be configured to collect intensity information and may be used to collect monocular image information in place of the camera 2140. Regardless of which camera is used, full frame information may be used, or patch tracking may be used to reduce the amount of image information processed, as described above.
[0370] FIG. 27 illustrates how processing at block 2720 selects between three alternative approaches for tracking hands. The approaches may be ordered according to the degree to which they satisfy one or more criteria, such as low processing or low power consumption. Processing at block 2720 may select a processing approach by selecting the first approach in the order that is operable in the detected scenario (e.g., hand location) and produces a quality metric that meets a threshold. Different or additional processing techniques may be included. For example, for an XR system with a plenoptic camera that provides image information indicative of depth, an approach may be based on the use of that depth information alone or in combination with any other data source. As another variation, hand features may be tracked using color-assisted DVS tracking, as described above in connection with block 2640b (FIG. 26).
[0371] Regardless of the approach selected or the set of approaches from which such selection is made, the hand model may be updated and used for XR functions in block 2760, such as to render a virtual object that interacts with the user's hand or to detect gestures. After block 2760, method 2700 may end in block 2799. However, it should be understood that hand tracking may occur continuously during operation of the XR system or may occur during an interval during which the hand is within the field of view of one or more cameras. Thus, once one iteration of method 2700 is completed, another iteration may be performed and the process may be performed over the interval during which hand tracking is performed. In some embodiments, information used in one iteration may be used in subsequent iterations. In various embodiments, for example, the processor may be configured to estimate an updated location of the user's hand based on a previously detected hand location. For example, the processor may estimate a location where the user's hand will next be based on a previous location and the velocity of the user's hand. Such information may be used to narrow down the amount of image information that is processed to detect the location of the object, as described above in connection with the patch tracking technique.
[0372] Having thus described several aspects of several embodiments, it is to be appreciated that various alterations, modifications, and improvements will readily occur to those skilled in the art.
[0373] As an example, the embodiments are described in relation to an augmented (AR) environment, it being understood that some or all of the techniques described herein may also be applied in MR environments, or more generally, other XR environments.
[0374] Also described are embodiments of an image array in which a patch is applied to the image array to control the selective output of image information for a movable object. It should be understood that there may be more than one movable object in a physical embodiment. Furthermore, in some embodiments, it may be desirable to selectively obtain frequent updates of image information in areas other than where the movable object is located. For example, a patch may be configured to selectively obtain image information for an area of the physical world where a virtual object is to be rendered. Thus, some image sensors may be capable of selectively providing information about more than one patch, with or without circuitry for tracking the trajectory of those patches.
[0375] As a still further example, the image array is described as outputting information related to the magnitude of the incident light. The magnitude may be a representation of power across a spectrum of light frequencies. The spectrum may be relatively broad capturing energy at frequencies corresponding to any color of visible light, such as in a black and white camera. Alternatively, the spectrum may be narrow, corresponding to a single color of visible light. Filters to limit the light incident on the image array to light of a particular color may be used for this purpose. If pixels are limited to receive light of a specific color, different pixels may be limited to different colors. In such an embodiment, the outputs of pixels that are sensitive to the same color may be processed together.
[0376] A process has been described for setting a patch in an image array and then updating the patch for an object of interest. This process may be performed for each movable object, for example, as it enters the field of view of the image sensor. The patch may be released when the object of interest exits the field of view such that the patch is no longer tracked or image information is not output for the patch. It should be understood that from time to time, the patch may be updated, such as by determining the location of the object associated with the patch and setting the position of the patch to correspond to that location. Similar adjustments can be made to the calculated trajectory of the patch. A motion vector for the object and / or a motion vector of the image sensor may be calculated from other sensor information and used to reset values programmed into the image sensor or other components for patch tracking.
[0377] For example, the location, motion, and other characteristics of the object may be determined by analyzing the output of a wide-angle video camera or a pair of video cameras with stereoscopic information. Data from these other sensors may be used to update the world model. In conjunction with the updates, patch position and / or trajectory information may be updated. Such updates may occur at a slower rate than the patch positions are updated by the patch tracking engine. The patch tracking engine may calculate new patch positions at a rate of, for example, about 1-30 times per second. Patch position updates based on other information may occur at a slower rate, such as once per second to about once per 30 second interval.
[0378] As yet a further example of the variant, FIG. 2 shows a system with a head-mounted display separate from the remote processing module. Image sensors as described herein may lead to a compact design of the system. Such sensors generate less data, which in turn leads to lower processing requirements and less power consumption. Less processing and power needs allow for size reduction, such as by reducing the size of the battery. Thus, in some embodiments, the entire augmented reality system may be integrated within the head-mounted display without a remote processing module. The head-mounted display may be configured as a pair of goggles, or may be similar in size and shape to a pair of glasses, as shown in FIG. 2.
[0379] Additionally, embodiments are described in which the image sensor is responsive to visible light. It should be understood that the techniques described herein are not limited to operation with visible light. They may alternatively or additionally be responsive to "light" in other parts of the spectrum, such as IR light or UV. Furthermore, image sensors as described herein are responsive to naturally occurring light. Alternatively or additionally, the sensor may be used in a system with an illumination source. In some embodiments, the sensitivity of the image sensor may be adjusted to the part of the spectrum in which the illumination source emits light.
[0380] As another example, it is described that a selected region of an image array where changes should be output from an image sensor is defined by defining a "patch" where image analysis is to be performed. However, it should be understood that the patch and the selected region may be of different sizes. The selected region may be larger than the patch, for example, to account for motion of an object in the image being tracked that deviates from a predicted trajectory and / or to allow processing around the edges of the patch.
[0381] Additionally, multiple processes are described, such as passable world model generation, object tracking, head pose tracking, and hand tracking. These, and in some embodiments other processes, may be executed by the same or different processors. The processors may be operated to enable simultaneous operation of these processes. However, each process may be implemented at different rates. If different processes require data from an image sensor or other sensor at different rates, sensor data acquisition may be managed by another process, etc., to provide data to each process at a rate appropriate for its operation.
[0382] Such alterations, modifications, and improvements are intended to be part of this disclosure and are intended to be within the spirit and scope of this disclosure. For example, in some embodiments, the color filter 102 of a pixel of an image sensor may not be a separate component, but instead may be incorporated into one of the other components of pixel sub-array 100. For example, in an embodiment including a single pixel with both an angle-of-arrival / position-intensity converter and a color filter, the angle-of-arrival / intensity converter may be a transmissive optical component formed from a material that filters certain wavelengths.
[0383] According to some embodiments, a wearable display system may be provided that includes a frame; a first camera mechanically coupled to the frame, the first camera configurable to output image data that satisfies an intensity change criterion within a first field of view for the first camera; and a processor operably coupled to the first camera, the processor configured to determine whether an object is within the first field of view and track movement of the object using image data received from the first camera for one or more portions of the first field of view.
[0384] In some embodiments, the object may be a hand, and tracking the movement of the object may include updating a corresponding portion of a model of the hand, including shape and / or movement constraints, based on image data from a first camera that satisfies an intensity change criterion.
[0385] In some embodiments, the processor may be further configured to provide instructions to the first camera to restrict image data acquisition to one or more patches of the first field of view that correspond to the object.
[0386] In some embodiments, the second camera may be mechanically coupled to the frame to provide a second field of view that at least partially overlaps with the first field of view, and the processor may be further configured to determine whether the object satisfies an occlusion criterion for the first field of view, enable the second camera or increase the frame rate of the second camera, determine depth information for the object using the second camera, and track the object using the determined depth information.
[0387] In some embodiments, the depth information may be determined stereoscopically using the images output by the first and second cameras.
[0388] In some embodiments, the depth information may be determined using light field information output by a second camera.
[0389] In some embodiments, the object may be a hand, and tracking the movement of the hand may include using the depth information determined by selecting points in a first field of view, associating depth information with the selected points, generating a depth map using the selected points, and matching portions of the depth map to corresponding portions of a model of the hand, including both shape and movement constraints.
[0390] In some embodiments, tracking the object's movement may include updating the object's location within the world model, and the interval between updates may be of duration between 1 ms and 15 ms.
[0391] In some embodiments, the second camera may be mechanically coupled to the frame to provide a second field of view that at least partially overlaps with the first field of view, and the processor may be operably coupled to the second camera and further configured to create a world model using images output by the first camera and the second camera, and to update the world model using light field information output by the second camera.
[0392] In some embodiments, the processor may be mechanically coupled to the frame.
[0393] In some embodiments, the display device, which is mechanically coupled to the frame, may include a processor.
[0394] In some embodiments, a local data processing module may comprise a processor, the local data processing module being operatively coupled to the display device through a communications link, the display device being mechanically coupled to the frame.
[0395] According to some embodiments, a wearable display system may be provided that includes a frame, two cameras mechanically coupled to the frame, a first camera and a second camera configurable to output image data that satisfies an intensity change criterion, the first camera and the second camera positioned to provide overlapping views of a central visual field, and a processor operably coupled to the first camera and the second camera.
[0396] In some embodiments, the processor may be further configured to determine that the object is within the central field of view, track the object using image data output by the first camera, determine whether the object tracking meets a quality criterion, enable the second camera or increase the frame rate of the second camera, and track the object using the first camera and the second camera when the object tracking does not meet the quality criterion.
[0397] In some embodiments, the first camera may be configurable to selectively output image frames or image data that satisfy an intensity change criterion, and the processor may be further configured to provide instructions to the first camera to restrict image data acquisition to one or more portions of the central field of view that correspond to the object, and the image data output by the first camera may relate to one or more portions of the central field of view.
[0398] In some embodiments, the first camera may be configurable to selectively output image frames or image data that satisfy an intensity change criterion, the second camera may comprise a plenoptic camera, and the processor may be further configured to determine whether the object satisfies a depth criterion, and tracking the object using the first camera and the second camera may include tracking the object using light field information obtained from the plenoptic camera when the depth criterion is satisfied, and tracking the object using depth information determined stereoscopically from the images output by the first camera and the second camera when the depth criterion is not satisfied.
[0399] In some embodiments, the plenoptic camera may include a transmissive diffractive mask.
[0400] In some embodiments, the plenoptic camera may have a horizontal field of view of 90 to 140 degrees, with the central field of view extending from 40 to 80 degrees.
[0401] In some embodiments, the first camera may provide a grayscale image and the second camera may provide a color image.
[0402] In some embodiments, the first camera may be configurable to selectively output image frames or image data that satisfy an intensity change criterion, the first camera comprising a global shutter and the second camera comprising a rolling shutter, and the processor may be further configured to compare a first image obtained using the first camera and a second image obtained using the second camera, detect distortion in at least a portion of the second image, and adjust at least a portion of the second image to compensate for the detected distortion.
[0403] In some embodiments, comparing a first image obtained using a first camera and a second image obtained using a second camera may include performing a line-by-line comparison between the first image and the second image.
[0404] In some embodiments, the processor may be mechanically coupled to the frame.
[0405] In some embodiments, the frame comprises a display device that is mechanically coupled to the processor.
[0406] In some embodiments, the local data processing module may comprise a processor, the local data processing module may be operably coupled to a display device through a communications link, and the frame may comprise a display device.
[0407] In some embodiments, the processor may be further configured to determine satisfaction of an occlusion criterion for the first camera, enable or increase the frame rate of the second camera, and track the object using image data output by the second camera.
[0408] In some embodiments, the occlusion criterion may be satisfied when the object occupies more than a threshold amount of the field of view for the first camera.
[0409] In some embodiments, the object may be a stationary object in the environment.
[0410] In some embodiments, the object may be the hand of a user of the wearable display system.
[0411] In some embodiments, the wearable display system further comprises an IR emitter mechanically coupled to the frame.
[0412] In some embodiments, the IR emitters are configured to be selectively activated to provide IR illumination.
[0413] Additionally, while advantages of the present disclosure are set forth, it should be understood that not all embodiments of the present disclosure include all described advantages. Some embodiments may not implement any feature described herein as advantageous. Thus, the foregoing description and drawings are by way of example only.
[0414] The foregoing embodiments of the present disclosure can be implemented in any of a number of ways. For example, the embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Such a processor may be implemented as an integrated circuit, with one or more processors in an integrated circuit component, including commercially available integrated circuit components known in the art, such as a CPU chip, a GPU chip, a microprocessor, a microcontroller, or a coprocessor, to name a few. In some embodiments, the processor may be implemented in a custom circuit, such as an ASIC, or in a semi-custom circuit resulting from configuring a programmable logic device. As a further alternative, the processor may be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. As a specific example, some commercially available microprocessors have multiple cores, such that one or a subset of those cores may constitute a processor. However, the processor may be implemented using a circuit in any suitable format.
[0415] Further, it should be understood that a computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer, etc. In addition, a computer may be embodied in devices not generally considered computers but with suitable processing capabilities, including a personal digital assistant (PDA), a smart phone, or any suitable portable or fixed electronic device.
[0416] A computer may also have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that may be used to provide a user interface include a printer or display screen for visual presentation of output, or a speaker or other sound generating device for audible presentation of output. Examples of input devices that may be used for a user interface include keyboards and pointing devices such as mice, touchpads, and digitizing tablets. As another example, a computer may receive input information through speech recognition or in other audible formats. In the illustrated embodiment, the input / output devices are illustrated as physically separate from the computing device. However, in some embodiments, the input and / or output devices may be physically integrated into the same unit as the processor or other elements of the computing device. For example, the keyboard may be implemented as a soft keyboard on a touch screen. In some embodiments, the input / output devices may be completely disconnected from the computing device and functionally integrated through a wireless connection.
[0417] Such computers may be interconnected by one or more networks in any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet. Such networks may be based on any suitable technology and operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.
[0418] Also, the various methods and processes outlined herein may be coded as software that is executable on one or more processors employing any one of a variety of operating systems or platforms. In addition, such software may be written using any of a number of suitable programming languages and / or programming or scripting tools, and may be compiled as executable machine language code or intermediate code that runs on a framework or virtual machine.
[0419] In this regard, the present disclosure may be embodied as a computer-readable storage medium (or multiple computer-readable media) (e.g., a computer memory, one or more floppy disks, compact disks (CDs), optical disks, digital video disks (DVDs), magnetic tapes, flash memories, circuitry in field programmable gate arrays or other semiconductor devices, or other tangible computer storage media) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods implementing various embodiments of the present disclosure discussed above. As is evident from the foregoing examples, a computer-readable storage medium may retain information for a sufficient time to provide computer-executable instructions in a non-transitory form. Such a computer-readable storage medium or media may be transportable such that one or more programs stored thereon may be loaded onto one or more different computers or other processors to implement various aspects of the present disclosure, as described above. As used herein, the term "computer-readable storage medium" encompasses only computer-readable media that may be considered a manufacture (i.e., an article of manufacture) or machine. In some embodiments, the present disclosure may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagated signal.
[0420] The term "program" or "software" is used herein in a general sense to refer to any type of computer code or set of computer executable instructions that can be employed to program a computer or other processor to implement various aspects of the present disclosure, as described above. In addition, according to one aspect of the present embodiment, it should be understood that one or more computer programs, when executed to perform the methods of the present disclosure, do not need to reside on a single computer or processor, but can be distributed in a modular manner among several different computers or processors to implement various aspects of the present disclosure.
[0421] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.
[0422] Also, the data structures may be stored in the computer-readable medium in any suitable form. For ease of illustration, the data structures may be shown to have fields that are related through locations within the data structure. Such relationships may be achieved by allocating storage for the fields with locations in the computer-readable medium that similarly convey the relationship between the fields. However, any suitable mechanism may be used to establish relationships between information in the fields of the data structures, including through the use of pointers, tags, or other mechanisms that establish relationships between data elements.
[0423] Various aspects of the disclosure may be used alone, in combination, or in various arrangements not specifically discussed in the foregoing embodiments, and therefore, its application is not limited to the details and arrangements of components set forth in the foregoing description or illustrated in the drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.
[0424] The present disclosure may also be embodied as a method, examples of which are provided. The acts performed as part of the method may be ordered in any suitable manner. Thus, while the illustrative embodiments are shown as sequential acts, embodiments may be constructed in which acts are performed in an order different from that shown, which may include performing some acts simultaneously.
[0425] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not, by itself, imply any priority, precedence, or ordering of one claim element relative to another element, or the temporal order in which acts of a method are performed, but rather the ordinal terms are used solely as markers to distinguish one claim element having a certain name from another element having the same name (due to the use of the ordinal terms) to distinguish between claim elements.
[0426] Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "with," and variations thereof herein is meant to encompass the items listed thereafter and equivalents and additional items.
Claims
1. A portable device, comprising: a first camera configurable to output image frames or image data that satisfy an intensity change criterion; A second camera; a processor operatively coupled to the first camera and the second camera; Equipped with the first camera and the second camera are positioned to provide overlapping views of a central field of view; The processor, creating a world model using stereoscopically determined depth information from images output by the first camera and the second camera; performing a tracking routine using the world model and image data output by the first camera; and 4. A portable device configured to:
2. A method for performing a tracking routine using a portable device, the portable device comprising: a first camera configurable to output image frames or image data that satisfy an intensity change criterion; A second camera; a processor operatively coupled to the first camera and the second camera; Equipped with the first camera and the second camera are positioned to provide overlapping views of a central field of view; The method includes using the processor: creating a world model using stereoscopically determined depth information from images output by the first camera and the second camera; performing a tracking routine using the world model and image data output by the first camera; and A method comprising:
3. A portable device, comprising: A frame, a first camera mechanically coupled to the frame, the first camera configurable to output image data that satisfies an intensity change criterion within a first field of view for the first camera; a processor operatively coupled to the first camera; Equipped with The processor, determining whether a point of interest is within the first field of view; performing a tracking routine using image data received from the first camera for one or more portions of the first field of view; 4. A portable device configured to:
4. A method for performing a tracking routine using a portable device, the portable device comprising: A frame, a first camera mechanically coupled to the frame, the first camera configurable to output image data that satisfies an intensity change criterion within a first field of view for the first camera; a processor operatively coupled to the first camera; Equipped with The method includes using the processor: determining whether the point of interest is within the first field of view; performing a tracking routine using image data received from the first camera for one or more portions of the first field of view; A method comprising:
5. A portable device, comprising: a first camera configurable to output image data that satisfies an intensity change criterion; A second camera; a processor operatively coupled to the first camera and the second camera; Equipped with A portable device, wherein the first camera and the second camera are positioned to provide overlapping views of a central field of view.
6. The portable device of claim 1, or the portable device of claim 3, or the portable device of claim 5, wherein the intensity change criteria include absolute or relative intensity change criteria.
7. The portable device of claim 1, or the portable device of claim 3, or the portable device of claim 5, wherein the first camera is configured to output the image data asynchronously.
8. The portable device of claim 7, wherein the processor is further configured to track the object asynchronously.
9. The portable device of claim 1, wherein executing the tracking routine includes limiting image data acquisition to points of interest within the world model.
10. The first camera can be configured to limit image acquisition to one or more portions of a field of view of the first camera; The tracking routine comprises: identifying a point of interest within the world model; determining one or more first portions of a field of view of the first camera that correspond to the point of interest; providing instructions to the first camera to limit image capture to the one or more first portions of the field of view; The portable device of claim 9 , further comprising:
11. The tracking routine, estimating one or more second portions of the field of view of the first camera that correspond to the point of interest based on a movement of the point of interest relative to the world model or a movement of the portable device relative to the point of interest; providing instructions to the first camera to limit image acquisition to the one or more second portions of the field of view; The portable device of claim 10 further comprising:
12. The portable device further comprising an inertial measurement unit; 10. The portable device of claim 9, wherein executing the tracking routine includes estimating an updated relative position of a point of interest of the plurality of points of interest based at least in part on an output of the inertial measurement unit.
13. The tracking routine comprising: iteratively calculating a position of the point of interest within the world model; The portable device of claim 9 , wherein the repeated calculations are performed with a temporal resolution of greater than 60 Hz.
14. The portable device of claim 13, wherein the interval between repeated calculations is between 1 ms and 15 ms in duration.
15. The processor, determining whether the tracking routine meets a quality criterion; enabling the second camera or modulating the frame rate of the second camera when the tracking routine does not meet the quality criterion; The portable device of claim 1 , further configured to:
16. The portable device of claim 1, further comprising a display device mechanically coupled to the processor.
17. The portable device of claim 1, further comprising an IR emitter.
18. The portable device of claim 17, wherein the processor is configured to selectively enable the IR emitter to enable head pose tracking under low light conditions.
19. The method of claim 2 or the method of claim 4, wherein the intensity change criteria include absolute or relative intensity change criteria.
20. The method of claim 2 or the method of claim 4, wherein the first camera is configured to output the image data asynchronously.
21. The method of claim 20, wherein the processor is further configured to asynchronously track head pose.
22. The method of claim 2, wherein the processor is further configured to limit image data acquisition to a plurality of points of interest within the world model by executing a tracking routine.
23. The first camera can be configured to limit image acquisition to one or more portions of a field of view of the first camera; The tracking routine comprises: identifying a point of interest within the world model; determining one or more first portions of a field of view of the first camera that correspond to the point of interest; providing instructions to the first camera to limit image capture to the one or more first portions of the field of view; 23. The method of claim 22, comprising:
24. The tracking routine, estimating one or more second portions of the field of view of the first camera that correspond to the point of interest based on a movement of the point of interest relative to the world model or a movement of the portable device relative to the point of interest; providing instructions to the first camera to limit image acquisition to the one or more second portions of the field of view; 24. The method of claim 23, further comprising:
25. The portable device further comprising an inertial measurement unit; 23. The method of claim 22, wherein performing the tracking routine includes estimating an updated relative position of a point of interest of the plurality of points of interest based at least in part on an output of the inertial measurement unit.
26. The tracking routine comprising: iteratively calculating a position of the point of interest within the world model; 23. The method of claim 22, wherein the repeated calculations are performed with a temporal resolution of greater than 60 Hz.
27. The method of claim 26, wherein the interval between repeated calculations is between 1 ms and 15 ms in duration.
28. The processor, determining whether the tracking routine meets a quality criterion; enabling the second camera or modulating the frame rate of the second camera when the tracking routine does not meet the quality criterion; The method of claim 2 , further comprising:
29. The method of claim 2, wherein the portable device is equipped with a display.
30. The method of claim 2, wherein the headset further includes an IR emitter.
31. The method of claim 30, wherein the processor is configured to selectively enable the IR emitter to enable execution of the tracking routine under low light conditions.
