Multi-camera cross-reality device

By using calibration routines with multiple cameras and inertial measurement units in a lightweight head-mounted device, the issues of weight and power consumption were addressed, ensuring that the XR system accurately acquires information about the physical world with low latency and low power consumption, thus improving the user experience.

CN118921450APending Publication Date: 2024-11-08MAGIC LEAP INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411023176.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-12-21
Filing Date
2019-12-19
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing wearable XR display systems suffer from excessively heavy head-mounted devices, which negatively impact user experience and consume significant amounts of power. Changes in sensor position or orientation can lead to inaccurate information acquisition, limiting the system's practicality and user experience.

Method used

A lightweight head-mounted device is used, which combines multiple cameras and an inertial measurement unit. The relative position and orientation of the sensors are determined by repeatedly executing calibration routines, reducing the reliance on active depth sensors. Image information is acquired using a combination of grayscale and color cameras, and resource-intensive sensors are selectively activated to reduce power consumption.

Benefits of technology

It enables accurate acquisition of physical world information with low latency and low power consumption, improving the system's usability and user experience, reducing the frequency of battery charging, and lowering the device weight.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118921450A_ABST
    Figure CN118921450A_ABST
Patent Text Reader

Abstract

A wearable display system with a limited number of cameras. The two cameras may be arranged to provide overlapping central and peripheral fields of view associated with one of the two cameras. The third camera may be arranged to provide a color field of view that overlaps the central field of view. The wearable display system may be coupled to a processor configured to generate a world model using the two cameras and track hand motion in the central field of view. The processor may be configured to execute a calibration routine to compensate for deformation during use of the wearable display system. The processor may be configured to identify and address portions of the world model that include incomplete depth information by obtaining additional depth information, such as by enabling a transmitter, detecting a flat surface in the physical world, or identifying related object templates in the world model.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of an application with a filing date of December 19, 2019, a PCT international application number of PCT / US2019 / 067476, a Chinese national phase application number of 201980092468.9, and an invention name of “Multi-camera cross-reality device”. Technical Field

[0002] The present application generally relates to a wearable cross-reality display system including a camera. Background Art

[0003] A computer may control a human user interface to create an X-reality (XR or cross-reality) environment, where part or all of the XR environment perceived by the user is generated by a computer. These XR environments may be virtual reality (VR), augmented reality (AR), or mixed reality (MR) environments, where part or all of the XR environment may be generated by a computer in part using data describing the environment. The data may, for example, describe virtual objects that may be presented in a manner that is felt or perceived by the user as part of the physical world, so that the user can interact with the virtual objects. Since the data is presented and displayed through a user interface device, such as a head-mounted display device, the user may experience these virtual objects. The data may be displayed for the user to view, or may control audio played for the user to hear, or may control a tactile (or haptic) interface, enabling the user to experience the tactile sensation that the user feels or perceives as touching a virtual object.

[0004] XR systems can be used for many applications, covering areas such as scientific visualization, medical training, engineering design and prototyping, remote operation and telepresence, and personal entertainment. Compared with VR, AR and MR include one or more virtual objects related to real objects in the physical world. The experience of virtual objects interacting with real objects greatly enhances the user's enjoyment of using XR systems, and also opens the door to a variety of applications that provide realistic and easy-to-understand information about how the physical world may be changed. Summary of the invention

[0005] Some aspects of the present application relate to a wearable cross-reality display system including multiple cameras.The techniques described herein can be used together, separately, or in any suitable combination.

[0006] According to some embodiments, a wearable display system is provided, wherein the wearable display system may include a head-mounted device; two first cameras, which are mechanically coupled to the head-mounted device; a first inertial measurement unit, which is mechanically coupled to a first first camera of the two first cameras, and a second inertial measurement unit, which is mechanically coupled to a second first camera of the two first cameras; and a processor, which is operably coupled to the two first cameras and is configured to: execute a calibration routine, the calibration routine being configured to determine the relative orientation of the two first cameras using images acquired by the two first cameras and outputs of the first inertial measurement unit and the second inertial measurement unit.

[0007] In some embodiments, the calibration routine, when executed, may further determine the relative position of the two first cameras. In some embodiments, executing the calibration routine may include identifying corresponding features in images acquired by each of the first first camera and the second first camera; for each of a plurality of estimated relative orientations of the two first cameras, calculating an error, wherein the error indicates a difference between the corresponding feature appearing in the image acquired with each of the two first cameras and an estimate of the identified feature calculated based on the estimated relative orientation of the two first cameras; and based on the calculated error, selecting a relative orientation from the plurality of estimated relative orientations as the determined relative orientation. In some embodiments, the calibration routine may also include: selecting at least an initial estimate of the plurality of estimated relative orientations based in part on the outputs of the first inertial measurement unit and the second inertial measurement unit.

[0008] In some embodiments, the wearable display system may further include a color camera coupled to the head mounted device; and the processor may be further configured to: create a world model using the two first cameras and the color camera; update the world model using the two first cameras at a first rate; and update the world model using the two first cameras and the color camera at a second rate slower than the first rate. In some embodiments, the two first cameras may be mechanically coupled to the head mounted device to provide a central field of view associated with both of the two first cameras and a peripheral field of view associated with a first of the two first cameras; and the processor may be further configured to track hand movements in the central field of view using depth information determined from images acquired by the two first cameras.

[0009] In some embodiments, tracking hand movement in the central field of view using depth information may include: selecting a point in the central field of view; using the images acquired by the two first cameras to stereoscopically determine depth information of the selected point; using the stereoscopically determined depth information, generating a depth map; and matching portions of the depth map with corresponding portions of a hand model that includes both shape constraints and motion constraints. In some embodiments, the processor may also be configured to match portions of the image with corresponding portions of a hand model that includes both shape constraints and motion constraints, thereby tracking hand movement in the peripheral field of view using one or more images acquired by the two first cameras.

[0010] In some embodiments, the head-mounted device may be a lightweight head-mounted device weighing between 30 and 300 grams. In some embodiments, the head-mounted device may further include the processor. In some embodiments, the head-mounted device may further include a battery pack. In some embodiments, the two first cameras may be configured to acquire grayscale images. In some embodiments, the processor may be mechanically coupled to the head-mounted device. In some embodiments, the head-mounted device may include a display device, which is mechanically coupled to the processor. In some embodiments, a local data processing module may include the processor, the local data processing module may be operably coupled to the display device via a communication link, and wherein the head-mounted device may include the display device.

[0011] According to some embodiments, a wearable display system is provided, wherein the wearable display system may include: a frame; two first cameras, which are mechanically coupled to the frame to provide a central field of view associated with both cameras and a first peripheral field of view associated with a first first camera of the two first cameras; a color camera, which is mechanically coupled to the frame to provide a color field of view overlapping with the central field of view; and a processor, which is operably coupled to the two first cameras and the color camera and is configured to: track hand movements in the central field of view using depth information stereoscopically determined based on first images acquired by the two first cameras; track hand movements in the first peripheral field of view using one or more second images acquired by the first first camera of the two first cameras; create a world model using the two first cameras and the color camera; and update the world model using the two first cameras.

[0012] In some embodiments, the two first cameras may be configured with a global shutter. In some embodiments, the wearable display system may further include a hardware accelerator for stereoscopically determining depth information using the first grayscale images acquired by the two first cameras. In some embodiments, the two first cameras may have equidistant lenses. In some embodiments, each of the two first cameras may have a horizontal field of view between 90 degrees and 175 degrees. In some embodiments, the central field of view may have an angular range between 40 degrees and 80 degrees. In some embodiments, the processor may also be configured to perform a calibration routine to determine the relative orientation of the two first cameras.

[0013] In some embodiments, the calibration routine may include: identifying corresponding features in images acquired with each of the two first cameras; calculating an error for each of a plurality of estimated relative orientations of the two first cameras, wherein the error indicates a difference between the corresponding feature appearing in the images acquired with each of the two first cameras and an estimate of the identified feature calculated based on the estimated relative orientation of the two first cameras; and selecting a relative orientation from the plurality of estimated relative orientations as a determined relative orientation based on the calculated error.

[0014] In some embodiments, the wearable display system may also include: a first inertial measurement unit mechanically coupled to a first first camera of the two first cameras; and a second inertial measurement unit mechanically coupled to a second first camera of the two first cameras, and the calibration routine may also include selecting at least one of a plurality of estimated relative orientations based in part on outputs of the first inertial measurement unit and the second inertial measurement unit.

[0015] In some embodiments, the processor may be further configured to repeatedly perform a calibration routine while wearing the wearable display system, such that the calibration routine compensates for frame deformation during use of the wearable display system. In some embodiments, the calibration routine may compensate for frame deformation caused by temperature changes. In some embodiments, the calibration routine may compensate for frame deformation caused by mechanical strain. In some embodiments, the two first cameras may be configured to acquire grayscale images. In some embodiments, the wearable display system may have a color camera. In some embodiments, the processor may be mechanically coupled to the frame. In some embodiments, the display device may be mechanically coupled to the frame, wherein the display device includes the processor. In some embodiments, the local data processing module may include a processor, the local data processing module may be operably coupled to the display device via a communication link, and the display device may be mechanically coupled to the frame. In some embodiments, the first image may include one or more second images.

[0016] According to some embodiments, a wearable display system is provided, wherein the wearable display system may include: a frame; two first cameras mechanically coupled to the frame; a color camera mechanically coupled to the frame; and a processor operably coupled to the two first cameras and the color camera and configured to: create a world model using first grayscale images acquired by the two first cameras and one or more color images acquired by the color camera; update the world model using second grayscale images acquired by the two first cameras; determine that a portion of the world model includes incomplete depth information; and update the world model with additional depth information for the portion of the world model that includes incomplete depth information.

[0017] In some embodiments, the wearable display system may further include one or more emitters, and the one or more emitters may be used to obtain additional depth information, and the processor may be further configured to enable the one or more emitters in response to determining that a portion of the world model includes incomplete depth information. In some embodiments, the one or more emitters may include an infrared emitter, and the two first cameras may include a filter configured to pass infrared light. In some embodiments, the infrared emitter may emit light with a wavelength between 900 nanometers and 1 micron.

[0018] In some embodiments, the processor may be further configured to detect a flat surface in the physical world in response to determining that a portion of the world model includes incomplete depth information, and may estimate additional depth information based on the detected flat surface. In some embodiments, the processor may be further configured to detect an object in the portion of the world model that includes incomplete depth information, identify an object template corresponding to the detected object, configure an instance of the object template based on an image of the object in the updated world model, and estimate additional depth information based on the instance of the configured object template. In some embodiments, the two first cameras may be mechanically coupled to the frame to provide a central field of view associated with the two first cameras and a peripheral field of view associated with a first of the two first cameras, and the processor may be further configured to track hand movement in the central field of view using depth information determined based on images acquired by the two first cameras.

[0019] In some embodiments, tracking hand movements in the central visual field using depth information may include: selecting a point in the central visual field; using images acquired by two first cameras, stereoscopically determining depth information of the selected point; generating a depth map using the stereoscopically determined depth information; and matching a portion of the depth map with a corresponding portion of a hand model that includes both shape constraints and motion constraints. In some embodiments, the processor may also be configured to track hand movements in the peripheral visual field using one or more images acquired by a first first camera of the two first cameras by matching a portion of the image with a corresponding portion of a hand model that includes both shape constraints and motion constraints. In some embodiments, the processor may be mechanically coupled to the frame. In some embodiments, a display device mechanically coupled to the frame may include a processor. In some embodiments, a local data processing module may include a processor, the local data processing module may be operably coupled to a display device via a communication link, and the display device may be mechanically coupled to the frame.

[0020] According to some embodiments, a wearable display system is provided, wherein the wearable display system may include: a frame; two grayscale cameras mechanically coupled to the frame, wherein the two grayscale cameras include a first grayscale camera having a first field of view and a second grayscale camera having a second field of view, and the first grayscale camera and the second grayscale camera are positioned to provide: a central field of view overlapping the first field of view with the second field of view; and a first peripheral field of view within the first field of view and outside the second field of view; and a color camera mechanically coupled to the frame to provide a color field of view overlapping the central field of view.

[0021] In some embodiments, the two grayscales may have a global shutter. In some embodiments, the color camera may have a rolling shutter. In some embodiments, the wearable display system may include two inertial measurement units mechanically coupled to a frame, and one or more emitters mechanically coupled to the frame. In some embodiments, one or more emitters may be infrared emitters, and the two grayscale cameras may include a filter configured to pass infrared light. In some embodiments, the infrared emitter may emit light with a wavelength between 900 nanometers and 1 micron. In some embodiments, the illumination field of one or more emitters may overlap with the central field of view. In some embodiments, a first inertial measurement unit of the two inertial measurement units may be mechanically coupled to a first grayscale camera, and a second inertial measurement unit of the two inertial measurement units may be mechanically coupled to a second grayscale camera of the two grayscale cameras. In some embodiments, the two inertial measurement units may be configured to measure inclination, acceleration, velocity, or any combination thereof. In some embodiments, the two grayscale cameras may have a horizontal field of view between 90 degrees and 175 degrees. In some embodiments, the central field of view may have an angular range between 40 degrees and 80 degrees.

[0022] The foregoing summary is provided by way of illustration and is not intended to be limiting. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component shown in various figures is represented by the same reference number. For clarity, not every component is labeled in every figure. In the drawings:

[0024] Figure 1 is a sketch showing an example of a simplified augmented reality (AR) scene in accordance with some embodiments.

[0025] Figure 2 is a schematic diagram illustrating an example of an AR display system according to some embodiments.

[0026] Figure 3A is a schematic diagram illustrating a user wearing an AR display system that renders AR content as the user moves in a physical world environment in accordance with some embodiments.

[0027] Figure 3B is a schematic diagram showing a viewing optics assembly and associated components according to some embodiments.

[0028] Figure 4 is a schematic diagram illustrating an image sensing system according to some embodiments.

[0029] Figure 5A is a diagram showing some embodiments of the present invention. Figure 4 Schematic diagram of the pixel primitive in .

[0030] Figure 5B is a diagram showing some embodiments of the present invention. Figure 5A Schematic diagram of the output events for the pixel primitive in .

[0031] Figure 6 is a schematic diagram illustrating an image sensor according to some embodiments.

[0032] Figure 7 is a schematic diagram illustrating an image sensor according to some embodiments.

[0033] Figure 8 is a schematic diagram illustrating an image sensor according to some embodiments.

[0034] Fig. 9 is a simplified flow chart of a method for image sensing according to some embodiments.

[0035] Fig.10 According to some embodiments Fig. 9 Simplified flowchart of the patch recognition action.

[0036] Fig.11 According to some embodiments Fig. 9 Simplified flowchart of the block-by-block trajectory estimation action.

[0037] Fig.12 is a diagram showing a perspective view according to some embodiments. Fig.11 Schematic diagram of block trajectory estimation.

[0038] Fig.13 is a diagram showing changes relative to viewpoint according to some embodiments Fig.11 Schematic diagram of block trajectory estimation.

[0039] Fig.14 is a schematic diagram illustrating an image sensing system according to some embodiments.

[0040] Fig.15 is a diagram showing some embodiments of the present invention. Fig.14 Schematic diagram of the pixel primitive in .

[0041] Fig.16 is a schematic diagram of a pixel subarray according to some embodiments.

[0042] Fig.17A is a cross-sectional view of a plenoptic device having an arrival angle-to-intensity converter in the form of two aligned stacked transmissive diffractive masks (TDMs) according to some embodiments.

[0043] Fig. 17Bis a cross-sectional view of an all-optical device with an arrival angle-to-intensity converter in the form of two stacked TDMs that are not aligned, according to some embodiments.

[0044] Fig.18A is a pixel subarray having color pixel primitives and angle-of-arrival pixel primitives according to some embodiments.

[0045] Fig.18B is a pixel subarray having color pixel primitives and angle-of-arrival pixel primitives according to some embodiments.

[0046] Fig.18C is a pixel subarray having white pixel primitives and angle-of-arrival pixel primitives according to some embodiments.

[0047] Fig.19A is a top view of a photodetector array with a single TDM according to some embodiments.

[0048] Fig.19B is a side view of a photodetector array with a single TDM according to some embodiments.

[0049] Fig. 20A is a top view of a photodetector array with multiple arrival angle-to-intensity converters in TDM format, according to some embodiments.

[0050] Fig. 20B is a side view of a photodetector array with multiple TDMs according to some embodiments.

[0051] Fig. 20C is a side view of a photodetector array with multiple TDMs according to some embodiments.

[0052] Fig.21 is a schematic diagram of a head mounted device including three camera and accessory components according to some embodiments.

[0053] Fig. 22 is a simplified flow chart of a calibration routine according to some embodiments.

[0054] FIG. 23A to FIG. 23C According to some embodiments Fig.21 An exemplary field of view diagram associated with a head-mounted device.

[0055] Fig.24 is a simplified flow chart of a method for updating a world model according to some embodiments.

[0056] Fig.25 is a simplified flow chart of a human hand tracking process according to some embodiments. DETAILED DESCRIPTION

[0057] The inventors have recognized and understood the design and operation techniques of wearable XR display systems that enhance the accessibility and usefulness of such systems. These designs and / or operation techniques can obtain information to perform a variety of functions, including hand tracking, head pose tracking, and world reconstruction using a limited number of cameras that can be used to realistically render virtual objects so that they appear to interact realistically with physical objects. The wearable cross-reality display system can be lightweight and consume low power in operation. The system can use sensors in a specific configuration to obtain image information about physical objects in the physical world with low latency. The system can perform various routines to improve the accuracy and / or realism of the displayed XR environment. Such routines may include calibration routines that improve the accuracy of stereo depth measurements even if the lightweight frame is deformed during use, and routines that include detecting and addressing incomplete depth information in the model of the physical world around the user.

[0058] The weight of known XR system headsets can limit a user's enjoyment. Such XR headsets can weigh over 340 grams (sometimes even over 700 grams). In contrast, glasses may weigh less than 50 grams. Wearing such relatively heavy headsets for long periods of time can cause users to feel fatigued or distract them from the ideal immersive XR experience. However, the inventors have recognized and understood that some designs that reduce the weight of the headset also increase the flexibility of the headset, making the lightweight headset susceptible to changes in sensor position or orientation during use or over a period of time. For example, when a user wears a lightweight headset that includes camera sensors, the relative orientation of these camera sensors may shift. Changes in the spacing of cameras used for stereoscopic imaging may affect the ability of those headsets to obtain accurate stereo information, which depends on cameras having a known positional relationship relative to each other. Therefore, a calibration routine that can be repeated while the headset is worn allows the lightweight headset to accurately obtain information about the world around the wearer of the headset using stereo imaging technology.

[0059] The need to equip XR systems with components to acquire information about objects in the physical world can also limit the usefulness and user enjoyment of these systems. This can limit the size, power consumption, and realism of XR systems when the acquired information is used to realistically render computer-generated virtual objects in the proper location and appearance relative to physical objects.

[0060] For example, an XR system may use sensors worn by a user to obtain information about objects in the physical world around the user, including information about the position of physical world objects in the user's field of view. Because objects move in the physical world or users change their posture relative to the physical world, so that physical objects enter or leave the user's field of view or the position of physical objects in the user's field of view changes, challenges arise because objects may move relative to the user's field of view. In order to present a realistic XR display, the model of physical objects in the physical world must be updated frequently enough to capture these changes, handle these changes with low enough latency, and accurately predict the future to cover the complete latency path including rendering, so that virtual objects displayed based on this information have appropriate positions and appearances relative to physical objects when the virtual objects are displayed. Otherwise, virtual objects will appear to be out of alignment with physical objects, thereby failing to realistically display a combined scene including physical objects and virtual objects. For example, a virtual object may appear to float in the air rather than being fixed to a physical object, or may appear to jump back and forth relative to a physical object. Errors in visual tracking are particularly amplified when the user moves at high speeds and there is significant movement in the scene.

[0061] These problems can be avoided by sensors that acquire new data at a high rate. However, the power consumed by such sensors results in the need for larger batteries, thereby increasing the weight of the system or limiting the length of time such systems can be used. Similarly, the processors required to process the data generated at a high rate can drain the battery and add additional weight to the wearable system, further limiting the practicality or enjoyability of such systems. For example, one known approach is to run at a higher resolution in order to capture sufficient visual detail, and operate the sensor at a higher frame rate to increase the temporal resolution. An alternative solution might be to supplement this solution with an IR time-of-flight sensor, which might directly indicate the position of a physical object relative to the sensor, perform simple processing when using this information to display a virtual object, and produce low latency. However, such sensors can consume a lot of power, especially if they are operated in sunlight.

[0062] The inventors have recognized and appreciated that an XR system can account for changes in sensor position or orientation during use or over a period of time by repeatedly performing a calibration routine. The calibration routine can determine the current relative spacing and orientation of sensors included in a head-mounted device. The wearable XR system can then take into account the current relative spacing and orientation of the head-mounted device sensors when calculating stereo depth information. With this calibration capability, the XR system can accurately obtain depth information to indicate the distance to objects in the physical world without the need for active depth sensors, or only occasionally using active depth sensing. Because active depth sensing consumes a lot of power, reducing or eliminating active depth sensing can enable the device to consume less power, thereby increasing the device operating time without the need to charge the battery, or reducing the device size due to the reduced battery size.

[0063] The inventors also recognize and appreciate that, with an appropriate combination of image sensors, and appropriate techniques for processing image information from these sensors, an XR system can obtain information about physical objects with low latency, and even with low power consumption, by: reducing the number of sensors used; eliminating, disabling, or selectively activating resource-intensive sensors; and / or reducing the overall use of sensors. As a specific example, an XR system may include a head-mounted device with two world cameras and one color camera. The world cameras may produce grayscale images and may have a global shutter. These grayscale images are smaller in size than color images of similar resolution. World cameras may require less power than color cameras of similar resolution. Information from these cameras may be used at different times and in different ways to support the operation of the XR system.

[0064] The techniques described herein may be used with various types of devices or alone in various types of scenarios. Figure 1 Such a scenario is shown. Figure 2 , Figure 3A and Figure 3B An exemplary AR system is shown, including one or more processors, memory, sensors, and user interfaces that operate in accordance with the techniques described herein.

[0065] refer to Figure 1, shows an AR scene 4 in which the AR system user sees a physical world park-like environment 6, which features people, trees, buildings, and a concrete platform 8 in the background. In addition to these physical objects, the AR technology user also perceives that they "see" virtual objects, which are shown here as a robot statue 10 standing on the physical world concrete platform 8, and a flying cartoon avatar character 2 that appears to be an incarnation of a bumblebee, even though these elements (e.g., avatar character 2 and robot statue 10) do not exist in the physical world. Due to the extreme complexity of human visual perception and nervous system, it is extremely challenging to make an AR system to promote comfortable, natural-feeling, richly rendered virtual image elements among other virtual or physical world image elements.

[0066] Such scenes can be presented to the user by presenting image information representing the actual environment around the user and superimposing information representing virtual objects that are not in the actual environment. In an AR system, the user is able to see objects in the physical world, and the AR system provides information for rendering virtual objects so that the virtual objects appear in the appropriate location and have appropriate visual features, and the virtual objects appear to coexist with objects in the physical world. For example, in an AR system, the user looks through a transparent screen so that the user can see objects in the physical world. The AR system can render virtual objects on the screen so that the user sees both the physical world and the virtual objects. In some embodiments, the screen can be worn by the user, such as a pair of goggles or glasses.

[0067] The scene may be presented to the user via a system that includes multiple components, including a user interface that stimulates one or more user senses, including vision, hearing, and / or touch. In addition, the system may include one or more sensors that measure parameters of the physical portion of the scene, including the position and / or movement of the user within the physical portion of the scene. In addition, the system may include one or more computing devices and associated computer hardware, such as memory. These components may be integrated into a single device or distributed across multiple interconnected devices. In some embodiments, some or all of these components may be integrated into a wearable device.

[0068] In some embodiments, an AR experience may be provided to a user via a wearable display system. Figure 2An example of a wearable display system 80 (hereinafter referred to as "system 80") is shown. System 80 includes a head mounted display device 62 (hereinafter referred to as "display device 62"), and various mechanical and electronic modules and systems that support the operation of display device 62. Display device 62 can be coupled to a frame 64, which can be worn by a display system user or viewer 60 (hereinafter referred to as "user 60") and is configured to position display device 62 in front of the eyes of user 60. According to various embodiments, display device 62 can be a sequential display. Display device 62 can be a monocular device or a binocular device.

[0069] In some embodiments, speaker 66 is coupled to frame 64 and positioned near the ear canal of user 60. In some embodiments, another speaker (not shown) is positioned near the other ear canal of user 60 to provide stereo / shapeable sound control.

[0070] The system 80 may include a local data processing module 70. The local data processing module 70 may be operably coupled to the display device 62 via a communication link 68, such as via a wired lead or a wireless connection. The local data processing module 70 may be mounted in a variety of configurations, such as fixedly attached to the frame 64, fixedly attached to a helmet or hat worn by the user 60, embedded in headphones, or otherwise removably attached to the user 60 (e.g., in a backpack configuration, in a belt-coupled configuration). In some embodiments, the local data processing module 70 may not be present, as components of the local data processing module 70 may be integrated into the display device 62, or implemented in a remote server or other component that is coupled to the display device 62, such as by wireless communication via a wide area network.

[0071] The local data processing module 70 may include a processor and digital memory, such as non-volatile memory (e.g., flash memory), both of which may be used to assist in processing, caching, and storing data. The data may include: a) data captured from a sensor (e.g., which is operably coupled to the frame 64 or otherwise attached to the user 60) such as an image capture device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a radio device, and / or a gyroscope; and / or b) data acquired and / or processed using the remote processing module 72 and / or the remote data repository 74, which data may be transmitted to the display device 62 after such processing or retrieval. The local data processing module 70 may be operably coupled to the remote processing module 72 and the remote data repository 74 by communication links 76, 78, such as via a wired or wireless communication link, respectively, so that these remote modules 72, 74 are operably coupled to each other and can be used as resources for the local processing and data module 70.

[0072] In some embodiments, the local data processing module 70 may include one or more processors (e.g., a central processing unit and / or one or more graphics processing units (GPUs)) configured to analyze and process data and / or image information. In some embodiments, the remote data repository 74 may include a digital data storage facility that may be available via the Internet or other network configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local data processing module 70, allowing for fully autonomous use from the remote module.

[0073] In some embodiments, the local data processing module 70 is operably coupled to a battery 82. In some embodiments, the battery 82 is a removable power source, such as an over the counter battery. In other embodiments, the battery 82 is a lithium ion battery. In some embodiments, the battery 82 includes both an internal lithium ion battery and a removable battery that can be charged by the user 60 during the non-operating time of the system 80, so that the user 60 can operate the system 80 for a longer period of time without having to be tethered to a power source to charge the lithium ion battery, or having to shut down the system 80 to replace the battery.

[0074] Figure 3A A user 30 is shown wearing an AR display system that renders AR content as the user 30 moves within a physical world environment 32 (hereinafter referred to as "environment 32"). The user 30 places the AR display system at a location 34, and the AR display system records environmental information of the passable world relative to the location 34 (e.g., digital representations of real objects in the physical world, which can be stored and updated as the real objects in the physical world change). Each location 34 can also be associated with a "pose" and / or mapped features or directional audio input relative to the environment 32. A user wearing the AR display system on his head can look in a particular direction and tilt his head, thereby creating a system head pose relative to the environment. At each location and / or each pose within the same location, sensors on the AR display system can capture different information about the environment 32. Therefore, the information collected at the location 34 can be aggregated into a data input 36 and processed by at least a connected world module 38, such processing being performed, for example, by Figure 2 The processing is implemented on the remote processing module 72.

[0075] The connected world module 38 determines where and how the AR content 40 can be placed relative to the physical world, as determined at least in part based on the data input 36. The AR content is placed in the physical world by presenting the AR content in a manner that the user can see both the AR content and the physical world. For example, such an interface can be created with glasses through which the user can view the physical world, and such glasses can also be controlled so that virtual objects appear in controlled positions within the user's field of view. The AR content is rendered as if it is interacting with objects in the physical world. The user interface can obscure the user's view of objects in the physical world, thereby presenting the appearance that the AR content can obscure the user's view of these objects when appropriate. For example, the AR content can be placed by appropriately selecting a portion of an element 42 in the environment 32 to be displayed (e.g., a table); and displaying the AR content 40 with an appearance and positioning as if it were resting on the element 42 or otherwise interacting with the element 42. The AR content can also be placed within structures that are not yet within the field of view 44 or relative to the mapping grid model 46 of the physical world.

[0076] As shown, element 42 is an example of multiple elements in the physical world that may be considered fixed and stored as such in connected world module 38. Once stored in connected world module 38, information about these fixed elements may be used to present information to the user, such that user 30 may perceive content fixed to element 42 without the system having to map fixed element 42 each time user 30 views it. Thus, fixed element 42 may be a mapped mesh model from a previous modeling session, or determined by a separate user but still stored on connected world module 38 for future reference by multiple users. Thus, connected world module 38 may recognize environment 32 from a previously mapped environment and display AR content without requiring user 30's device to first map environment 32, thereby saving computational processing and cycles and avoiding delays in any rendered AR content.

[0077] Similarly, a mapping mesh model 46 of the physical world may be created by the AR display system, and appropriate surfaces and metrics for interacting and displaying AR content 40 may be mapped and stored in the connected world module 38 for future retrieval by the user 30 or other users without the need for re-mapping or modeling. In some embodiments, data input 36 is an input such as geographic location, user identification, and current activity that indicates to the connected world module 38 which of one or more fixed elements 42 is available, which AR content 40 was last placed on the fixed element 42, and whether to display the same content (such AR content is "persistent" content, regardless of the user viewing the particular connected world model).

[0078] Even in embodiments where objects are considered fixed, the connected world module 38 may be updated from time to time to account for the possibility of changes in the physical world. Models of fixed objects may be updated with very low frequency. Other objects in the physical world may be mobile or otherwise not considered fixed. In order to render a realistic AR scene, the AR system may update the positions of these non-fixed objects at a much higher frequency than for updating fixed objects. In order to be able to accurately track all objects in the physical world, the AR system may extract information from multiple sensors, including one or more image sensors.

[0079] Figure 3B 48 and its optional components. Fig.21 Specific configurations are described in . In some embodiments, when facing the user's eyes 49, two eye tracking cameras 50 detect metrics of the user's eyes 49, such as eye shape, eyelid occlusion, pupil direction, and flicker on the user's eyes 49. In some embodiments, one of the sensors may be a depth sensor 51, such as a time-of-flight sensor, which transmits signals to the world and detects reflections of those signals from nearby objects to determine the distance to a given object. For example, a depth sensor can quickly determine whether an object has entered the user's field of view due to its movement or a change in the user's posture. However, other sensors may be used to collect information about the position of objects in the user's field of view instead of or in addition to the user's field of view. In some embodiments, the world camera 52 records a view larger than the periphery to map the environment 32 and detect inputs that may affect AR content. In some embodiments, the world camera 52 and / or the camera 53 may be a grayscale and / or color image sensor that can output grayscale and / or color image frames at fixed time intervals. The camera 53 may also capture images of the physical world within the user's field of view at a specific time. The pixels of a frame-based image sensor may also be repeatedly sampled even if the values ​​of those pixels do not change. Each of the world camera 52, camera 53 and depth sensor 51 has a respective field of view 54, 55 and 56 to collect data from and record the physical world scene, such as Figure 3A The physical world environment 32 is shown in FIG.

[0080] Inertial measurement unit 57 may determine movement and / or orientation of viewing optical assembly 48. In some embodiments, each component is operably coupled to at least one other component. For example, depth sensor 51 may be operably coupled to eye tracking camera 50 to confirm the actual distance of a point and / or area in the physical world that the user's eye 49 is viewing.

[0081] It should be understood that the viewing optics assembly 48 may include Figure 3BSome of the components shown. For example, viewing optics assembly 48 may include a different number of components. In some embodiments, for example, viewing optics assembly 48 may include one world camera 52, two world cameras 52, or more world cameras in place of the four world cameras shown. Alternatively or additionally, cameras 52 and 53 need not capture visible light images of their full fields of view. Viewing optics assembly 48 may include other types of components. In some embodiments, viewing optics assembly 48 may include one or more dynamic vision sensors (DVS) whose pixels may asynchronously respond to relative changes in light intensity exceeding a threshold.

[0082] In some embodiments, the observation optical assembly 48 may not include a depth sensor 51 based on time-of-flight information. In some embodiments, for example, the observation optical assembly 48 may include one or more plenoptic cameras whose pixels can capture not only light intensity but also the angle of incident light. For example, a plenoptic camera may include an image sensor covered with a transmissive diffraction mask (TDM). Alternatively or additionally, a plenoptic camera may include an image sensor including angle-sensitive pixels and / or phase detection autofocus pixels (PDAF) and / or a microlens array (MLA). As an alternative or in addition to the depth sensor 51, such a sensor may be considered a source of depth information.

[0083] It should also be understood that Figure 3B The component configuration in is shown as an example. The viewing optical assembly 48 may include components having any suitable configuration so that the user can have the maximum field of view for a particular set of components. For example, if the viewing optical assembly 48 has a world camera 52, the world camera can be placed in the central area of ​​the viewing optical assembly rather than on the side.

[0084] Information from these sensors in the viewing optical assembly 48 can be coupled to one or more processors in the system. The processor can generate data that can be rendered so that the user perceives virtual content that interacts with objects in the physical world. The rendering can be implemented in any suitable manner, including generating image data showing both physical objects and virtual objects. In other embodiments, physical and virtual content can be shown in a scene by modulating the opacity of the display device through which the user views the physical world. The opacity can be controlled to create the appearance of virtual objects and also prevent the user from viewing objects in the physical world that are blocked by virtual objects. In some embodiments, the image data can only include virtual content that can be modified to realistically interact with the physical world (e.g., considering the clipped content of the blockage), which can be viewed through the user interface. No matter how the content is presented to the user, the physical world model can be used so that the characteristics of the virtual objects affected by the physical objects can be correctly calculated, including the shape, position, motion and visibility of the virtual objects.

[0085] The physical world model may be created based on data collected from sensors on a user's wearable device. In some embodiments, the model may be created based on data collected from multiple users, which may be aggregated in a computing device remote from all users (may be "in the cloud").

[0086] In some embodiments, at least one of the sensors may be configured to acquire information about physical objects in the scene, particularly non-stationary objects, at high frequency and low latency using compact, low-power components. The sensor may employ block tracking to limit the amount of data output.

[0087] Figure 4 An image sensing system 400 is shown in accordance with some embodiments. Image sensing system 400 may include image sensor 402, which may include image array 404, which may include a plurality of pixels, each pixel responsive to light, as in a conventional image sensor. Sensor 402 may also include circuitry to access each pixel. Accessing a pixel requires obtaining information about incident light generated by the pixel. Alternatively or additionally, accessing a pixel requires controlling the pixel, such as by configuring the pixel to provide an output only when a certain event is detected.

[0088] In the illustrated embodiment, the image array 404 is configured as an array having multiple rows and columns of pixels. In such an embodiment, the access circuit may be implemented as a row address encoder / decoder 406 and a column address encoder / decoder 408. The image sensor 402 may also include circuits that generate inputs to the access circuits to control the timing and order of reading information from the pixels in the image array 404. In the illustrated embodiment, the circuit is a block tracking engine 410. Compared to conventional image sensors that can continuously output image information captured by pixels in each row, the image sensor 402 can be controlled to output image information in a specified block. In addition, the positions of those blocks relative to the image array may change over a period of time. In the illustrated embodiment, the block tracking engine 410 may output image array access information to control the output of image information from the portion of the image array 404 corresponding to the block position, and the access information may change dynamically based on the motion of objects in the environment and / or the estimation of the motion of the image sensor relative to those objects.

[0089] In some embodiments, the image sensor 402 may have a dynamic vision sensor (DVS) function so that image information is provided by the sensor only when the image property (e.g., intensity) of the pixel changes. For example, the image sensor 402 may apply one or more thresholds that define the on and off states of the pixels. The image sensor can detect that the pixels have changed state and selectively provide outputs only for those pixels or those pixels in the block whose state has changed. These outputs can be provided asynchronously when detected, rather than as part of the readout of all pixels in the array. For example, the output can take the form of an address event representation (AER) 418, which can include a pixel address (e.g., row and column) and an event type (on or off). An on event can indicate that the pixel primitive at the corresponding pixel address senses an increase in light intensity; an off event can indicate that the pixel primitive at the corresponding pixel address senses a decrease in light intensity. The increase or decrease can be relative to an absolute level, or can be a change relative to a level at the last output from the pixel. For example, the change can be represented as a fixed offset or a percentage of the value at the last output from the pixel.

[0090] Using DVS technology in conjunction with tile tracking makes image sensors suitable for XR systems. When combined in an image sensor, the amount of data generated is limited to the data from pixel cells that are located within a tile and detect a change that will trigger an event output.

[0091] In some cases, high-resolution image information is required. However, when using DVS technology, large sensors with more than one million pixel units to generate high-resolution image information can generate a large amount of image information. The inventors have recognized and understood that DVS sensors can generate a large number of events, reflecting motion or image changes in the background that are not caused by the motion of the tracked object. Currently, the resolution of DVS sensors is limited to less than 1MB, such as 128x128, 240x180, 346x260 to limit the number of events generated. Such sensors sacrifice the resolution of the tracked object and may not be able to detect fine finger movements of the hand, for example. In addition, if the image sensor outputs image information in other formats, limiting the resolution of the sensor array to output a manageable number of events will also limit the image sensor to be used with the DVS function to generate high-resolution image frames. In some embodiments, the sensor described herein may have a resolution higher than VGA, including up to 8 megapixels or 12 megapixels. Nevertheless, block tracking as described herein can also be used to limit the number of events output by the image sensor per second. Therefore, an image sensor operating in at least two modes can be enabled. For example, an image sensor with a megapixel resolution may operate in a first mode where it outputs events in a particular tile being tracked. In a second mode, it may output a high-resolution image frame or portion of an image frame. Such an image sensor may be controlled in an XR system to operate in these different modes based on system functionality.

[0092] Image array 404 may include a plurality of pixel cells 500 arranged in an array. Figure 5A An example of a pixel primitive 500 is shown, which in this embodiment is configured for use in an imaging array implementing DVS technology. The pixel primitive 500 may include a photoreceptor circuit 502, a differential circuit 506, and a comparator 508. The photoreceptor circuit 502 may include a photodiode 504, which converts light that strikes the photodiode into a measurable electrical signal. In this example, the conversion is to a current I. A transconductance amplifier 510 converts the photocurrent I into a voltage. The conversion may be linear or nonlinear, such as according to a function of log I. Regardless of the specific transfer function, the output of the transconductance amplifier 510 indicates the amount of light detected at the photodiode 504. Although a photodiode is shown as an example, it should be understood that other photosensitive components that produce a measurable output in response to incident light may be implemented in the photoreceptor circuit in place of or in addition to the photodiode.

[0093] exist Figure 5AIn an embodiment of the invention, the circuitry for determining whether a change in the pixel output is sufficient to trigger an output of the pixel primitive is incorporated into the pixel itself. In this example, this functionality is implemented by a differential circuit 506 and a comparator 508. The differential circuit 506 can be configured to reduce DC mismatch between pixel primitives, for example, by balancing the output of the differential circuit to a reset level after a generated event. In this example, the differential circuit 506 is configured to generate an output that shows the change in the output of the photodiode 504 since the last output. The differential circuit can include an amplifier 512 with a gain of -A, a capacitor 514 that can be implemented as a single circuit element or one or more capacitors connected in a network, and a reset switch 516.

[0094] In operation, the pixel primitive is reset by temporarily closing switch 516. Such a reset occurs at the beginning of circuit operation and at any time after an event is detected. When the pixel 500 is reset, the voltage across capacitor 514 is such that when subtracted from the output of transconductance amplifier 510, a zero voltage is produced at the input of amplifier 512. When switch 516 is open, the output of transconductance amplifier 510 will be combined with the voltage drop across capacitor 514, and there will be zero voltage at the input of amplifier 512. The output of transconductance amplifier 510 changes due to changes in the amount of light illuminating photodiode 504. When the output of transconductance amplifier 510 increases or decreases, the output of amplifier 512 will swing positively or negatively by a change amount, which is amplified by the gain of amplifier 512.

[0095] The comparator 508 can determine whether an event is generated and the sign of the event by, for example, comparing the output voltage V of the differential circuit with a predetermined threshold voltage C. In some embodiments, the comparator 508 may include two comparators including transistors, one of which may operate when the output of the amplifier 512 shows a positive change and may detect an increased change (a switch-on event); when the output of the amplifier 512 shows a negative change, the other comparator may operate and may detect a decreased change (a switch-off event). However, it should be understood that the amplifier 512 may have a negative gain. In such an embodiment, an increase in the output of the transconductance amplifier 510 may be detected as a negative voltage change at the output of the amplifier 512. Similarly, it should be understood that the positive and negative voltages may be relative to ground or any suitable reference level. In any case, the value of the threshold voltage C may be controlled by the characteristics of the transistor (e.g., transistor size, transistor threshold voltage) and / or by the value of a reference voltage that may be applied to the comparator 508.

[0096] Figure 5BAn example of event output (on, off) of the pixel primitive 500 over time t is shown. In the example shown, at time t1, the output value of the differential circuit is V1; at time t2, the output value of the differential circuit is V2; and at time t3, the output value of the differential circuit is V3. Between time t1 and time t2, although the photodiode senses an increase in light intensity, the pixel primitive does not output any event because the change in V does not exceed the value of the threshold voltage C. At time t2, the pixel primitive outputs an on event because V2 is greater than V1 by the value of the threshold voltage C. Between time t2 and time t3, although the photodiode senses a decrease in light intensity, the pixel primitive does not output any event because the change in V does not exceed the value of the threshold voltage C. At time t3, the pixel primitive outputs an off event because V3 is less than V2 by the value of the threshold voltage C.

[0097] Each event may trigger an output at AER 418. The output may include, for example, an indication of whether the event is an on or off event and an identification of the pixel, such as its row and column. Other information may alternatively or additionally be included in the output. For example, a timestamp may be included, which is useful in situations where events are queued for later transmission or processing. As another example, the current level at the output of amplifier 510 may be included. Such information may be selectively included, for example, if further processing is to be performed in addition to detecting object motion.

[0098] It should be understood that the frequency of the event outputs, and therefore the sensitivity of the pixel primitive, can be controlled by the value of the threshold voltage C. For example, the frequency of the event outputs can be reduced by increasing the value of the threshold voltage C, or increased by reducing the value of the threshold voltage C. It should also be understood that the threshold voltage C can be different for the on event and the off event, for example by setting different reference voltages for the comparator used to detect the on event and the comparator used to detect the off event. It should also be understood that the pixel primitive can also output a value indicating the magnitude of the light intensity change as an alternative or supplement to the sign signal indicating event detection.

[0099] Figure 5A and Figure 5B Pixel primitive 500 is shown as an example according to some embodiments. Other designs may also be applicable to pixel primitives. In some embodiments, a pixel primitive may include a photoreceptor circuit and a differential circuit, but share a comparator circuit with one or more other pixel primitives. In some embodiments, a pixel primitive may include a circuit configured to calculate a change value, such as an active pixel sensor at the pixel level.

[0100] Regardless of how events are detected for each pixel primitive, the ability to configure pixels to output only when an event is detected can be used to limit the information required to maintain a position model of a non-fixed (i.e., movable) object. For example, a threshold voltage C that is triggered when a relatively small change occurs can be used to set the pixels within a block. Other pixels outside the block may have a larger threshold, such as three or five times the threshold. In some embodiments, the threshold voltage C of any pixel outside the block can be set large enough to effectively disable the pixel and not produce any output, regardless of the amount of change. In other embodiments, pixels outside the block can be disabled in other ways. In such embodiments, the threshold voltage is fixed for all pixels, but the pixel can be selectively enabled or disabled based on whether the pixel is located in a block.

[0101] In another embodiment, the threshold voltage for one or more pixels may be adaptively set in a manner that modulates the amount of data output from the image array. For example, the AR system may have the processing capability to process multiple events per second. When the number of events output per second exceeds an upper limit, the threshold of some or all pixels may be increased. Alternatively or additionally, when the number of events per second drops below a lower limit, the threshold may be lowered so that there is more data for more accurate processing. As a specific example, the number of events per second may be between 200 and 2,000 events. For example, compared to processing all pixel values ​​scanned from an image sensor (constituting 30 million or more pixel values ​​per second), this number of events constitutes a significant reduction in the amount of data to be processed per second. This number of events is even less than processing only pixels within a block (which may be lower, but may still be tens of thousands or more pixel values ​​per second).

[0102] The control signals used to enable and / or set the threshold voltage for each of the plurality of pixels may be generated in any suitable manner. However, in the illustrated embodiment, those control signals are set by the block tracking engine 410 or based on processing within the processing module 72 or other processor.

[0103] Return to reference Figure 4 , the image sensing system 400 may receive input from any suitable component such that the block tracking engine 410 may dynamically select at least one region of the image array 404 to be enabled and / or disabled based at least on the received input in order to implement the block. The block tracking engine 410 may be a digital processing circuit having a memory storing one or more parameters for the block. The parameters may be, for example, the boundaries of the block, and may include other information such as information related to a scaling factor between the image array motion and the motion within the image array of the movable object image associated with the block. The block tracking engine 410 may also include circuitry configured to perform calculations on the stored values ​​and other measurements provided as input.

[0104] In the illustrated embodiment, the tile tracking engine 410 receives as input a specification of a current tile. The tile may be specified based on its size and location within the image array 404, such as by specifying a range of row and column addresses for the tile. This specification may be used as a parameter in the processing module 72 ( Figure 2 ) or other component processing information about the physical world. For example, the processing module 72 may specify that a tile contains the current position of each movable object or a subset of the tracked movable objects in the physical world so as to render virtual objects with appropriate appearance relative to the physical world. For example, if the AR scene includes a toy doll balanced on a physical object, such as a moving toy car, as a virtual object, a tile may be specified to contain the toy car. A tile may not be specified for another toy car moving in the background because there is less need for up-to-date information about that object to render a realistic AR scene.

[0105] Regardless of the selection of the tiles, information about the current location of the tiles is provided to the tile tracking engine 410. In some embodiments, the tiles may be rectangular so that the location of the tiles may be specified simply as a start row / column and an end row / column. In other embodiments, the tiles may have other shapes, such as circular, and the tiles may be specified in other ways, such as by a center point and a radius.

[0106] In some embodiments, trajectory information about the tiles may also be provided. For example, the trajectory may specify the movement of the tiles relative to the coordinates of the image array 404. For example, the processing module 72 may construct a model of the movement of movable objects in the physical world and / or the movement of the image array 404 relative to the physical world. Since either or both of the above movements may affect the position of the projected object image in the image array 404, the trajectory of the tiles in the image array 404 may be calculated based on either or both of the above movements. The trajectory may be specified in any suitable manner, such as parameters of a linear equation, a quadratic equation, a cubic equation, or other polynomial equation.

[0107] In other embodiments, the tile tracking engine 410 may dynamically calculate the tile positions based on input from sensors that provide information about the physical world. The information from these sensors may be provided directly from the sensors. Alternatively or additionally, the sensor information may be processed to extract information about the physical world before being provided to the tile tracking engine 410. For example, the extracted information may include the motion of the image array 404 relative to the physical world, the distance between the image array 404 and the object whose image falls within the tile, or other information that may be used to dynamically align the tiles in the image array 404 with the object's image in the physical world as the image array 404 and / or the object moves.

[0108] Examples of input components may include image sensors 412 and inertial sensors 414. Examples of image sensors 412 may include eye tracking cameras 50, depth sensors 51, world cameras 52, and / or cameras 52. Examples of inertial sensors 414 may include inertial measurement units 57. In some embodiments, the input components may be selected to provide data at a relatively high rate. For example, the inertial measurement unit 57 may have an output rate between 200 and 2,000 measurements per second, such as between 800 and 1,200 measurements per second. The tile positions may be updated at a similarly high rate. As a specific example, by using the inertial measurement unit 57 as an input source to the tile tracking engine 410, the tile positions may be updated 800 to 1,200 times per second. In this way, movable objects may be tracked with high accuracy using relatively small tiles that limit the number of events that need to be processed. This approach may result in very low latency between changes in the relative position of the image sensor and the movable object, and equally low latency in updating virtual object rendering, thereby providing an ideal user experience.

[0109] In some cases, the movable objects tracked using tiles can be stationary objects in the physical world. For example, the AR system can identify stationary objects by analyzing multiple images of the physical world taken, and select one or more features of the stationary objects as reference points to determine the movement of a wearable device having an image sensor thereon. Frequent, low-latency position updates of these reference points relative to the sensor array can be used to provide frequent, low-latency calculations of the head pose of the wearable device user. Since the head pose can be used to realistically render virtual objects via a user interface on the wearable device, the frequent, low-latency updates of the head pose improve the user experience of the AR system. Therefore, having the input of the tile tracking engine 410 that controls the tile position come only from sensors with a high output rate, such as one or more inertial measurement units, can result in an ideal AR system user experience.

[0110] However, in some embodiments, other information may be provided to the tile tracking engine 410 to enable it to calculate trajectories and / or apply trajectories to tiles. This other information may include stored information 416, such as the connected world module 38 and / or the mapping grid model 46. This information may indicate one or more previous positions of the object relative to the physical world, so that changes in these previous positions and / or changes in the current position relative to previous positions may indicate object trajectories within the physical world, which are then mapped to tile trajectories across the image array 404. Other information in the physical world model may be used alternatively or additionally. For example, the size and / or distance of a movable object or other information about the position relative to the image array 404 may be used to calculate the tile position or trajectory across the image array 404 associated with the object.

[0111] Regardless of how the trajectory is determined, tile tracking engine 410 can apply the trajectory to calculate tile update positions within image array 404 at a high rate, such as faster than once per second or more than 800 times per second. In some embodiments, this rate may be limited by processing power and therefore be less than 2,000 times per second.

[0112] It should be understood that the process of tracking the changes of movable objects is not sufficient to reconstruct the complete physical world. However, the time interval for reconstructing the physical world is longer than the time interval between the updates of the positions of movable objects, such as every 30 seconds or every 5 seconds. When reconstructing the physical world, the positions of the objects to be tracked and the positions of the blocks that capture information about those objects can be recalculated.

[0113] Figure 4 An embodiment is shown in which the processing circuitry that both dynamically generates the tiles and controls the selective output of image information from the tiles is configured to directly control the image array 404 so as to limit the image information output from the array to selected information. For example, such circuitry may be integrated into the same semiconductor chip that houses the image array 404, or may be integrated into a separate controller chip for the image array 404. However, it should be understood that the circuitry that generates control signals for the image array 404 may be distributed throughout the XR system. For example, some or all of the functions may be performed by programming in the processing module 72 or other processor within the system.

[0114] Image sensing system 400 may output image information for each of a plurality of pixels. Each pixel of the image information corresponds to one of the pixel primitives of image array 404. The output image information from image sensing system 400 may be image information for each of one or more tiles corresponding to at least one region of image array 404 selected by tile tracking engine 410. In some embodiments, such as when each pixel of image array 404 has a different Figure 5A As configured as shown, pixels in the output image information may identify pixels where a change in light intensity was detected by image sensor 400 within one or more tiles.

[0115] In some embodiments, the output image information from the image sensing system 400 may be image information of pixels outside each of the one or more tiles corresponding to at least one area of ​​the image array selected by the tile tracking engine 410. For example, a deer may be running in a physical world with a flowing river. The details of the river's waves may not be of interest but may trigger pixel primitives of the image array 402. The tile tracking engine 410 may create a tile surrounding the river and disable a portion of the image array 402 corresponding to the tile surrounding the river.

[0116] Further processing may be performed based on identifying pixel changes. For example, the portion of the world model corresponding to the portion of the physical world imaged by the changed pixel may be updated. These updates may be performed based on information collected with other sensors. In some embodiments, further processing is conditioned on or triggered by a number of changed pixels in a tile. For example, updates may be performed once 10% or some other threshold amount of pixels in a tile are detected to have changed.

[0117] In some embodiments, image information in other formats may be output from the image sensor and may be used in conjunction with the change information to update the world model. In some embodiments, the format of the image information output from the image sensor may change from time to time during operation of the VR system. In some embodiments, for example, the pixel primitive 500 is operable to produce a differential output at certain times, such as produced in the comparator 508. The output of the amplifier 510 is switchable to output the amount of light incident on the photodiode 504 at other times. For example, the output of the amplifier 510 is switchably connected to a sense line, which in turn is connected to an A / D converter, which may provide a digital indication of the amount of incident light based on the output of the amplifier 510.

[0118] An image sensor in this configuration can be operated as part of an AR system to perform differential output most of the time, outputting events only for pixels where a change above a threshold is detected, or outputting events only for pixels within a block where a change above a threshold is detected. A full image frame with amplitude information for all pixels in the image array can be output periodically, such as every 5 to 30 seconds. In this way, low-latency, accurate processing can be achieved, where differential information is used to quickly update selected parts of the world model whose changes are most likely to affect the user's perception of what is happening, while the majority of the world model is updated more using the full image. Although the full update of the world model is performed only at a slower rate, any model update delay cannot have a meaningful impact on the user's perception of the AR scene.

[0119] The output mode of the image sensor may be changed from time to time throughout operation of the image sensor, such that the sensor outputs one or more of intensity information for some or all pixels and indications of changes for some or all pixels in the array.

[0120] It is not required to selectively output image information from the tiles from the image sensor by limiting the information output from the image array. In some embodiments, image information may be output from all pixels in the image array, and only information about specific areas of the array may be output from the image sensor. Figure 6An image sensor 600 is shown according to some embodiments. The image sensor 600 may include an image array 602. In this embodiment, the image array 602 may be similar to a conventional image array that scans out rows and columns of pixel values. The operation of such an image array may be adapted by other components. The image sensor 600 may also include a block tracking engine 604 and / or a comparator 606. The image sensor 600 may provide an output 610 to an image processor 608. For example, the processor 608 may be a processing module 72 ( Figure 2 ) part.

[0121] The block tracking engine 604 may have a structure and function similar to the block tracking engine 410. It may be configured to receive a signal specifying at least one selected area of ​​the image array 602, and then generate a control signal specifying the dynamic position of the area based on the calculated trajectory within the image array 602 of the object image represented by the area. In some embodiments, the block tracking engine 604 may receive a signal specifying at least one selected area of ​​the image array 602, and the signal may include trajectory information for the one or more areas. The block tracking engine 604 may be configured to perform calculations to dynamically identify pixel primitives within the at least one selected area based on the trajectory information. The implementation of the block tracking engine 604 may vary. For example, the block tracking engine may update the block position based on a sensor indicating the motion of the image array 602 and / or the projected motion of an object associated with the block.

[0122] exist Figure 6 In the illustrated embodiment, the image sensor 600 is configured to output differential information of pixels within the identified blocks. The comparator 606 may be configured to receive control signals from the block tracking engine 604 that identify the pixels within the blocks. The comparator 606 may selectively operate on pixels output from the image array 602 that have addresses within the blocks as indicated by the block tracking engine 604. The comparator 606 may operate on pixel primitives to generate signals indicating changes in sensed light detected by at least one region of the image array 602. As an example of an implementation, the comparator 606 may include a storage element that stores reset values ​​for pixel primitives within the array. When the current values ​​of those pixels are scanned out of the image array 602, the circuitry within the comparator 606 may compare the stored values ​​with the current values ​​and output an indication when the difference exceeds a threshold. For example, digital circuits may be used to store values ​​and perform such comparisons. In this example, the output of the image sensor 600 may be processed like the output of the image sensor 400.

[0123] In some embodiments, the image array 602, the block tracking engine 604, and the comparator 606 may be implemented in a single integrated circuit, such as a CMOS integrated circuit. In some embodiments, the image array 602 may be implemented in a single integrated circuit. The block tracking engine 604 and the comparator 606 may be implemented in another single integrated circuit, which is configured, for example, as a driver for the image array 602. Alternatively or additionally, some or all of the functionality of the block tracking engine and / or the comparator 606 may be distributed to other digital processors within the AR system.

[0124] Other configurations or processing circuits are possible. Figure 7 An image sensor 700 is shown according to some embodiments. Image sensor 700 may include image array 702. In this embodiment, image array 702 may include pixel primitives having a differential configuration, such as Figure 5A 500 in FIG. However, the embodiments herein are not limited to differential pixel primitives, since block tracking may be implemented using an image sensor that outputs intensity information.

[0125] exist Figure 7 In the illustrated embodiment, a block tracking engine 704 generates control signals indicating addresses of pixel primitives within one or more blocks being tracked. The block tracking engine 704 can be constructed and operate similarly to the block tracking engine 604. Here, the block tracking engine 704 provides control signals to a pixel filter 706, which passes image information from only those pixels within the block to an output 710. As shown, the output 710 is coupled to an image processor 708, which further processes the image information of the pixels within the block using the techniques described herein or in other suitable manners.

[0126] Further changes in Figure 8 , which shows an image sensor 800 according to some embodiments. Image sensor 800 may include an image array 802, which may be a conventional image array that scans out pixel intensity values. The image array may be adapted to provide differential image information as described herein using a comparator 806. Similar to comparator 606, comparator 806 may calculate differential information based on stored pixel values. Selected values ​​of those differential values ​​may be passed to output 812 by pixel filter 808. Like pixel filter 706, pixel filter 808 may receive control inputs from block tracking engine 804. Block tracking engine 804 may be similar to block tracking engine 704. Output 812 may be coupled to image processor 810. Some or all of the above components of image sensor 800 may be implemented in a single integrated circuit. Alternatively, these components may be distributed across one or more integrated circuits or other components.

[0127] The image sensors described herein may operate as part of an augmented reality system to maintain information about movable objects or other information about the physical world that is combined with information about the physical environment to realistically render images of virtual objects. Fig. 9 A method 900 for image sensing is shown in accordance with some embodiments.

[0128] At least a portion of method 900 may be performed to operate an image sensor, such as image sensor 400, 600, 700, or 800. Method 900 begins by receiving (act 902) imaging information from one or more inputs, such as image sensor 412, inertial sensor 414, and stored information 416. Method 900 may include identifying (act 904) one or more patches on an image output of the image sensing system based at least in part on the received information. Fig.10 An example of act 904 is shown. In some embodiments, method 900 may include calculating (act 906) movement trajectories for one or more tiles. Fig.11 An example of action 906 is shown.

[0129] The method 900 may also include setting (action 908) the image sensing system based at least in part on the identified one or more blocks and / or their estimated movement trajectories. For example, the setting may be achieved by enabling a portion of the pixel primitives of the image sensing system based at least in part on the identified one or more blocks and / or their estimated movement trajectories through the comparator 606, the pixel filter 706, etc. In some embodiments, the comparator 606 may receive a first reference voltage value for the pixel primitive corresponding to the selected block on the image, and a second reference voltage value for the pixel primitive not corresponding to any selected block on the image. The comparator 606 may set the second reference voltage to be much higher than the first reference voltage, so that an unreasonable light intensity change sensed by the pixel primitive including the comparator unit with the second reference voltage may result in the output of the pixel primitive. In some embodiments, the pixel filter 706 may disable the output from the pixel primitive having an address (e.g., row and column) that does not correspond to any selected block on the image.

[0130] Fig.10 Block identification 904 is shown in accordance with some embodiments. Block identification 904 may include segmenting (act 1002) one or more images from one or more inputs based at least in part on color, light intensity, angle of arrival, depth, and semantics.

[0131] Block recognition 904 may also include recognizing (action 1004) one or more objects in one or more images. In some embodiments, object recognition 1004 may be based at least in part on predetermined features of an object, including, for example, hand, eye, facial features. In some embodiments, object recognition 1004 may be based on one or more virtual objects. For example, a virtual animal character is walking on a physical pencil. Object recognition 1004 may target the virtual animal character as an object. In some embodiments, object recognition 1004 may be based at least in part on artificial intelligence (AI) training received by the image sensing system. For example, the image sensing system may be trained by reading images of cats of different types and colors, thereby learning the characteristics of a cat and being able to recognize a cat in the physical world.

[0132] Tile identification 904 may include generating (act 1006) tiles based on one or more objects. In some embodiments, object tile 1006 may generate tiles by computing a convex hull or bounding box of one or more objects.

[0133] Fig.11 Block trajectory estimation 906 is shown according to some embodiments. Block trajectory estimation 906 may include predicting (action 1102) the movement of one or more blocks over a period of time. The movement of one or more blocks may be caused by a variety of reasons, including, for example, a moving object and / or a moving user. Motion prediction 1102 may include deriving a moving speed of a moving object and / or a moving user based on received images and / or received AI training.

[0134] Block trajectory estimation 906 may include calculating (action 1104) a trajectory of one or more blocks over a period of time based at least in part on the predicted movement. In some embodiments, the trajectory may be calculated by modeling with a first order linear equation, assuming that the moving object will continue to move in the same direction at the same speed. In some embodiments, the trajectory may be calculated by curve fitting or using heuristics, including pattern detection.

[0135] Fig.12 and Fig.13 Factors that can be applied in the block trajectory calculation are shown. Fig.12 An example of a movable object is shown, in which the movable object is a moving object 1202 (e.g., a hand) that moves relative to a user of the AR system. In this example, the user wears an image sensor as part of a head mounted display 62. In this example, the user's eye 49 is looking straight ahead so that the image array 1200 captures the field of view (FOV) of the eye 49 relative to one viewpoint 1204. The object 1202 is in the FOV and therefore appears in the corresponding pixel in the array 1200 by creating intensity variations.

[0136] Array 1200 has a plurality of pixels 1208 arranged in an array. For a system tracking hand 1202, a block 1206 in the array containing object 1202 at time t0 may include a portion of the plurality of pixels. If object 1202 is moving, the position of the block that captures the object will change over time. This change may be captured in a block track, from block 1206 to blocks X and Y used later.

[0137] The segment trajectory may be estimated by identifying features 1210 of an object in the segment, such as a fingertip in the illustrated example, such as in action 906. A motion vector 1212 of the feature may be calculated. In this example, the trajectory is modeled as a first order linear equation, and the prediction is based on the object 1202 continuing to be on the same segment trajectory 1214 over a period of time, resulting in an assumption of segment positions X and Y at each of two consecutive times.

[0138] As the position of the block changes, the image of the moving object 1202 remains within the block. Even if the image information is limited to information collected with pixels within the block, the image information is sufficient to represent the motion of the moving object 1202. This is true whether the image information is intensity information or differential information generated by a differential circuit. For example, in the case of a differential circuit, when the image of the moving object 1202 moves across a pixel, an event indicating an increase in intensity may occur. Conversely, when the image of the moving object 1202 moves away from a pixel, an event indicating a decrease in intensity occurs. A pixel pattern with increasing and decreasing events can be used as a reliable indication of the motion of the moving object 1202, and since the amount of data indicating the event is relatively small, the motion can be updated quickly with low latency. As a specific example, such a system can result in a realistic XR system that tracks a user's hand and changes the rendering of a virtual object, thereby creating a sense for the user that the user is interacting with the virtual object.

[0139] The tile positions may change for other reasons, and any or all of them are reflected in the trajectory calculation. One such other change is the movement of the user while the user is wearing the image sensor. Fig.13 An example of a moving user is shown, which creates a changing viewpoint for the user as well as the image sensor. Fig.13 13, the user initially looks directly at the object from viewpoint 1302. In this configuration, pixel array 1300 of the image array will capture the object in front of the user. The object in front of the user may be in block 1312.

[0140] The user then changes the viewpoint, such as by turning their head. The viewpoint may change to viewpoint 1304. Even though the object that was previously directly in front of the user has not moved, it will have a different location within the user's field of view at viewpoint 1304. It will also appear at a different point within the field of view of the image sensor worn by the user, and therefore at a different location within image array 1300. For example, the object may be contained within the tile at location 1314.

[0141] If the user further changes their viewpoint to viewpoint 1306, and the image sensor moves with the user, the location of the object that was previously directly in front of the user will be imaged at a different point within the field of view of the image sensor worn by the user, and therefore at a different location within image array 1300. For example, the object may be contained within the tile at location 1316.

[0142] It can be seen that as the user further changes his viewpoint, the tile position in the image array required to capture the object moves further. The trajectory of this movement from position 1312 to position 1314 to position 1316 can be estimated and used to track future tile positions.

[0143] Trajectories may be estimated in other ways. For example, measurements with inertial sensors may indicate the acceleration and velocity of the user's head when the user has viewpoint 1302. This information may be used to predict the trajectory of a tile within the image array based on the user's head motion.

[0144] Tile trajectory estimation 906 may predict, based at least in part on these inertial measurements, that the user will have viewpoint 1304 at time t1 and viewpoint 1306 at time t2. Thus, tile trajectory estimation 906 may predict that tile 1308 moves to tile 1310 at time t1 and to tile 1312 at time t2.

[0145] As an example of this approach, it can be used to provide accurate and low latency head pose estimation in an AR system. The tiles can be positioned so as to contain images of stationary objects within the user's environment. As a specific example, processing of the image information may identify a corner of a picture frame hanging on a wall as a recognizable stationary object to be tracked. The processing may center the tile on the object. Combined with the above Fig.12As with the moving object 1202 described herein, relative motion between the object and the user's head will generate events that can be used to calculate relative motion between the user and the tracked object. In this example, since the tracked object is stationary, the relative motion indicates motion of the imaging array worn by the user. Therefore, the motion indicates changes in the user's head pose relative to the physical world and can be used to maintain an accurate calculation of the user's head pose, which can be used to realistically render virtual objects. Because the imaging arrays described herein can provide fast updates, the amount of data for each update is relatively small, so the calculations for rendering virtual objects remain accurate (they can be performed quickly and updated frequently).

[0146] Return to reference Fig.11 , the block trajectory estimation 906 may include adjusting (action 1106) the size of at least one block based at least in part on the calculated block trajectory. For example, the block size may be set to be large enough to include pixels of an image of a movable object or at least a portion of an object on which image information is generated. The block may be set to be slightly larger than the projected size of the image of the portion of the object of interest so that if there is any error in estimating the block trajectory, the block may still include the relevant portion of the image. When an object moves relative to the image sensor, the image size (in pixels) of the object may change based on distance, angle of incidence, orientation of the object, or other factors. A processor that defines blocks associated with an object may set the block size, such as by measuring the size of the object-related block based on other sensor data or calculating the size of the object-related block based on a world model. Other parameters of the block, such as its shape, may be similarly set or updated.

[0147] Fig.14 1400 is shown configured for an XR system according to some embodiments. Figure 4 ), the image sensing system 1400 includes circuitry to selectively output values ​​within a block, and, as also described above, can be configured to output events for pixels within a block. In addition, the image sensing system 1400 is configured to selectively output measured intensity values, which can be output for a complete image frame.

[0148] In the illustrated embodiment, separate outputs are shown for events and intensity values ​​generated using the DVS techniques described above. Output generated using the DVS techniques may be represented as AER 1418 using the representation output described above in conjunction with AER 418. Output representing intensity values ​​may be output via outputs designated herein as APS 1420. These intensity outputs may be used for blocks or for the entire image frame. The AER and APS outputs may be activated simultaneously. However, in the illustrated embodiment, the image sensor 1400 operates in either a mode of outputting events or a mode of outputting intensity information at any given time. A system in which such an image sensor is used may selectively use the event output and / or the intensity information.

[0149] Image sensing system 1400 may include image sensor 1402, which may include image array 1404, which may include a plurality of pixels 1500, each pixel responsive to light. Sensor 1402 may also include circuitry to access pixel elements. Sensor 1402 may also include circuitry to generate inputs to the access circuitry to control a mode of reading out information from pixel elements in image array 1404.

[0150] In the illustrated embodiment, the image array 1404 is configured as an array having multiple rows and columns of pixel primitives that can be accessed via two readout modes. In such an embodiment, the access circuitry may include a row address encoder / decoder 1406, a column address encoder / decoder 1408 that controls a column select switch 1422, and / or a register 1424 that may temporarily hold information about incident light sensed by one or more corresponding pixel primitives. A tile tracking engine 1410 may generate inputs to the access circuitry to control which pixel primitives provide image information at any time.

[0151] In some embodiments, image sensor 1402 may be configured to operate in a rolling shutter mode and / or a global shutter mode. For example, block tracking engine 1410 may generate inputs to access circuits to control a readout mode of image array 1402 .

[0152] When the sensor 1402 is operated in a rolling shutter readout mode, a single column of pixel primitives is selected during each system clock by, for example, closing a single column switch 1422 of the plurality of column switches. During the system clock, the selected column of pixel primitives is exposed and read out to the APS 1420. To generate an image frame using the rolling shutter mode, the columns of pixel primitives in the sensor 1402 may be read out column by column and then processed by an image processor to generate an image frame.

[0153] When the sensor 1402 is operated in a global shutter mode, for example, columns of pixel primitives are simultaneously exposed in a single system clock and the information is saved in registers 1424, so that the information captured by the pixel primitives in multiple columns can be read out to the APS 1420b simultaneously. This readout mode allows the image frame to be directly output without further data processing. In the example shown, information about the incident light sensed by the pixel primitive is saved in the corresponding register 1424. It should be understood that multiple pixel primitives can share one register 1424.

[0154] In some embodiments, the sensor 1402 may be implemented in a single integrated circuit, such as a CMOS integrated circuit. In some embodiments, the image array 1404 may be implemented in a single integrated circuit. The block tracking engine 1410, the row address encoder / decoder 1406, the column address encoder / decoder 1408, the column selection switch 1422, and / or the register 1424 may be implemented in another single integrated circuit, which is configured, for example, as a driver for the image array 1404. Alternatively or additionally, some or all of the functions of the block tracking engine 1410, the row address encoder / decoder 1406, the column address encoder / decoder 1408, the column selection switch 1422, and / or the register 1424 may be distributed to a digital processor in other AR systems.

[0155] Fig.15 An exemplary pixel primitive 1500 is shown. In the illustrated embodiment, each pixel primitive may be configured to output either event or intensity information. However, it should be understood that in some embodiments, the image sensor may be configured to output both types of information simultaneously.

[0156] Both the event information and the intensity information are based on the output of the photodetector 504, as described above in conjunction with FIG. 5. The pixel primitive 1500 includes a circuit for generating the event information. As also described above, the circuit includes a photoreceptor circuit 502, a differential circuit 506, and a comparator 508. When in a first state, the switch 1520 connects the photodetector 504 to the event generating circuit. The switch 1520 or other control circuit can be controlled by a processor controlling the AR system, thereby providing a relatively small amount of image information over a relatively long period of time when the AR system is running.

[0157] The switch 1520 or other control circuitry may also be controlled to configure the pixel primitive 1500 to output intensity information. In the information shown, the intensity information is provided as a complete image frame that is continuously represented as a stream of pixel intensity values ​​for each pixel in the image array. To operate in this mode, the switch 1520 in each pixel primitive may be set in a second position that exposes the output of the photodetector 504 after passing through the amplifier 510 so that it can be connected to an output line.

[0158] In the illustrated embodiment, the output lines are shown as column lines 1510. There may be one such column line for each column in the image array. Each pixel primitive in the column may be coupled to a column line 1510, but the pixel array may be controlled so that one pixel primitive is coupled to a column line 1510 at a time. Switch 1530, where there is one such switch in each pixel primitive, controls when the pixel primitive 1500 is connected to its corresponding column line 1510. Access circuitry, such as row address decoder 410, may close switch 1530 to ensure that only one pixel primitive is connected to each column line at a time. Switches 1520 and 1530 may be implemented using one or more transistors that are part of an image array or similar component.

[0159] Fig.15 Another component that may be included in each pixel primitive according to some embodiments is shown. A sample and hold circuit (S / H) 1532 may be connected between the photodetector 504 and the column line 1510. When present, the S / H 1532 may cause the image sensor 1402 to operate in a global shutter mode. In global shutter mode, a trigger signal is sent to each pixel primitive in the array at the same time. Within each pixel primitive, the S / H 1532 captures a value indicating intensity at the time of the trigger signal. The S / H 1532 stores the value and generates an output based on the value until the next value is captured.

[0160] like Fig.15 As shown, when switch 1530 is closed, a signal representing the value stored by S / H 1532 can be coupled to column line 1510. The signal coupled to the column line can be processed to generate the output of the image array. For example, the signal can be buffered and / or amplified in amplifier 1512 at the end of column line 1510 and then applied to analog-to-digital converter (A / D) 1514. The output of A / D 1514 can be passed to output 1420 through other readout circuit 1516. Readout circuit 1516 can include, for example, column switch 1422. Other components within readout circuit 1516 can perform other functions, such as serializing the multi-bit output of A / D 1514.

[0161] Those skilled in the art will understand how to implement circuits to perform the functions described herein. For example, S / H 1532 may be implemented as one or more capacitors and one or more switches. However, it should be understood that S / H 1532 may use other components or be implemented in a manner different from that described herein. Fig.15 A. It should be understood that other components other than those shown can also be implemented. For example, Fig.15 One amplifier and one A / D converter per column is indicated. In other embodiments, there is one A / D converter shared across multiple columns.

[0162] In a pixel array configured for a global shutter, each S / H 1532 can store intensity values ​​reflecting image information at the same time. The values ​​stored in each pixel can be stored during the readout phase as they are read out continuously. For example, continuous readout can be achieved by connecting the S / H 1532 of each pixel primitive in a row to its corresponding column line. The values ​​on the column line can then be passed to the APS output 1420 one at a time. This information flow can be controlled by sequencing the opening and closing of the column switches 1422. For example, the operation can be controlled by the column address decoder 1408. Once the value of each pixel in a row is read out, the pixel primitives in the next row can be connected to the column line in situ. These values ​​can be read out one column at a time. The process of reading out the values ​​of a row at a time can be repeated until the intensity values ​​of all pixels in the image array are read out. In an embodiment where the intensity values ​​of one or more blocks are read out, the process will be completed when the values ​​of the pixel primitives within the block are read out.

[0163] The pixel primitives may be read out in any suitable order. For example, the rows may be interlaced so that they may be read out sequentially in alternate rows. Nevertheless, the AR system may still process the image data into a frame of image data by deinterlacing the data.

[0164] In embodiments where S / H 1532 is not present, values ​​may still be read sequentially from each pixel primitive as rows and columns of values ​​are scanned out. However, the value read from each pixel primitive may represent the light intensity detected at the photodetector of that primitive when the value in the primitive was captured as part of the readout process, such as when the value was applied to A / D 1514. Thus, in a rolling shutter, the pixels of an image frame may represent images incident on the image array at slightly different times. For an image sensor that outputs an entire frame at a 30 Hz rate, the time difference between the first pixel value of a captured frame and the last pixel value of a captured frame may differ by 1 / 30 of a second, which is imperceptible for many applications.

[0165] For certain XR functions, such as tracking objects, the XR system can perform calculations using image information collected by an image sensor using a rolling shutter. Such calculations can be interpolated between consecutive image frames to calculate an interpolated value for each pixel, which represents an estimated pixel value at a point in time between consecutive frames. The same time can be used for all pixels so that, via calculation, the interpolated image frame contains pixels representing the same point in time, such as can be produced by an image sensor with a global shutter. Alternatively, a global shutter image array can be used for one or more image sensors in a wearable device that forms part of the XR system. A global shutter for a full or partial image frame can avoid interpolating other processes that can be performed to compensate for capture time variations in image information captured with a rolling shutter. Thus, interpolation calculations can be avoided even if the image information is used to track the movement of an object, such as may occur when tracking a hand or other movable object, or determining the head pose of a user of a wearable device in AR, or even when using a camera on a wearable device that moves when collecting image information to build an accurate representation of the physical environment.

[0166] Differentiated pixel primitives

[0167] In some embodiments, each pixel primitive in the sensor array may be identical. For example, each pixel primitive may respond to a broad spectrum of visible light. Thus, each photodetector may provide image information indicative of the intensity of visible light. In this case, the image array output may be a "grayscale" output representing the amount of visible light incident on the image array.

[0168] In other embodiments, the pixel primitives may be differentiated. For example, different pixel primitives in the sensor array may output image information indicating the intensity of light in a particular portion of the spectrum. A suitable technique for distinguishing pixel primitives is to place a filter element in the light path leading to the photodetector in the pixel primitive. The filter element may be a bandpass filter element, for example, allowing visible light of a particular color to pass. Applying such a color filter on a pixel primitive configures the pixel primitive to provide image information indicating the intensity of light of the color corresponding to the color filter.

[0169] Filters can be applied to pixel primitives regardless of their structure. For example, these filters can be applied to pixel primitives in a sensor array with a global shutter or a rolling shutter. Likewise, filters can be applied to pixel primitives that are configured to output intensity or intensity changes using DVS techniques.

[0170] In some embodiments, a filter element that selectively passes primary color light may be mounted on a photodetector in each pixel primitive in the sensor array. For example, a filter that selectively passes red, green, or blue light may be used. The sensor array may have multiple subarrays, each with one or more pixels configured to sense light of each primary color. In this way, the pixel primitives in each subarray provide both intensity and color information about the object being imaged by the image sensor.

[0171] The inventors have recognized and understood that in an XR system, some functions require color information, while some functions can be performed with grayscale information. A wearable device equipped with an image sensor to provide image information for operating an XR system may have multiple cameras, some of which may be formed with image sensors capable of providing color information. Other cameras may be grayscale cameras. The inventors have recognized and understood that a grayscale camera representing the same physical world range with the same resolution as a camera formed with a comparable image sensor configured to sense color consumes less power, is more sensitive in low light conditions, outputs data faster, and / or outputs less data. However, the image information output by a grayscale camera is sufficient for many functions performed in an XR system. Therefore, an XR system may be configured with both grayscale cameras and color cameras, primarily using one or more grayscale cameras, and selectively using color cameras.

[0172] For example, an XR system may collect and process image information to create a connected world model. This processing may use color information, which can enhance the effectiveness of certain functions, such as distinguishing objects, identifying surfaces associated with the same object, and / or identifying objects. Such processing is performed or updated from time to time, such as when a user first turns on the system, moves to a new environment such as walking into another room, or otherwise detects a change in the user's environment.

[0173] Other functions may not be significantly improved by using color information. For example, once a connected world model is created, the XR system may use images from one or more cameras to determine the orientation of the wearable device relative to features in the connected world model. For example, this functionality may be accomplished as part of head pose tracking. Some or all of the cameras used for such functionality may be grayscale. Because head pose tracking is frequently performed while the XR system is running, in some embodiments, using one or more grayscale cameras for this functionality can significantly save power, reduce computation, or provide other advantages.

[0174] Similarly, at various points in time during operation of the XR system, the system may use stereo information from two or more cameras to determine the distance to a movable object. This functionality requires processing image information at a high rate as part of tracking a user's hands or other movable objects. Using one or more grayscale cameras for this functionality may reduce the low latency associated with processing high-resolution image information or provide other advantages.

[0175] In some embodiments of an XR system, the XR system may have both a color camera and at least one grayscale camera, and may selectively enable the grayscale and / or color camera based on the functionality that requires use of image information from that camera.

[0176] Pixel primitives in an image sensor may be differentiated in a manner other than based on the spectrum to which the pixel primitives are sensitive. In some embodiments, some or all pixel primitives may produce an output having an intensity that indicates the angle of arrival of light incident on the pixel primitive. The angle of arrival information may be processed to calculate the distance to the imaged object.

[0177] In such an embodiment, the image sensor may passively acquire depth information. Passive depth information may be acquired by placing components in the optical path to a pixel element in the array so that the information output by the pixel element indicates the angle of arrival of light impinging on the pixel element. An example of such a component is a transmissive diffraction mask (TDM) filter.

[0178] The angle of arrival information can be converted into distance information by calculation, indicating the distance to the object reflecting the light. In some embodiments, pixel primitives configured to provide angle of arrival information can be interspersed with pixel primitives that capture light intensity of one or more colors. Therefore, the angle of arrival information can be combined with other image information about the object, and thus the distance information can also be combined with other image information about the object.

[0179] In some embodiments, one or more sensors may be configured to acquire information about physical objects in a scene at high frequency and low latency using compact, low-power components. For example, the power consumption of an image sensor is less than 50 milliwatts, so that a sufficiently small battery as part of a wearable system can be used to power the device. The sensor can be an image sensor configured to passively acquire depth information as a supplement or alternative to image information indicating the intensity of one or more colors and / or intensity change information. Such sensors may also be configured to provide a small amount of data using block tracking or by using DVS techniques to provide a differential output.

[0180] Passive depth information may be obtained by configuring an image array, such as an image array incorporating any one or more of the techniques described herein, wherein components in the image array adapt one or more pixel primitives in the array to output information indicative of a light field emanating from an imaged object. The information may be based on the angle of arrival of light impinging on the pixel. In some embodiments, a pixel primitive, such as described above, may be configured to output an indication of the angle of arrival by placing a plenoptic component in the light path to the pixel primitive. An example of a plenoptic component is a transmissive diffraction mask (TDM). The angle of arrival information may be computationally converted to distance information indicating the distance to an object that reflected light to form a captured image. In some embodiments, pixel primitives configured to provide angle of arrival information are interspersed with pixel primitives that capture light intensities in grayscale or one or more colors. Thus, the angle of arrival information may also be combined with other image information about the object.

[0181] Fig.16 A pixel subarray 100 is shown in accordance with some embodiments. In the illustrated embodiment, the subarray has two pixel primitives, but the number of pixel primitives in the subarray is not limiting of the present invention. Here, a first pixel primitive 121 and a second pixel primitive 122 are shown, one of which is configured to capture angle of arrival information (first pixel primitive 121), but it should be understood that the number and position within the array of pixel primitives configured to measure angle of arrival information is variable. In this example, the other pixel primitive (second pixel primitive 122) is configured to measure the intensity of one color of light, but other configurations are possible, including pixel primitives that are sensitive to different colors of light or one or more pixel primitives that are sensitive to a broad spectrum of light, such as in a grayscale camera.

[0182] Fig.16 The first pixel element 121 of the pixel subarray 100 includes an arrival angle-intensity converter 101, a photodetector 105, and a differential readout circuit 107. The second pixel element 122 of the pixel subarray 100 includes a color filter 102, a photodetector 106, and a differential readout circuit 108. It should be understood that not all pixels are identical. Fig.16 All components shown need to be included in each embodiment. For example, some embodiments may not include differential readout circuits 107 and / or 108, and some embodiments may not include color filter 102. In addition, Fig.16 Other components not shown in the figure. For example, some embodiments may include a polarizer arranged to allow light of a specific polarization to reach the photodetector. As another example, some embodiments may include a scan output circuit as a supplement or alternative to the differential readout circuit 107. As another example, the first pixel primitive 121 may also include a color filter so that the first pixel 121 measures the arrival angle and intensity of light of a specific color incident on the first pixel 121.

[0183] The arrival angle-intensity converter 101 of the first pixel 121 is an optical component that converts the angle θ of the incident light 111 into an intensity that can be measured by a photodetector. In some embodiments, the arrival angle-intensity converter 101 may include a refractive optical device. For example, one or more lenses may be used to convert the light incident angle into a position on the image plane, that is, the amount of incident light detected by one or more pixel elements. In some embodiments, the arrival angle-position intensity converter 101 may include a diffractive optical device. For example, one or more diffraction gratings (e.g., a transmissive diffraction mask (TDM)) may convert the light incident angle into an intensity that can be measured by a photodetector below the TDM.

[0184] The photodetector 105 of the first pixel element 121 receives the incident light 110 transmitted through the arrival angle-intensity converter 101 and generates an electrical signal based on the intensity of the light incident on the photodetector 105. The photodetector 105 is located at an image plane associated with the arrival angle-intensity converter 101. In some embodiments, the photodetector 105 may be a single pixel of an image sensor, such as a CMOS image sensor.

[0185] The differential readout circuit 107 of the first pixel 121 receives the signal from the photodetector 105 and outputs an event only when the amplitude of the electrical signal from the photodetector is different from the previous signal amplitude from the photodetector 105, thereby implementing the above-described DVS technique.

[0186] The second pixel element 122 includes a color filter 102 for filtering the incident light 112 so that only light within a specific wavelength range passes through the color filter 102 and is incident on the photodetector 106. The color filter 102 may be, for example, a bandpass filter that allows one of red, green, or blue light to pass and rejects light of other wavelengths and / or may limit the IR light reaching the photodetector 106 to only a specific portion of the spectrum.

[0187] In this example, the second pixel cell 122 also includes a photodetector 106 and a differential readout circuit 108 , which may operate similarly to the photodetector 105 and the differential readout circuit 107 of the first pixel cell 121 .

[0188] As described above, in some embodiments, an image sensor may include an array of pixels, each pixel being associated with a photodetector and readout circuitry. Subsets of pixels may be associated with angle-of-arrival-to-intensity converters for determining the angle of detected light incident on the pixel. Other subsets of pixels may be associated with color filters for determining color information about the observed scene or for selectively passing or blocking light based on other characteristics.

[0189] In some embodiments, a single photodetector and diffraction gratings located at two different depths may be used to determine the angle of arrival of light. For example, light may be incident on a first TDM, the angle of arrival is converted to position, and a second TDM may be used to selectively pass light incident at a particular angle. This arrangement may take advantage of the Talbot effect, which is a near-field diffraction effect in which an image of the diffraction grating is created at a certain distance from the diffraction grating when a plane wave is incident on the diffraction grating. If a second diffraction grating is placed at an image plane that forms an image of the first diffraction grating, the angle of arrival may be determined based on the light intensity measured by a single photodetector located after the second grating.

[0190] Fig.17A A first arrangement of pixel primitives 140 is shown, which includes a first TDM 141 and a second TDM 143 aligned to each other such that the ridges and / or regions of increased refractive index of the two gratings are aligned in the horizontal direction (Δs=0), where Δs is the horizontal offset between the first TDM 141 and the second TDM 143. Both the first TDM 141 and the second TDM 143 may have the same grating period d, and the two gratings may be separated by a distance / depth z. The depth z at which the second TDM 143 is positioned relative to the first TDM 141, referred to as the Talbot length, may be determined based on the grating period d and the wavelength λ of the light being analyzed, and is given by the following expression:

[0191]

[0192] like Fig.17A As shown, incident light 142 having a zero angle of arrival is diffracted by the first TDM 141. The second TDM 143 is located at a depth equal to the Talbot length, thereby creating an image of the first TDM 141, resulting in most of the incident light 142 being transmitted through the second TDM 143. The optional dielectric layer 145 can separate the second TDM 143 from the photodetector 147. When the light passes through the dielectric layer 145, the photodetector 147 detects the light and generates an electrical signal whose characteristics (e.g., voltage or current) are proportional to the intensity of the light incident on the photodetector. On the other hand, when the incident light 144 having a non-zero angle of arrival θ is also diffracted by the first TDM 141, the second TDM 143 prevents at least a portion of the incident light 144 from reaching the photodetector 147. The amount of incident light that reaches the photodetector 147 depends on the angle of arrival θ, and less light reaches the photodetector at a larger angle. The dashed line generated by the light 144 illustrates that the amount of light reaching the photodetector 147 is attenuated. In some cases, light 144 may be completely blocked by diffraction grating 143. Therefore, information about the angle of arrival of incident light may be obtained using a single photodetector 147 using two TDMs.

[0193] In some embodiments, information acquired by neighboring pixel primitives that do not have an arrival angle-to-intensity converter can provide an indication of the intensity of the incident light and can be used to determine the portion of the incident light that is passed through the arrival angle-to-intensity converter. Based on this image information, the angle of arrival of the light detected by the photodetector 147 can be calculated, as described in more detail below.

[0194] Fig. 17B A second arrangement of pixel primitives 150 is shown, which includes a first TDM 151 and a second TDM 153 that are misaligned with each other, such that the ridges and / or regions of increased refractive index of the two gratings are misaligned in the horizontal direction (Δs≠0), where Δs is the horizontal offset between the first TDM 151 and the second TDM 153. Both the first TDM 151 and the second TDM 153 may have the same grating period d, and the two gratings may be separated by a distance / depth z. In combination with Fig.17A Unlike the case where the two TDMs are aligned as discussed above, misalignment causes incident light having an angle different from zero to pass through the second TDM 153 .

[0195] like Fig. 17B As shown, incident light 152 having a zero degree arrival angle is diffracted by the first TDM 151. The second TDM 153 is located at a depth equal to the Talbot length, but due to the horizontal offset between the two gratings, at least a portion of the light 152 is blocked by the second TDM 153. The dashed line generated by the light 152 illustrates that the amount of light reaching the photodetector 157 is attenuated. In some cases, the light 152 may be completely blocked by the diffraction grating 153. On the other hand, incident light 154 having a non-zero arrival angle θ is diffracted by the first TDM 151, but is passed through the second TDM 153. After traversing the optional dielectric layer 155, the photodetector 157 detects the light incident on the photodetector 157 and generates an electrical signal whose characteristic (e.g., voltage or current) is proportional to the intensity of the light incident on the photodetector.

[0196] Pixel primitives 140 and 150 have different output functions, where the light detected for different angles of incidence has different intensities. However, in each case, the relationship is fixed and can be determined based on the design of the pixel primitive or by measurements as part of a calibration process. Regardless of the exact transfer function, the measured intensity can be converted to an angle of arrival, which in turn can be used to determine the distance to the imaged object.

[0197] In some embodiments, different pixel primitives of an image sensor may have different TDM arrangements. For example, a first subset of pixel primitives may include a first horizontal offset between the gratings of the two TDMs associated with each pixel, and a second subset of pixel primitives may include a second horizontal offset between the gratings of the two TDMs associated with each pixel primitive, wherein the first offset is different from the second offset. Each subset of pixel primitives having a different offset may be used to measure a different angle of arrival or a different range of angles of arrival ... Fig.17A Similar to the TDM arrangement of the pixel primitives 140, the second pixel subset may include Fig. 17B The pixel primitives 150 are arranged similarly to the TDM.

[0198] In some embodiments, not all pixel primitives of the image sensor include TDMs. For example, one subset of pixel primitives may include a color filter, while another subset of pixel primitives may include a TDM for determining angle of arrival information. In other embodiments, color filters are not used, so that the first subset of pixel primitives only measures the total intensity of incident light, while the second subset of pixel primitives measures the angle of arrival information. In some embodiments, information about the light intensity from nearby pixel primitives that do not have TDMs can be used to determine the angle of arrival of light incident on pixel primitives with one or more TDMs. For example, using two TDMs arranged to utilize the Talbot effect, the light intensity incident on the photodetector after the second TDM is a sinusoidal function of the angle of arrival of the light incident on the first TDM. Therefore, if the total intensity of light incident on the first TDM is known, the light arrival angle can be determined based on the light intensity detected by the photodetector.

[0199] In some embodiments, the configuration of pixel primitives in the subarrays may be selected to provide various types of image information with appropriate resolution. Fig.18A -C shows an example arrangement of pixel primitives in a pixel subarray of an image sensor. The example shown is non-limiting, as it should be understood that the inventors contemplate alternative pixel arrangements. This arrangement can be repeated across an image array that can contain millions of pixels. A subarray can include one or more pixel primitives that provide information about the angle of arrival of incident light and one or more other pixel primitives (with or without filters) that provide information about the intensity of the incident light.

[0200] Fig.18A1 is an example of a pixel subarray 160, which includes a first pixel primitive group 161 and a second pixel primitive group 163 that are different from each other and are rectangular rather than square. The pixel primitives marked as "R" are pixel primitives with red filters, so that red incident light passes through the filter to the associated photodetector; the pixel primitives marked as "B" are pixel primitives with blue filters, so that blue incident light passes through the filter to the associated photodetector; the pixel primitives marked as "G" are pixels with green filters, so that green incident light passes through the filter to the associated photodetector. In the example subarray 160, there are more green pixel primitives than red or blue pixel primitives, indicating that the various types of pixel primitives do not need to exist in the same proportion.

[0201] The pixel primitives labeled A1 and A2 are pixels that provide angle of arrival information. For example, pixel primitives A1 and A2 may include one or more gratings for determining angle of arrival information. Pixel primitives that provide angle of arrival information may be configured similarly or differently, such as to be sensitive to different ranges of angles of arrival or to angles of arrival relative to different axes. In some embodiments, the pixels labeled A1 and A2 include two TDMs, and the TDMs of pixel primitives A1 and A2 may be oriented in different directions, such as perpendicular to each other. In other embodiments, the TDMs of pixel primitives A1 and A2 may be oriented parallel to each other.

[0202] In an embodiment using pixel subarray 160, both color image data and angle of arrival information may be acquired. In order to determine the angle of arrival of light incident on pixel primitive group 161, the total light intensity incident on pixel primitive group 161 is estimated using electrical signals from the RGB pixel primitives. Taking advantage of the fact that the light intensity detected by the A1 / A2 pixels varies in a predictable manner as a function of the angle of arrival, the angle of arrival may be determined by comparing the total intensity (estimated from the RGB pixel primitives within the group of pixels) with the intensity measured by the A1 and / or A2 pixel primitives. For example, the light intensity incident on the A1 and / or A2 pixels may vary sinusoidally with respect to the angle of arrival of the incident light. The angle of arrival of light incident on pixel primitive group 163 is determined in a similar manner using electrical signals generated by pixel group 163.

[0203] It should be understood that Fig.18A Specific embodiments of sub-arrays are shown, and other configurations are possible. In some embodiments, for example, a sub-array may be only a pixel primitive group 161 or 163 .

[0204] Fig.18B1 is an alternative pixel subarray 170 including a first pixel primitive group 171, a second pixel primitive group 172, a third pixel primitive group 173, and a fourth pixel primitive group 174. Each pixel primitive group 171-174 is square and has the same pixel primitive arrangement therein, but may have pixel primitives for determining different angular ranges or arrival angle information relative to different planes (for example, the TDMs of pixels A1 and A2 may be oriented perpendicular to each other). Each pixel group 171-174 includes a red pixel primitive (R), a blue pixel primitive (B), a green pixel primitive (G), and an arrival angle pixel primitive (A1 or A2). Note that in the example pixel subarray 170, there are the same number of red / green / blue pixel primitives in each group. In addition, it should be understood that the pixel subarray may be repeated in one or more directions to form a larger pixel array.

[0205] In embodiments using pixel subarray 170, both color image data and angle of arrival information may be acquired. To determine the angle of arrival of light incident on pixel primitive group 171, the total light intensity incident on pixel primitive group 171 may be estimated using signals from the RGB pixel primitives. Using the fact that the light intensity detected by the angle of arrival pixel primitives has a sinusoidal or other predictable response with respect to the angle of arrival, the angle of arrival may be determined by comparing the total intensity (estimated from the RGB pixel primitives) to the intensity measured by the A1 pixel. The angle of arrival of light incident on pixel primitive groups 172-174 may be determined in a similar manner using electrical signals generated by the pixel primitives of each respective pixel group.

[0206] Fig.18C 1 is an alternative pixel subarray 180 including a first pixel primitive group 181, a second pixel primitive group 182, a third pixel primitive group 183, and a fourth pixel primitive group 184. Each pixel primitive group 181-184 is square and has identically arranged pixel primitives therein, without any color filters. Each pixel primitive group 181-184 includes two "white" pixels (e.g., without any color filters so that red, blue, and green light are detected to form a grayscale image), an arrival angle pixel primitive (A1) having a TDM oriented in a first direction, and an arrival angle pixel primitive (A2) having a TDM oriented at a second pitch or in a second direction relative to (e.g., perpendicular to) the first direction. Note that there is no color information in the pixel subarray 170 in the example. The resulting image is a grayscale image, illustrating that passive depth information can be obtained in a color or grayscale image array using the techniques described herein. As with other subarray configurations described herein, the pixel subarray arrangement can be repeated in one or more directions to form a larger pixel array.

[0207] In embodiments using pixel subarray 180, both grayscale image data and angle of arrival information may be acquired. To determine the angle of arrival of light incident on pixel primitive group 181, the total light intensity incident on group 181 is estimated using electrical signals from two white pixels. Using the fact that the light intensity detected by the A1 and A2 pixels has a sinusoidal or other predictable response with respect to the angle of arrival, the angle of arrival may be determined by comparing the total intensity (estimated from the white pixels) to the intensity measured by the A1 and / or A2 pixel primitives. The angle of arrival of light incident on pixel primitive groups 182-184 may be determined in a similar manner using electrical signals generated by the pixels of each respective pixel group.

[0208] In the above examples, the pixel primitives are shown as squares and arranged in a square grid. The embodiments are not limited thereto. For example, in some embodiments, the shape of the pixel primitives may be rectangular. In addition, the subarrays may be triangular or arranged on a diagonal or have other geometric shapes.

[0209] In some embodiments, the angle of arrival information is obtained using the image processor 708 or a processor associated with the local data processing module 70, and the local data processing module 70 can further determine the distance of the object based on the angle of arrival. For example, the angle of arrival information can be combined with one or more other types of information to obtain the distance of the object. In some embodiments, the objects of the grid model 46 can be associated with the angle of arrival information from the pixel array. The grid model 46 can include the location of the object, including the distance from the user, which can be updated to a new distance value based on the angle of arrival information.

[0210] Using angle of arrival information to determine distance values ​​is particularly useful in scenarios where objects are close to the user. This is because changes in distance from the image sensor cause changes in the angle of arrival of light for nearby objects to be greater than similar magnitude distance changes for objects positioned farther away from the user. Therefore, a processing module that utilizes passive distance information based on the angle of arrival may selectively use this information based on estimating the distance of the object, and may utilize one or more other techniques to determine the distance to an object that exceeds a threshold distance, which in some embodiments may be, for example, up to 1 meter, up to 3 meters, or up to 5 meters. As a specific example, a processing module of an AR system may be programmed to use passive distance measurement using angle of arrival information for objects within 3 meters of a wearable device user, but for objects outside this range, stereoscopic image processing may be used, which uses images captured by two cameras.

[0211] Similarly, pixels configured to detect angle of arrival information are most sensitive to distance changes within an angular range relative to the image array normal. The processing module can similarly be configured to use distance information derived from angle of arrival measurements within this angular range, but use other sensors and / or other techniques to determine distances outside of this range.

[0212] An example application of determining the distance of an object from an image sensor is hand tracking. Hand tracking may be used, for example, in an AR system to provide a gesture-based user interface for system 80 and / or to allow a user to move virtual objects within an environment in an AR experience provided by system 80. The combination of an image sensor that provides angle of arrival information for accurate depth determination and a differential readout circuit that reduces the amount of processed data to determine the user's hand motion provides an effective interface through which a user can interact with virtual objects and / or provide input to system 80. A processing module that determines the position of a user's hand may use distance information that is acquired using different techniques, depending on the position of the user's hand in the field of view of the image sensor of the wearable device. According to some embodiments, hand tracking may be implemented as a form of block tracking during the image sensing process.

[0213] Another application that utilizes depth information is occlusion processing. Occlusion processing uses depth information to determine that certain parts of the physical world model do not need to or cannot be updated based on image information captured by one or more image sensors that collect image information related to the physical environment around the user. For example, if it is determined that there is a first object at a first distance from the sensor, the system 80 may determine not to update the physical world model for distances greater than the first distance. For example, even if the model includes a second object at a second distance from the sensor, the second distance is greater than the first distance, but if the second object is behind the first object, the model information of the object may not be updated. In some embodiments, the system 80 may generate an occlusion mask based on the position of the first object, and only update the parts of the model that are not masked by the occlusion mask. In some embodiments, the system 80 may generate more than one occlusion mask for more than one object. Each occlusion mask may be associated with a corresponding distance from the sensor. For each occlusion mask, the model information associated with the following objects will not be updated: the distance of the object from the sensor is greater than the distance associated with the corresponding occlusion mask. By limiting the parts of the model that are updated at any given time, the speed of generating the AR environment and the amount of computing resources required to generate the AR environment are reduced.

[0214] Although not in Fig.18A -C, but some embodiments of the image sensor may include pixels having an IR filter as a supplement or alternative to the color filter. For example, the IR color filter may allow light of a wavelength such as approximately equal to 940nm to pass through and be detected by an associated photodetector. Some embodiments of the wearable device may include an IR light source (e.g., an IRLED) that emits light having the same wavelength (e.g., 940nm) associated with the IR color filter. The IR light source and IR pixels may be used as an alternative way to determine the distance of an object from the sensor. By way of example and not limitation, the IR light source may be a pulsed light source and a time-of-flight measurement may be used to determine the distance of an object from the sensor.

[0215] In some embodiments, the system 80 may operate in one or more operating modes. A first mode may be a mode in which a depth determination is made using passive depth measurement, for example based on the angle of arrival of light determined using pixels with an angle of arrival-intensity converter. A second mode may be a mode in which a depth determination is made using active depth measurement, for example based on the time of flight of IR light measured using IR pixels of an image sensor. A third mode may use stereo measurements from two independent image sensors to determine the distance of an object. When the object is far from the sensor, such stereo measurements may be more accurate than the angle of arrival of light determined using pixels with an angle of arrival-intensity converter. Other suitable methods of determining depth may be used for one or more additional depth determination operating modes.

[0216] In some embodiments, passive depth determination may be preferred because this technique consumes less power. However, the system may determine that it should operate in active mode under certain conditions. For example, if the visible light intensity detected by the sensor is below a threshold, it may be too dark to accurately perform passive depth determination. As another example, the object may be too far away for passive depth determination to be accurate enough. Therefore, the system may be programmed to choose to operate in a third mode, in which depth is determined based on scene stereo measurement using two spatially separated image sensors. As another example, determining the depth of an object may not be accurate enough at the periphery of the image sensor based on the angle of arrival of light determined using pixels with an angle of arrival-intensity converter. Therefore, if the object is detected by pixels near the periphery of the image sensor, the system may choose to operate in the second mode, using active depth determination.

[0217] While the above-described embodiments of the image sensor use a single pixel primitive with a stacked TDM to determine the angle of arrival of light incident on the pixel primitive, other embodiments may use a plurality of groups of pixel primitives with a single TDM on all pixels of the group to determine the angle of arrival information. The TDM may project a light pattern across the entire sensor array that depends on the angle of arrival of the incident light. Multiple photodetectors associated with one TDM may more accurately detect the pattern because each of these multiple photodetectors is located at a different position in the image plane (the image plane includes the photodetectors that sense the light). The relative intensity sensed by each photodetector may indicate the angle of arrival of the incident light.

[0218] Fig.19A is a top view of an example of multiple photodetectors (in the form of a photodetector array 120, which may be a subarray of image sensor pixel elements) associated with a single transmissive diffraction mask (TDM) in accordance with some embodiments. Fig.19B is with Fig.19A The photodetector array is the same as the photodetector array along Fig.19A 1. A cross-sectional view of line A of FIG. 1. In the example shown, the photodetector array 120 includes 16 individual photodetectors 121, which may be located within pixel primitives of the image sensor. The photodetector array 120 includes a TDM 123 disposed above the photodetectors. It should be understood that for clarity and simplicity (e.g., forming a four-pixel by four-pixel grid), each pixel primitive group is shown as having four pixels. Some embodiments may include more than four pixel primitives. For example, each group may include 16 pixel primitives, 64 pixel primitives, or any other number of pixels.

[0219] TDM 123 is located at a distance x from photodetector 121. In some embodiments, TDM 123 is formed on the top surface of dielectric layer 125, such as Fig.19B As shown. For example, as shown, TDM 123 is formed by ridges, or by valleys etched into the surface of dielectric layer 125. In other embodiments, TDM 123 may be formed within the dielectric layer. For example, a portion of the dielectric layer may be modified to have a higher or lower refractive index relative to other portions of the dielectric layer, thereby producing a holographic phase grating. Light incident on the photodetector array 120 from above is diffracted by the TDM, resulting in the incident light arrival angle being converted to a position in the image plane that is a distance x from TDM 123, where photodetector 121 is located. The incident light intensity measured at each photodetector 121 in the photodetector array can be used to determine the incident light arrival angle.

[0220] Fig. 20A An example of multiple photodetectors (in the form of photodetector array 130) associated with multiple TDMs is shown in accordance with some embodiments. Fig. 20B is through Fig. 20A The line B intercepts Fig. 20A Cross-sectional view of the same photodetector array. Fig. 20C is through Fig. 20A The line C intercepts Fig. 20A 1. A cross-sectional view of the same photodetector array. In the example shown, the photodetector array 130 includes 16 individual photodetectors that may be located within an image sensor pixel primitive. Four groups 131a, 131b, 131c, 131d of four pixel primitives are shown. The photodetector array 130 includes four individual TDMs 133a, 133b, 133c, 133d, each TDM being disposed above an associated group of pixel primitives. It should be understood that for clarity and simplicity, each pixel primitive group is shown as having four pixel primitives. Some embodiments may include more than four pixel primitives. For example, each group may include 16 pixel primitives, 64 pixel primitives, or any other number of pixels.

[0221] Each TDM 133a-d is located at a distance x from the photodetectors 131a-d. In some embodiments, the TDMs 133a-d are formed on the top surface of the dielectric layer 135, such as Fig. 20B 133a-d. For example, the TDMs 123a-d may be formed by ridges, as shown, or by valleys etched into the surface of the dielectric layer 135. In other embodiments, the TDMs 133a-d may be formed within the dielectric layer. For example, a portion of the dielectric layer may be modified to have a higher or lower refractive index relative to other portions of the dielectric layer to produce a holographic phase grating. Light incident on the photodetector array 130 from above is diffracted by the TDMs, resulting in the incident light arrival angle being converted to a position in the image plane that is a distance x from the TDMs 133a-4, where the photodetectors 131a-d are located. The incident light intensity measured at each photodetector 131a-d in the photodetector array may be used to determine the incident light arrival angle.

[0222] TDMs 133a-d may be oriented in different directions from each other. For example, TDM 133a is perpendicular to TDM 133b. Therefore, the light intensity detected using photodetector group 131a can be used to determine the angle of arrival of incident light in a plane perpendicular to TDM 133a, and the light intensity detected using photodetector group 131b can be used to determine the angle of arrival of incident light in a plane perpendicular to TDM 133b. Similarly, the light intensity detected using photodetector group 131c can be used to determine the angle of arrival of incident light in a plane perpendicular to TDM 133c, and the light intensity detected using photodetector group 131d can be used to determine the angle of arrival of incident light in a plane perpendicular to TDM 133d.

[0223] Pixel primitives configured to passively acquire depth information may be integrated into an image array having the features described herein to support useful operations in an X-reality system. According to some embodiments, pixel primitives configured to acquire depth information may be implemented as part of an image sensor for implementing a camera with a global shutter. For example, this configuration may provide an entire frame output. The entire frame may include image information of different pixels indicating both depth and intensity. Using an image sensor of this configuration, a processor may acquire depth information for an entire scene at once.

[0224] In other embodiments, image sensor pixel primitives providing depth information may be configured to operate according to the above-described DVS techniques. In such scenarios, an event may indicate a change in the depth of an object, as indicated by a pixel primitive. An event output by an image array may indicate a pixel primitive for which a depth change was detected. Alternatively or additionally, the event may include the value of the depth information for the pixel primitive. With this configured image sensor, a processor may obtain depth information updates at a very high rate, thereby providing high temporal resolution.

[0225] In further embodiments, the image sensor may be configured to operate in either a full frame or DVS mode. In such embodiments, a processor processing image information from the image sensor may programmatically control the operating mode of the image sensor based on the function being performed by the processor. For example, when performing a function involving tracking an object, the processor may configure the image sensor to output image information as DVS events. On the other hand, when performing a process of updating world reconstruction, the processor may configure the image sensor to output full frame depth information.

[0226] Wearable Configuration

[0227] Multiple image sensors may be used in an XR system. Image sensors may be combined with optical components, such as lenses, and control circuitry to create a camera. These image sensors may use one or more of the above-mentioned techniques to acquire imaging information, such as grayscale imaging, color imaging, global shutter, DVS techniques, plenoptic pixel primitives, and / or dynamic blocking. Regardless of the imaging technique used, the resulting camera may be mounted to a support member to form a head-mounted device, which may include or be connected to a processor.

[0228] Fig.21 2 is a schematic diagram of a head mounted device 2100 showing a wearable display system consistent with the disclosed embodiments. Fig.21 As shown, the head mounted device 2100 may include a display device including a monocular 2110a and a monocular 2110b, which may be an optical eyepiece or display configured to transmit and / or display visual information to the user's eyes. The head mounted device 2100 may also include a frame 2101, which may be similar to the above description of the head mounted device 2100. Figure 3B The frame 64. The head mounted device 2100 may also include three cameras (camera 2120a, camera 2120b, and camera 2140) and additional components such as a transmitter 2130a, a transmitter 2130b, an inertial measurement unit 2170a (IMU 2170a), and an inertial measurement unit 2170b (IMU 2170b).

[0229] Camera 2120a, camera 2120b, and camera 2140 are world cameras because they are oriented to image the physical world as seen by the user wearing the head mounted device 2100. In some embodiments, these three cameras are sufficient to acquire image information about the physical world, and these three cameras may be the only world-facing cameras. The head mounted device 2100 may also include additional components, such as those described above with respect to the Figure 3B The eye tracking camera in question.

[0230] Monocular 2110a and monocular 2110b may be mechanically coupled to a support member such as frame 2101 using techniques such as adhesives, fasteners, or pressure fittings. Similarly, three cameras and accessory components (e.g., transmitter, inertial measurement unit, eye tracking camera, etc.) may be mechanically coupled to frame 2101 using techniques such as adhesives, fasteners, pressure fittings, etc. These mechanical couplings may be direct or indirect. For example, one or more cameras and / or one or more accompanying components may be directly attached to frame 2101. As an additional example, one or more cameras and / or one or more accessory components may be directly attached to a monocular, which is then attached to frame 2101. The attachment mechanisms are not intended to be limiting.

[0231] Alternatively, the monocular mirror assembly can be formed and then attached to the frame 2101. Each subassembly can include, for example, a support member to which the monocular 2110a or 2110b is attached. The IMU and one or more cameras can be similarly attached to the support member. Attaching both the camera and the IMU to the same support member enables inertial information about the camera to be obtained based on the output of the IMU. Similarly, attaching the monocular to the same support member as the camera allows image information about the world to be spatially associated with information rendered on the monocular.

[0232] The head mounted device 2100 may be lightweight. For example, the weight of the head mounted device 2100 may be between 30 and 300 grams. The head mounted device 2100 may be made of a material that flexes during use, such as plastic or thin metal parts. Such materials may enable a lightweight and comfortable head mounted device that a user may wear for a long time. Nevertheless, an XR system with such a lightweight head mounted device may support the use of a calibration routine to perform high-precision stereo image analysis, which requires knowing the spacing between the cameras, and such a calibration routine may be repeatedly performed while the head mounted device is worn to compensate for any inaccuracies caused by the bending of the head mounted device during use. In some embodiments, the lightweight head mounted device may include a battery pack. The battery pack may include one or more batteries, which may be rechargeable or non-rechargeable. The battery pack may be built into the lightweight frame or may be removable. The battery pack and the lightweight frame may be formed as a single unit, or the battery pack may be formed as a unit separate from the lightweight frame.

[0233] Camera 2120a and camera 2120b may include an image sensor and a lens, respectively. The image sensor may be configured to generate a grayscale image. The image sensor may be configured to acquire an image having a size between 1 megapixel and 4 megapixels. For example, the image sensor may be configured to acquire an image having a horizontal resolution of 1016 lines and a vertical resolution of 1016 lines. The image sensor may be configured to acquire images repeatedly or periodically. For example, the image sensor may be configured to acquire images at a frequency between 30 Hz and 120 Hz (such as 60 Hz). The image sensor may be a CMOS image sensor. The image sensor may be configured with a global shutter. As described above with respect to Fig.14 and Fig.15 As described above, a global shutter enables each pixel to acquire intensity measurements simultaneously.

[0234] Consistent with the disclosed embodiments, camera 2120a and camera 2120b may each be configured to have a wide field of view. For example, camera 2120a and camera 2120b may include equidistant lenses (e.g., fisheye lenses). Camera 2120a and camera 2120b may each be angled inward on head mounted device 2100. For example, through field of view 2121a, a vertical plane at the center of the field of view associated with camera 2120a may intersect and form an angle with a vertical plane passing through the midline of head mounted device 2100. The angle may be between 1 and 40 degrees. In some embodiments, field of view 2121a may have a horizontal field of view and a vertical field of view. The horizontal field of view may range between 90 degrees and 175 degrees, while the vertical field of view may range between 70 degrees and 125 degrees. Similarly, through field of view 2121b, the field of view associated with camera 2120b, a vertical plane at the center may intersect and form an angle with a vertical plane through the centerline of head mounted device 2100. This angle may also be between 1 degree and 40 degrees. In some embodiments, camera 2120a and camera 2120b may have the same angle inward. Consistent with the disclosed embodiments, field of view 2121b may have a horizontal field of view and a vertical field of view. The horizontal field of view may range between 90 degrees and 175 degrees, while the vertical field of view may range between 70 degrees and 125 degrees.

[0235] Camera 2120a and camera 2120b may be configured to provide overlapping views of a central field of view 2150. The angular range of the central field of view 2150 may be between 20 degrees and 80 degrees. For example, the angular range of the central field of view 2150 may be approximately 40 degrees (e.g., 40 ± 4 degrees). In addition to the central field of view 2150, camera 2120a and camera 2120b may be positioned to provide at least two peripheral fields of view. Peripheral field of view 2160a may be associated with camera 2120a and may include a portion of field of view 2121a that does not overlap with field of view 2121b. In some embodiments, the angular range of peripheral field of view 2160b may be between 40 degrees and 80 degrees. For example, the angular range of peripheral field of view 2160a may be approximately 60 degrees (e.g., 60 ± 6 degrees). Peripheral field of view 2160b may be associated with camera 2120b and may include a portion of field of view 2121b that does not overlap with field of view 2121a. In some embodiments, the angular range of the peripheral field of view 2160b may be between 40 degrees and 80 degrees. For example, the angular range of the peripheral field of view 2160b may be approximately 60 degrees (eg, 60±6 degrees).

[0236] Emitters 2130a and 2130b may be used for imaging in low light conditions and / or when active depth sensing is performed by the head mounted device 2100. Emitters 2130a and 2130b may be configured to emit light of a particular wavelength. This light may be reflected by physical objects in the physical world around the user. The head mounted device 2100 may be configured with sensors to detect this reflected light, including image sensors as described herein. In some embodiments, these sensors may be incorporated into at least one of the cameras 2120a, 2120b, or 2140. For example, as described above with respect to Figures 18A-18C As described above, these cameras may be configured with detectors corresponding to emitter 2130a and / or emitter 2130b. For example, these cameras may include pixels configured to detect light emitted by emitter 2130a and / or emitter 2130b.

[0237] Consistent with the disclosed embodiments, emitter 2130a and emitter 2130b may be configured to emit IR light. The wavelength of IR light is between 900 nanometers and 1 micron. The IR light may be, for example, a 940nm light source, the emitted light energy being concentrated near 940nm. Emitters emitting light of other wavelengths may be used alternatively or additionally. For example, for a system intended for indoor use only, an emitter emitting light concentrated near 850nm may be used. At least one of camera 2120a, camera 2120b, or camera 2140a may include one or more IR color filters disposed over at least one subset of pixels in the image sensor of the camera. The filter may pass light of a certain wavelength emitted by emitter 2130a and / or emitter 2130b while attenuating light of other wavelengths. For example, the IR filter may be a notch filter that passes IR light having a wavelength matching the wavelength of the emitter. The notch filter may significantly attenuate other IR light. In some embodiments, the notch filter may be an IR notch filter that blocks IR light and allows light from the emitter to pass. The IR notch filter may also allow light outside the IR band to pass. Such a notch filter may enable the image sensor to receive both visible light and light from the emitter that has been reflected from objects in the image sensor's field of view. In this way, a subset of pixels may serve as a detector of IR light emitted by emitter 2130a and / or emitter 2130b.

[0238] In some embodiments, the processor of the XR system may selectively enable the emitter, such as to enable imaging in low light conditions. The processor may process image information generated by one or more image sensors, and may detect whether the images output by these image sensors provide sufficient information about objects in the physical world without enabling the emitter. The processor may enable the emitter in response to detecting that the image does not provide sufficient image information due to low ambient light conditions. For example, the emitter may be turned on when an object is tracked using stereo information, and the lack of ambient light causes insufficient contrast between features of the tracked object in the image to accurately determine the distance using stereo image technology.

[0239] Alternatively or additionally, emitter 2130a and / or emitter 2130b may be configured to perform active depth measurements, such as by emitting short pulses of light. The wearable display system may be configured to perform time-of-flight measurements by detecting reflections of such pulses by objects in the illumination field 2131a of emitter 2130a and / or the illumination field 2131b of emitter 2130b. These time-of-flight measurements may provide additional depth information for tracking objects or updating a connected world model. In other embodiments, one or more emitters may be configured to emit patterned light, and the XR system may be configured to process images of objects illuminated by the patterned light. Such processing may detect changes in the pattern, thereby revealing the distance to the object.

[0240] In some embodiments, the illumination field range associated with the emitters is sufficient to illuminate at least the field of view of the camera to obtain image information about the object. For example, the emitters may collectively illuminate the central field of view 2150. In the illustrated embodiment, emitters 2130a and emitters 2130b may be positioned to illuminate fields 2131a and illumination fields 2131b that collectively span a range where active illumination may be provided. In this example embodiment, two emitters are shown, but it should be understood that more or fewer emitters may be used to span the desired range.

[0241] In some embodiments, transmitters, such as transmitters 2130a and 2130b, are in an off state by default, but can be turned on when additional illumination is needed to obtain more information than can be obtained through passive imaging. The wearable display system can be configured to enable transmitter 2130a and / or transmitter 2130b when additional depth information is needed. For example, when the wearable display system detects that it cannot obtain sufficient depth information using stereoscopic image information to track hand or head gestures, the wearable display system can be configured to enable transmitter 2130a and / or transmitter 2130b. When additional depth information is not needed, transmitter 2130a and / or transmitter 2130b can be disabled, thereby reducing power consumption and extending battery life.

[0242] In addition, it is not required that the IR transmitter be mounted on the head-mounted device 2100 or only on the head-mounted device 2100, even if the head-mounted device is equipped with an image sensor configured to detect IR light. In some embodiments, the IR transmitter can be an external device installed in a space, such as a room, where the head-mounted device 2100 can be used. Such a transmitter can project IR light that is invisible to the human eye, such as an ArUco pattern at 940nm. Light with such a pattern can facilitate "instrumentation / assisted tracking", in which the head-mounted device 2100 does not have to supply power to provide the IR pattern, but due to the presence of the pattern, IR image information can be provided anyway, so that the distance or position of an object in the space can be determined by processing the image information. A system with an external illumination source can also allow more devices to perform operations in this space. If multiple head-mounted devices perform operations in the same space, each head-mounted device moves in the space without a fixed positional relationship, and there is a risk that light emitted by one head-mounted device will be projected on the image sensor of another head-mounted device, thereby interfering with its operation. For example, the risk of such interference between head mounted devices limits the number of head mounted devices that can operate in a space to 3 or 4. With one or more IR emitters in the space illuminating objects that can be imaged by image sensors on the head mounted devices, more head mounted devices (more than 10 in some embodiments) can operate in the same space without interference.

[0243] As mentioned above about Figure 3B As disclosed, the camera 2140 may be configured to capture an image of the physical world within the field of view 2141. The camera 2140 may include an image sensor and a lens. The image sensor may be configured to produce a color image. The image sensor may be configured to acquire an image having a size between 6 megapixels and 24 megapixels. For example, the image sensor may acquire an image having a resolution of 4Kx2K (e.g., 3840 or 4096 lines of horizontal resolution and 1716 or 2160 lines of vertical resolution). The image sensor may be configured to acquire images repeatedly or periodically. For example, the image sensor may be configured to acquire images at a frequency between 30 Hz and 120 Hz, such as 60 Hz.

[0244] The image sensor may be a CMOS image sensor. The image sensor may be configured with a rolling shutter. Fig.14 and Fig.15 As discussed, a rolling shutter may iteratively read subsets of pixels in an image sensor such that pixels in different subsets reflect light intensity data collected at different times. For example, an image sensor may be configured to read a first row of pixels in the image sensor at a first time, and to read a second row of pixels in the image sensor at a later time. In some embodiments, the sensor may be a CMOS sensor.

[0245] The field of view 2141 of the camera 2140 may include a horizontal field of view and a vertical field of view. The horizontal field of view ranges from 75 to 125 degrees, while the vertical field of view ranges from 60 to 125 degrees.

[0246] IMU 2170a and / or IMU 2170b may be configured to provide acceleration and / or velocity and / or inclination information to the wearable display system. For example, when a user wearing head mounted device 2100 moves, IMU 2170a and / or IMU 2170b may provide information describing the acceleration and / or velocity of the user's head.

[0247] The wearable display system may be coupled to a processor that may be configured to process image information acquired with a camera to extract information from the image captured with the camera as described herein and / or render a virtual object on a display device. The processor may be mechanically coupled to the frame 2101. Alternatively, the processor may be mechanically coupled to a display device, such as a display device including a monocular 2110a or a monocular 2110b. As a further alternative, the processor may be operably coupled to the head-mounted device 2100 and / or the display device via a communication link. For example, the XR system may include a local data processing module. The local data processing module may include a processor and may be connected to the head-mounted device 2100 or the display device via a physical connection (e.g., a wire or cable) or a wireless connection (e.g., Bluetooth, Wi-Fi, Zigbee, etc.).

[0248] The processor may be configured to perform world reconstruction, head pose tracking, and object tracking operations. For example, the processor may be configured to create a connected world model using camera 2120a, camera 2120b, and camera 2140. When creating the connected world model, the processor may be configured to use multiple images of the same physical object acquired by camera 2120a and camera 2120b to stereoscopically determine depth information. As an additional example, the processor may be configured to update an existing connected world model using camera 2120a and camera 2120b instead of camera 2140. As described above, camera 2120a and camera 2120b are grayscale cameras with a relatively lower resolution than color camera 2140. Therefore, the connected world model can be quickly updated using images acquired by camera 2120a and camera 2120b instead of camera 2140, which can reduce power consumption and extend battery life. In some embodiments, the processor may be configured to occasionally or periodically update the connected world model using camera 2120a, camera 2120b, and camera 2140. For example, the processor may be configured to determine that the connected world quality standard is no longer met, and / or a predetermined time interval has passed since the last acquisition and / or use of an image acquired by camera 2140, and / or an object within a portion of the physical world currently within the field of view of cameras 2120a, camera 2120b, and camera 2140 has changed.

[0249] According to some embodiments, the XR system may include a hardware accelerator. The hardware accelerator may be implemented as an application specific integrated circuit (ASIC) or other semiconductor device and may be integrated into the head mounted device 2100 or otherwise coupled to the head mounted device so that it receives image information from the camera 2120a and the camera 2120b. The hardware accelerator may use images acquired by the two world cameras 2120a and 2120b to assist in stereoscopically determining depth information. These images may be grayscale images. Using hardware acceleration may speed up the determination of depth information and reduce power consumption, thereby extending battery life.

[0250] The processor may be configured to perform object tracking in the central field of view 2150 using images from the camera 2120a and the camera 2120b. In some embodiments, the processor may perform object tracking using depth information determined stereoscopically from the first images acquired by the two cameras. As a non-limiting example, the tracked object may be a hand of a user of the wearable display system. The processor may be configured to perform object tracking in the peripheral field of view using images acquired from one of the cameras 2120a and the camera 2120b.

[0251] Example Calibration Process

[0252] Fig. 22A simplified flow chart of a calibration routine (method 2200) according to some embodiments is shown. The processor may be configured to perform the calibration routine while wearing the wearable display system. The calibration routine may account for deformation caused by the lightweight structure of the head mounted device 2100. In some embodiments, the calibration routine may account for deformation of the frame 2101 caused by temperature changes or mechanical stress of the frame 2101 during use. For example, the processor may repeatedly perform the calibration routine so that the calibration routine compensates for deformation of the frame 2101 during use of the wearable display system. The compensation routine may be performed automatically or in response to manual input (e.g., a user request to perform the calibration routine). The calibration routine may include determining the relative position and orientation of the camera 2120a and the camera 2120b. The processor may be configured to perform the calibration routine using images acquired by the camera 2120a and the camera 2120b. In some embodiments, the processor may be configured to further use the output of the IMU 2170a and the IMU 2170b.

[0253] After starting in box 2201, method 2200 continues to box 2210. In box 2210, the processor may identify corresponding features in the images acquired from camera 2120a and camera 2120b. The corresponding features may be part of an object in the physical world. In some embodiments, for the purpose of calibration, the object may be placed by the user in the central field of view 2150 and have easily identifiable features in the image, which may have a predetermined relative position. However, the calibration techniques described herein may be performed based on features on the object that appear in the central field of view 2150 during calibration, allowing calibration to be repeated during use of the head-mounted device 2100. In various embodiments, the processor may be configured to automatically select features detected in both the field of view 2121a and the field of view 2121b. In some embodiments, the processor may be configured to determine the correspondence between the features using the estimated positions of the features in the field of view 2121a and the field of view 2121b. This estimation may be based on a connected world model constructed for the object containing these features or other information about the features.

[0254] Method 2200 continues to box 2230. In box 2230, the processor may receive inertial measurement data. The inertial measurement data may be received from IMU 2170a and / or IMU 2170b. The inertial measurement data may include inclination and / or acceleration and / or velocity measurements. In some embodiments, IMU 2170a and 2170b may be mechanically coupled directly or indirectly to camera 2120a and camera 2120b, respectively. In such embodiments, the difference in inertial measurements, such as inclination, made by IMU 2170a and 2170b may indicate a difference in the position and / or orientation of camera 2120a and camera 2120b. Therefore, the output of IMU 2170a and 2170b may provide a basis for an initial estimate of the relative position of camera 2120a and camera 2120b.

[0255] After box 2230, method 2200 continues to box 2250. In box 2250, the processor may calculate an initial estimated relative position and orientation of camera 2120a and camera 2120b. This initial estimate may be calculated using measurements received from IMU 2170a and / or IMU 2170b. For example, in some embodiments, the head mounted device may be designed to have a nominal relative position and orientation of camera 2120a and camera 2120b. The processor may be configured to attribute the difference between the measurements received from IMU 2170a and IMU 2170b to deformation of frame 2101, which may change the position and / or orientation of camera 2120a and camera 2120b. For example, IMU 2170a and IMU 2170b may be directly or indirectly mechanically coupled to frame 2101 so that the inclination and / or acceleration and / or velocity measurements made by these sensors have a predetermined relationship. This relationship may be affected when the frame 2101 is deformed. As a non-limiting example, the IMU 2170a and the IMU 2170b may be mechanically coupled to the frame 2101 so that in the absence of deformation of the frame 2101, these sensors measure similar inclination, acceleration, or velocity vectors during movement of the head mounted device. In this non-limiting example, a twist or bend that rotates the IMU 2170a relative to the IMU 2170b may cause the inclination, acceleration, or velocity vector measurements of the IMU 2170a to rotate accordingly relative to the corresponding vector measurements of the IMU 2170b. Since the IMU 2170a and the IMU 2170b are mechanically coupled to the camera 2120a and the camera 2170b, respectively, the processor may adjust the nominal relative position and orientation of the camera 2120a and the camera 2120b according to the measured relationship between the IMU 2170a and the IMU 2170b.

[0256] Other techniques may alternatively or additionally be used to make the initial estimate.In embodiments where calibration method 2200 is performed repeatedly during operation of the XR system, the initial estimate may be, for example, the most recently calculated estimate.

[0257] After block 2250, a subprocess is initiated in which the relative position and orientation of camera 2120a and camera 2120b are further estimated. One of the estimates is selected as the relative position and orientation of camera 2120a and camera 2120b to calculate stereo depth information based on the images acquired by camera 2120a and camera 2120b. This subprocess may be performed iteratively, making further estimates in each iteration until an acceptable estimate is identified. Fig. 22 In the example of , the subprocess includes boxes 2270, 2272, 2274 and 2290.

[0258] In box 2270, the processor may calculate an error for the estimated relative orientation of the camera and the compared feature. In calculating this error, the processor may be configured to estimate how the identified feature should appear or where the identified feature should be within the corresponding image acquired by camera 2120a and camera 2120b based on the estimated relative orientation of camera 2120a and camera 2120b and the estimated position feature used for calibration. In some embodiments, this estimate can be compared with the appearance position or apparent position of the corresponding feature in the image acquired by each of the two cameras to generate an error for each estimated relative orientation. This error can be calculated using linear algebra techniques. For example, the mean square deviation between the calculated position and the actual position of each of the multiple features within the image can be used as a measure of the error.

[0259] After block 2270, method 2200 continues to block 2272, where it can be checked whether the error meets the acceptance criteria. For example, the criterion can be the total size of the error, or it can be the error change between iterations. If the error meets the acceptance criteria, method 2200 continues to block 2290.

[0260] In block 2290, the processor may select one of the estimated relative orientations based on the error calculated in block 2272. The selected estimated relative position and orientation may be the estimated relative position and orientation with the lowest error. In some embodiments, the processor may be configured to select the estimated relative position and orientation associated with the lowest error as the current relative position and orientation of camera 2120a and camera 2120b. After block 2290, method 2200 continues to block 2299. Method 2200 may end in block 2299, where the selected positions and orientations of camera 2120a and camera 2120b are used to calculate stereoscopic image information based on images formed by these cameras.

[0261] If the error does not meet the acceptance criteria at box 2272, the method 2200 continues to box 2274. In box 2274, the estimates used to calculate the error in box 2270 can be updated. These updates can be the estimated relative positions and / or orientations of camera 2120a and camera 2120b. In an embodiment where the relative positions of a set of features used for calibration are estimated, the updated estimates selected in box 2274 may alternatively or additionally include updates to the positions of the features in the set. This update can be performed based on linear algebra techniques for solving a system of multivariable equations. As a specific example, one or more of the estimated positions or orientations can be increased or decreased. If in one iteration of the subprocess, the change reduces the calculated error, then in subsequent iterations, the same estimated position or orientation is further changed in the same direction. Conversely, if the change increases the error, then in subsequent iterations, those estimated positions or orientations are changed in the opposite direction. The estimated positions and orientations of the cameras and features used in the calibration process can be changed sequentially or in combination in this manner.

[0262] Once the updated estimate is calculated, the subprocess returns to block 2270. Here, further iterations of the subprocess begin by calculating the error of the estimated relative position. In this manner, the estimated position and orientation are updated until an updated relative position and orientation that provides an acceptable error is selected. However, it should be understood that the processing in block 2272 may apply other criteria for terminating the iterative subprocess, such as completing multiple iterations without finding an acceptable error.

[0263] Although method 2200 is described in conjunction with camera 2120a and camera 2120b, similar calibration can be performed for any camera pair used for stereoscopic imaging or for any plurality of camera groups that need to have relative positions and orientations. For example, the position and orientation of camera 2140 relative to either or both of camera 2120a and camera 2120b can be calculated.

[0264] Example Camera Configuration

[0265] According to some embodiments, components are incorporated into the head mounted device 2100 to provide a field of view and a field of illumination to support the various functions of the XR system. FIG. 23A to FIG. 23C According to some embodiments Fig.21 21. Example diagrams of fields of view or illumination associated with a head mounted device 2100. Each example diagram shows the field of view or illumination from a different orientation and distance relative to the head mounted device. Fig.23A The field of view or illumination is shown from elevated off-axis angles at a distance of 1 meter from the head mounted device. Fig.23AThe overlap between the fields of view of cameras 2120a, 2120b, and 2140 is shown, and in particular how camera 2120a and camera 2120b are angled so that field of view 2121a and field of view 2121b pass through the centerline of head mounted device 2100. As shown, the illumination fields of emitter 2130a and emitter 2130b largely overlap. In this way, emitter 2130a and emitter 2130b can be configured to support imaging or depth measurement of objects in center field of view 2150 in low ambient light conditions. Fig. 23B The field of view or illumination is shown from a top-down perspective at a distance of 0.3 meters from the head-mounted device. Fig. 23B The overlap of field of view 2121a, field of view 2121b, and field of view 2141 is shown at 0.3 meters from the head mounted device. Fig.23C The field of view or illumination is shown from a frontal perspective at a distance of 0.25 meters from the head mounted device. Fig.23C The overlap of field of view 2121a, field of view 2121b, and field of view 2141 is shown at 0.25 meters from the head mounted device.

[0266] As per FIG. 23A to FIG. 23C As can be appreciated, the overlap of fields of view 2121a, 2121b, and 2141 forms a central field of view, wherein stereoscopic imaging techniques may be employed using grayscale images acquired by cameras 2120a and 2120b with or without IR illumination from emitters 2130a and 2130b. In the central field of view, color information from camera 2140 may be combined with grayscale image information. In addition, there is a peripheral field of view in which there is no overlap, but monocular grayscale image information may be obtained from one of camera 2120a or camera 2120b. As described herein, different operations may be performed on image information acquired for the central field of view and the peripheral field of view.

[0267] World Model Generation

[0268] In some embodiments, image information acquired in the central field of view may be used to construct or update a world model. Fig.24 is a simplified flow chart of a method 2400 for creating or updating a connected world model according to some embodiments. Fig.21As disclosed, the wearable display system may be configured to use a processor to determine and update a connected world model. In some embodiments, the processor may be configured to perform this determination and update based on the output of camera 2120a, camera 2120b, and camera 2140 without using transmitters 2130a and 2130b. However, in some embodiments, the connected world model may be incomplete. For example, the processor may incompletely determine the depth of a wall or other flat surface. As an additional example, the connected world model may incompletely represent objects with many corners, curved surfaces, transparent surfaces, or large surfaces, such as windows, doors, balls, tables, etc. The processor 2400 may be configured to identify such incomplete information, obtain additional information, and update the world model using the additional depth information.

[0269] In some embodiments, the transmitters 2130a and 2130b may be selectively enabled to collect additional image information based on which the connected world model is constructed or updated. In some scenarios, the processor may be configured to perform object recognition in the acquired image, select a template for the recognized object, and add information to the connected world model based on the template. In this way, the wearable display system can improve the connected world model while using few or no power-intensive components such as the transmitters 2130a and 2130b, thereby extending battery life.

[0270] Method 2400 may be initiated one or more times during operation of the wearable display system. The processor may be configured to create a connected world model when the user first turns on the system, moves to a new environment, such as walking into another room, or generally when the processor detects a change in the user's physical environment. Alternatively or additionally, method 2400 may be performed periodically during operation of the wearable display system or when a significant change in the physical world is detected or in response to user input, such as input indicating that the world model is out of sync with the physical world.

[0271] In some embodiments, all or portions of the connected world model may be stored, provided by other users of the XR system, or otherwise acquired. Thus, while the creation of a world model is described, it should be understood that method 2400 may be used for a portion of a world model while other portions of the world model are derived from other sources.

[0272] After starting in block 2401, method 2400 continues to block 2410. In block 2410, a connected world model may be created. In the illustrated embodiment, the processor may use camera 2120a and camera 2120b together with camera 2140 to create the connected world model. As described above, when generating the connected world model, the processor may be configured to use the grayscale images obtained from camera 2120a and camera 2120b to stereoscopically determine depth information of objects in the physical world when constructing the connected world model. In some embodiments, the processor may receive color information from camera 2140. The color information may be used to distinguish objects or identify surfaces associated with the same object. The color information may also be used to identify objects.

[0273] After creating the connected world model in block 2410, method 2400 continues to block 2430. In block 2430, the processor may identify surfaces and / or objects for updating the connected world model. The processor may identify these surfaces or objects using grayscale images acquired from camera 2120a and camera 2120b. In some embodiments, the processor may use these grayscale images to stereoscopically determine depth information for objects in the physical world. This depth information may be used when updating the connected world model. Alternatively or additionally, monocular information may be used to update the world model.

[0274] For example, once a world model is created in block 2410 indicating a surface at a particular location within the connected world, grayscale images acquired from camera 2120a and / or camera 2120b may be used to detect surfaces having substantially the same characteristics and determine that the connected world model should be updated by updating the location of the surface in the connected world model. For example, a surface having substantially the same shape as a surface in the connected world model at substantially the same location is equivalent to the surface in the connected world model, and the connected world model may be updated accordingly. As another example, the location of an object represented in the connected world model may be updated based on the grayscale images acquired from camera 2120a and / or camera 2120b.

[0275] Grayscale information can be stereo information or monocular information. In some embodiments, the update can be based on the stereo information of the object in the central field of view, with available stereo information, and based on the monocular information in the peripheral field of view of one of the cameras 2120a or 2120b, in which only monocular information is available. In some embodiments, the update process can be performed differently based on whether the object is in the central field of view or the peripheral field of view. For example, the update can be performed on the surface detected in the central field of view. In the peripheral field of view, for example, the update can be performed only on the object for which the processor has a model, so that the processor can confirm that any update to the connected world model is consistent with the object. Alternatively or additionally, new objects or surfaces can be identified based on the processing of grayscale images. Even if the processing results in a lower accuracy of the representation of the object or surface than the processing in box 2410, in some cases, a better overall system can be achieved by balancing the accuracy of faster and less power-consuming processing. Furthermore, by periodically repeating method 2400, low accuracy information may be periodically replaced with high accuracy information, thereby creating portions of the world model generated using only a grayscale camera and portions having information generated using a combination of a color camera and a grayscale camera.

[0276] After updating the connected world model in box 2430, method 2400 continues to box 2450. In box 2450, the processor may identify whether the connected world model includes incomplete depth information. Incomplete depth information can be generated in any variety of ways. For example, some objects do not produce detectable structures in the image. For example, extremely dark areas in the physical world cannot be imaged with a resolution sufficient to extract depth information from an image acquired using ambient lighting. As another example, a window or glass tabletop may not appear in a visible image or be recognized by computer processing. As yet another example, a large uniform surface, such as a tabletop or a wall, may lack sufficient features to be correlated in two stereo images to perform stereo image processing. Therefore, the processor cannot use stereo processing to determine the location of such objects. In these scenes, there are "holes" in the world model because the process of seeking to determine the distance to the surface using the connected world model, in a particular direction through the "hole", cannot obtain any depth information.

[0277] When the connected world model does not include incomplete depth information, method 2400 may return to updating the connected world model using grayscale images acquired from camera 2120a and camera 2120b.

[0278] After identifying the incomplete depth information, the processor controls the method 2400 to take one or more actions to obtain additional depth information. The method 2400 may continue to box 2471 and / or box 2473. In box 2471, the processor may enable the emitter 2130a and / or the emitter 2130b. As disclosed above, one or more of the cameras 2120a, 2120b, or 2140 may be configured to detect light emitted by the emitter 2130a and / or the emitter 2130b. The processor may then acquire depth information by causing the emitter 2130a and / or 2130b to emit light that can enhance the acquired image of the physical world object. For example, when the camera 2120a and the camera 2120b are sensitive to the emitted light, the images acquired with the camera 2120a and the camera 2120b may be processed to extract stereo information. When emitter 2130a and / or emitter 2130b are enabled, other analysis techniques may alternatively or additionally be used to obtain depth information.In some embodiments, time-of-flight measurement and / or structured light techniques may alternatively or additionally be used.

[0279] In block 2473, the processor may determine additional depth information based on previously acquired depth information. In some embodiments, for example, the processor may be configured to identify objects in an image formed with camera 2120a and / or camera 2120b and fill any holes in the connected world model based on a model of the identified object. For example, the process may detect planes in the physical world. These planes may be detected using existing depth information acquired by camera 2120a and / or camera 2120b or depth information stored in the connected world model. These planes may also be detected in response to determining that a portion of the world model includes incomplete depth information. The processor may be configured to estimate additional depth information based on the detected planes. For example, the processor may be configured to extend the identified plane through an area of ​​incomplete depth information. In some embodiments, the processor may be configured to interpolate missing depth information based on surrounding portions of the connected world model when extending the plane.

[0280] In some embodiments, as an additional example, the processor may be configured to detect an object in a portion of a world model that includes incomplete depth information. In some embodiments, the detection may involve using a neural network or other machine learning tool to identify the object. In some embodiments, the processor may be configured to access a database storing templates and select an object template corresponding to the identified object. For example, when the identified object is a window, the processor may be configured to access a database storing templates and select a corresponding window template. As a non-limiting example, the template may be a three-dimensional model representing a class of objects, such as windows, doors, balls, and the like. The processor may configure an instance of the object template based on an image of the object in the updated world model. For example, the processor may scale, rotate, and translate the template to match the position of the object detected in the updated world model. Additional depth information may then be estimated based on the boundaries of the configured template, representing the surface of the identified object.

[0281] After block 2471 and / or block 2473, method 2400 continues to block 2490. In block 2490, the processor may update the connected world model using the additional depth information obtained in block 2471 and / or block 2473. For example, the processor may be configured to blend additional depth information obtained from measurements made with active IR illumination into an existing connected world model. As additional examples, the processor may be configured to blend interpolated depth information obtained by extending a detected plane into an existing connected world model, or blend additional depth information estimated from a boundary of a configured template into an existing connected world model. The information may be blended in one or more ways, depending on the nature of the additional depth information and / or the information in the connected world model. For example, blending may be performed by adding additional depth information to the connected world model, the depth information collected for locations where holes exist in the connected world model. Alternatively, the additional depth information may overwrite information at corresponding locations in the connected world model. As yet another alternative, blending may involve selecting between information already in the connected world model and the additional depth information. Such a selection may, for example, be based on selecting depth information already in the connected world model or depth information in the additional depth information that represents the surface closest to a camera used to collect the additional depth information.

[0282] In some embodiments, a connected world model may be represented by a mesh of connected points. Updating the world model may be accomplished by computing a mesh representation of an object or surface to be added to the world model, and then combining that mesh representation with the mesh representation of the world model. The inventors have recognized and appreciated that performing the process in this order may require less processing effort than adding an object or surface to the world model and then computing a mesh to update the model.

[0283] Fig.24The world model updated in both blocks 2430 and 2490 is shown. The processing of each block can be performed in the same manner, such as by generating a mesh representation of the object or surface to be added to the world model, and then combining the generated mesh with the mesh of the world model, or in other ways. In some embodiments, this merging operation can be performed once for the objects or surfaces identified in blocks 2430 and 2490. This combining process can be performed, for example, in the manner described in conjunction with block 2490.

[0284] In some embodiments, method 2400 may loop back to block 2430 to repeat the process of updating the world model based on information acquired with one or more grayscale cameras. Since the processing of block 2430 is performed on fewer and smaller images than the processing of block 2410, the process may be repeated at a higher rate. The process may be performed at a rate of less than 10 times per second, such as between 3 and 7 times per second.

[0285] Method 2400 may be repeated in this manner until an end condition is detected. For example, method 2400 may be repeated for a predetermined time period until user input is received, or until a change of a particular type or magnitude is detected in the portion of the physical world model in the field of view of the camera of the head mounted device 2100. Method 2400 then terminates in box 2499. Method 2400 may be initiated again to capture new information for the world model in box 2410, including information acquired with a higher resolution color camera. Method 2400 may be terminated and restarted to repeat the processing in box 2401 using the color camera, thereby creating a portion of the world model at an average rate that is slower than the rate at which the world model is updated based solely on grayscale image information. For example, the processing using the color camera may be repeated at an average rate of once per second or slower.

[0286] Object Tracking

[0287] As described above, the processor of the XR system can track objects in the physical world to support realistic rendering of virtual objects relative to the physical objects. For example, tracking is described in conjunction with movable objects, such as the hands of a user of the XR system. Rapidly updating the position of the movable objects can enable realistic rendering of virtual objects because the rendering can reflect occlusion of physical objects by virtual objects or occlusion of virtual objects by physical objects, or reflect interactions between virtual objects in physical objects. Tracking fixed objects in the physical world is also described as part of head pose tracking. Determining the user's head pose enables information in the connected world model to be converted to a reference frame of the user's wearable display device so that the information in the connected world model can be used to render objects on the wearable display device.

[0288] According to some embodiments, grayscale, stereo information may be used to track objects in the central field of view 2150. These objects are not tracked when they are in the peripheral fields of view 2160a or 2160b. Alternatively or additionally, different tracking methods that use only monocular image information may be used for tracking in these peripheral fields of view.

[0289] Hand tracking can be used as an example of object tracking. Fig.25 is a simplified flow chart of a hand tracking method (method 2500) according to some embodiments. In various embodiments, the processor may be configured to perform hand tracking using camera 2120a and camera 2120b when the user's hand is in the central field of view 2150, using only camera 2120a when the user's hand is in the peripheral field of view 2120b, and using only camera 2120b when the user's hand is in the peripheral field of view 2120b. In this way, the wearable display system can be configured to provide adequate hand tracking using a reduced number of available cameras in this configuration, thereby reducing power consumption and extending battery life.

[0290] Method 2500 may be executed under the control of an XR system processor. The method may be initiated when an object to be tracked, such as a hand, is detected as a result of analyzing an image acquired with any camera on the head mounted device 2100. This analysis requires identifying the object as a hand based on an image area having photometric features that are characteristic of a hand. Alternatively or additionally, depth information acquired based on stereo image analysis may be used to detect the hand. As a specific example, the depth information may indicate the presence of an object whose shape matches the 3D model of the hand. Detecting the presence of a hand in this manner also requires setting the hand model parameters to match the hand orientation. In some embodiments, this model may also be used to achieve fast hand tracking by using photometric information from one or more grayscale cameras to determine how the hand has moved from an original position.

[0291] Other trigger conditions may also initiate method 2500, such as the XR system performing an operation involving a tracked object, such as rendering a virtual button that a user may attempt to press with a hand, such that the user's hand is expected to enter the field of view of one or more cameras. Method 2500 may be repeated at a higher rate, such as between 30 and 100 times per second, such as between 40 and 60 times per second. Thus, updated position information of tracked objects may be provided to be processed at low latency to render virtual objects that interact with physical objects.

[0292] After starting in block 2501, method 2500 may continue to block 2510. In block 2510, the processor may determine possible hand positions. In some embodiments, the possible hand positions are positions of objects detected in the acquired image. In embodiments where the hand is detected based on matching depth information with a 3D model of the hand, the same information may be used as the initial position of the hand in block 2510.

[0293] After box 2510, method 2500 may continue to box 2520. In box 2520, the processor may determine whether the object is in the central field of view or in the peripheral field of view. When the object is in the central field of view, method 2500 may continue to box 2531. In box 2531, the processor may obtain depth information of the object. The depth information may be obtained based on stereo image analysis, according to which the distance between the camera collecting the image information and the tracked object may be calculated. For example, the processor may select a feature in the central field of view and determine the depth information of the selected feature. The processor may use the images obtained by cameras 2120a and 2120b to stereoscopically determine the depth information of the feature.

[0294] In some embodiments, the selected features may represent different parts of a human hand defined by bones and joints. Feature selection may be based on matching image information with a human hand model. For example, this matching may be performed heuristically. For example, a human hand may be represented by a limited number of parts, such as 16 parts, and points in a hand image may be mapped to one of these parts so that features on each part may be selected. Alternatively or additionally, this matching may use a deep neural network or a classification / decision forest to apply a series of yes / no decisions in the analysis to identify different parts of the hand and select features representing different parts of the hand. For example, this matching may identify whether a specific point in the image belongs to a palm part, back of hand, non-thumb finger, thumb, fingertip and / or knuckle. Any suitable classification method may be used at this analysis stage. For example, a deep learning module or a neural network mechanism may be used as an alternative or supplement to a classification forest. In addition, in addition to the classification forest, a regression forest (e.g., using Hough transformation, etc.) may also be used.

[0295] Regardless of the specific number of features selected and the techniques used to select those features, after block 2531, method 2500 may continue to block 2550. In block 2550, the processor may then configure a hand model based on the depth information. In some embodiments, the hand model may reflect structural information about a person's hand, such as representing each bone of the hand as a part of the hand, and each joint defining a possible range of angles between adjacent parts. By assigning a position to each part in the hand model based on the depth information of the selected features, information about the position of the hand may be provided for subsequent processing by the XR system.

[0296] In some embodiments, the processing in blocks 2531 and 2550 may be performed iteratively, where the selection of features for which depth information is collected is refined based on the hand model configuration. The hand model may include shape constraints and motion constraints, and the processor may be configured to use these constraints to refine the selection of features representing the hand part. For example, when the features selected to represent the hand part indicate that the position or motion of the part violates the hand model constraints, a different feature may be selected to represent the part.

[0297] In some embodiments, photometric image information may be used as an alternative or in addition to depth information to perform successive iterations of the hand tracking process. At each iteration, the 3D hand model may be updated to reflect possible movements of the hand. Possible movements may be determined, for example, based on depth information, photometric information, or projections of hand trajectories. If depth information is used, the set of features used by the depth information is more limited than the set of features used to set the initial configuration of the hand model, thereby speeding up the process.

[0298] Regardless of how the 3D hand model is updated, the updated model can be refined based on the photometric image information. For example, the model can be used to render a virtual image of the hand, representing how the hand image is expected to appear. The expected image can be compared with the photometric image information acquired with the image sensor. The 3D model can be adjusted to reduce the error between the expected and acquired photometric information. The adjusted 3D model then provides an indication of the hand position. When this updating process is repeated, the 3D model provides an indication of the hand position as the hand moves.

[0299] In contrast, when the processing in block 2520 determines that the tracked object is in the peripheral field of view, depth information from the stereo image analysis technique is not available. Nevertheless, in the processing in block 2535, the processor may select features in the image that represent the structure of a person's hand. These features may be identified heuristically or using AI techniques, as described above for the processing in block 2531.

[0300] In box 2555, the processor may then attempt to match the selected features and the movement of those selected features between images to the hand model without utilizing depth information. Such matching may result in information that is less robust or potentially less accurate than the information generated in box 2550. Nevertheless, the information recognized based on the monocular information may still provide useful information for XR system operation. For example, the determined information may include recognition of hand movements corresponding to gestures. The gesture recognition may be performed using the hand tracking methods described in U.S. Patent Publication No. 2016 / 0026253, which is incorporated by reference in its teachings in conjunction with the use of hand tracking in an XR system and information about the hand obtained from image information.

[0301] After matching the image portion to the hand model portion in box 2550, method 2500 may end in box 2599. However, it should be understood that object tracking may occur continuously during operation of the XR system or during time intervals when the object is in the field of view of one or more cameras. Therefore, after completing one iteration of method 2500, another iteration may be performed, and the process may be performed during the time interval in which object tracking is performed. In some embodiments, information used in one iteration may be used in subsequent iterations. In various embodiments, for example, the processor may be configured to estimate an updated position of a user's hand based on previously detected hand positions. For example, the processor may estimate where the user's hand will be next based on the previous position and velocity of the user's hand. Such information can be used to reduce the amount of image information processed to detect the position of the object, as described above in conjunction with the block tracking technique.

[0302] Having thus described several aspects of some embodiments, it is to be appreciated various alterations, modifications, and improvements will readily occur to those skilled in the art.

[0303] As an example, embodiments are described in conjunction with an augmented (AR) environment. It should be understood that some or all of the techniques described herein may be applied to MR environments or more generally to other XR environments.

[0304] In addition, embodiments of image arrays are described in which a partition is applied to the image array to control the selective output of image information about a movable object. It should be understood that there can be more than one movable object in the physical environment. In addition, in some embodiments, it is necessary to selectively obtain frequent updates of image information in areas other than the area where the movable object is located. For example, the partitions can be set to selectively obtain image information about the area of ​​the physical world where the virtual object is to be rendered. Therefore, some image sensors are capable of selectively providing information from two or more partitions, with or without circuitry to track the trajectory of those partitions.

[0305] As yet another example, an image array is described as outputting information related to the amplitude of incident light. The amplitude can be a representation of power across a spectrum of light frequencies. The spectrum can capture energy relatively broadly at frequencies corresponding to any color of visible light, such as in a black and white camera. Alternatively, the spectrum can be narrower, corresponding to a single color of visible light. To this end, a filter can be used that limits the light incident on the image array to a particular color of light. In the case where pixels are restricted to receiving a particular color of light, different pixels are restricted to different colors. In such embodiments, the outputs of pixels sensitive to the same color can be processed together.

[0306] A process for setting up tiles in an image array and then updating the tiles for an object of interest is described. For example, the process may be performed for each movable object as it enters the field of view of the image sensor. When the object of interest leaves the field of view, the tiles may be cleared so that the tiles are no longer tracked or image information for the tiles is outputted. It should be understood that the tiles are updated from time to time, such as by determining the position of an object associated with the tile and setting the tile position to correspond to the object position. Similar adjustments may be performed on the calculated tile trajectories. Object motion vectors and / or image sensor motion vectors may be calculated based on other sensor information and used to reset values ​​programmed into the image sensor or other components for tile tracking.

[0307] For example, the position, motion and other characteristics of an object can be determined by analyzing the output of a wide-angle camera or a pair of cameras with stereo information. Data from these other sensors can be used to update the world model. In conjunction with the update, the block position and / or trajectory information can be updated. Such updates can occur at a rate lower than that of the block position being updated by the block tracking engine. For example, the block tracking engine can calculate new block positions at a rate between about 1 and 30 times per second. The update of the block position based on other information can occur at a slower rate, such as at intervals of once per second to about once every 30 seconds.

[0308] As yet another example of a variation, Figure 2 A system with a head mounted display separate from a remote processing module is shown. The image sensors described herein can result in a compact system design. Such sensors generate less data, which in turn results in lower processing requirements and lower power consumption. The reduced need for processing and power results in size reduction, such as through reduced battery size. Thus, in some embodiments, the entire augmented reality system can be integrated into a head mounted display without a remote processing module. The head mounted display can be configured as a pair of goggles, or as Figure 2 As shown, the size and shape are similar to a pair of glasses.

[0309] In addition, embodiments are described where the image sensor responds to visible light. It should be understood that the techniques described herein are not limited to operation with visible light. They may alternatively or additionally respond to IR light or "light" in other parts of the spectrum, such as UV. Furthermore, the image sensors described herein respond to naturally occurring light. Alternatively or additionally, the sensor may be used in a system having an illumination source. In some embodiments, the sensitivity of the image sensor may be tuned to the portion of the spectrum where the illumination source emits light.

[0310] As another example, it is described that a selected area of ​​an image array is specified by specifying a "block" on which image analysis is to be performed, and the image sensor should output changes in the selected area. However, it should be understood that the block and the selected area can be of different sizes. For example, the selected area can be larger than the block to account for the movement of the tracked object in the image deviating from the predicted trajectory and / or to allow processing around the edge of the block.

[0311] Such alterations, modifications, and improvements are intended to be part of this disclosure, and are intended to be within the spirit and scope of the disclosure.

[0312] For example, in some embodiments, image sensor pixel filter 102 may not be a separate component, but may be incorporated into one of the other components of pixel subarray 100. For example, in an embodiment including a single pixel having both an arrival angle-to-position intensity converter and a color filter, the arrival angle-to-intensity converter may be a transmissive optical component formed of a material that filters a particular wavelength.

[0313] According to some embodiments, a wearable display system is provided, wherein the wearable display system may include: a frame; two first cameras, which are mechanically coupled to the frame to provide a central field of view associated with the two cameras and a first peripheral field of view associated with a first first camera of the two first cameras; a color camera, which is mechanically coupled to the frame to provide a color field of view overlapping with the central field of view; and a processor, which is operably coupled to the two first cameras and the color camera and is configured to track hand movements in the central field of view using depth information stereoscopically determined based on first images acquired by the two first cameras; track hand movements in the first peripheral field of view using one or more second images acquired by the first first camera of the two first cameras; create a world model using the two first cameras and the color camera; and update the world model using the two first cameras.

[0314] In some embodiments, the two first cameras may be configured with a global shutter. In some embodiments, the wearable display system may further include a hardware accelerator for stereoscopically determining depth information using the first grayscale images acquired by the two first cameras. In some embodiments, the two first cameras may have equidistant lenses. In some embodiments, each of the two first cameras may have a horizontal field of view between 90 degrees and 175 degrees. In some embodiments, the central field of view may have an angular range between 40 degrees and 80 degrees. In some embodiments, the processor may also be configured to perform a calibration routine to determine the relative orientation of the two first cameras.

[0315] In some embodiments, the calibration routine may include: identifying corresponding features in images acquired by each of the two first cameras; calculating an error for each of a plurality of estimated relative orientations for the two first cameras, wherein the error indicates a difference between the corresponding feature as appearing in the images acquired by each of the two first cameras and an estimate of the identified feature calculated based on the estimated relative orientations of the two first cameras; and selecting one of the plurality of estimated relative orientations as the determined relative orientation based on the calculated error.

[0316] In some embodiments, the wearable display system may also include: a first inertial measurement unit mechanically coupled to a first first camera of the two first cameras; and a second inertial measurement unit mechanically coupled to a second first camera of the two first cameras, and the calibration routine may also include selecting at least one of a plurality of estimated relative orientations based in part on outputs of the first inertial measurement unit and the second inertial measurement unit.

[0317] In some embodiments, the processor may be further configured to repeatedly perform a calibration routine while wearing the wearable display system, such that the calibration routine compensates for frame deformation during use of the wearable display system. In some embodiments, the calibration routine may compensate for frame deformation caused by temperature changes. In some embodiments, the calibration routine may compensate for frame deformation caused by mechanical strain. In some embodiments, the two first cameras may be configured to acquire grayscale images. In some embodiments, the wearable display system may have a color camera. In some embodiments, the processor may be mechanically coupled to the frame. In some embodiments, the display device may be mechanically coupled to the frame, wherein the display device includes a processor. In some embodiments, the local data processing module may include a processor, the local data processing module may be operably coupled to the display device via a communication link, and the display device may be mechanically coupled to the frame. In some embodiments, the first image may include one or more second images.

[0318] According to some embodiments, a wearable display system is provided, wherein the wearable display system may include: a frame; two first cameras mechanically coupled to the frame; a color camera mechanically coupled to the frame; and a processor operably coupled to the two first cameras and the color camera and configured to: create a world model using first grayscale images acquired by the two first cameras and one or more color images acquired by the color camera; update the world model using second grayscale images acquired by the two first cameras; determine that a portion of the world model includes incomplete depth information; and update the world model with additional depth information for a portion of the world model including the incomplete depth information.

[0319] In some embodiments, the wearable display system may further include one or more emitters, and the one or more emitters may be used to obtain additional depth information, and the processor may be further configured to enable the one or more emitters in response to determining that a portion of the world model includes incomplete depth information. In some embodiments, the one or more emitters may include an infrared emitter, and the two first cameras may include a filter configured to pass infrared light. In some embodiments, the infrared emitter may emit light with a wavelength between 900 nanometers and 1 micron.

[0320] In some embodiments, the processor may be further configured to detect a flat surface in the physical world in response to determining that a portion of the world model includes incomplete depth information, and may estimate additional depth information based on the detected flat surface. In some embodiments, the processor may be further configured to detect an object in a portion of the world model that includes incomplete depth information, identify an object template corresponding to the detected object, configure an instance of the object template based on an image of the object in the updated world model, and estimate additional depth information based on the instance of the configured object template. In some embodiments, the two first cameras may be mechanically coupled to the frame to provide a central field of view associated with the two first cameras and a peripheral field of view associated with a first of the two first cameras, and the processor may be further configured to track hand movement in the central field of view using depth information determined based on images acquired by the two first cameras.

[0321] In some embodiments, tracking hand movements in the central visual field using depth information may include: selecting a point in the central visual field; using images acquired by two first cameras, stereoscopically determining depth information of the selected point; generating a depth map using the stereoscopically determined depth information; and matching a portion of the depth map with a corresponding portion of a hand model that includes both shape constraints and motion constraints. In some embodiments, the processor may also be configured to track hand movements in the peripheral visual field using one or more images acquired by a first first camera of the two first cameras by matching a portion of the image with a corresponding portion of a hand model that includes both shape constraints and motion constraints. In some embodiments, the processor may be mechanically coupled to the frame. In some embodiments, a display device mechanically coupled to the frame may include a processor. In some embodiments, a local data processing module may include a processor, the local data processing module may be operably coupled to a display device via a communication link, and the display device may be mechanically coupled to the frame.

[0322] According to some embodiments, a wearable display system is provided, wherein the wearable display system may include: a frame; two grayscale cameras mechanically coupled to the frame, wherein the two grayscale cameras include a first grayscale camera having a first field of view and a second grayscale camera having a second field of view, and the first grayscale camera and the second grayscale camera are positioned to provide: a central field of view in which the first field of view overlaps with the second field of view; and a first peripheral field of view within the first field of view and outside the second field of view; and a color camera mechanically coupled to the frame to provide a color field of view overlapping with the central field of view.

[0323] In some embodiments, the two grayscales may have a global shutter. In some embodiments, the color camera may have a rolling shutter. In some embodiments, the wearable display system may include two inertial measurement units mechanically coupled to a frame, and one or more emitters mechanically coupled to the frame. In some embodiments, one or more emitters may be infrared emitters, and the two grayscale cameras may include a filter configured to pass infrared light. In some embodiments, the infrared emitter may emit light with a wavelength between 900 nanometers and 1 micron. In some embodiments, the illumination field of one or more emitters may overlap with the central field of view. In some embodiments, a first inertial measurement unit of the two inertial measurement units may be mechanically coupled to a first grayscale camera, and a second inertial measurement unit of the two inertial measurement units may be mechanically coupled to a second grayscale camera of the two grayscale cameras. In some embodiments, the two inertial measurement units may be configured to measure inclination, acceleration, velocity, or any combination thereof. In some embodiments, the two grayscale cameras may have a horizontal field of view between 90 degrees and 175 degrees. In some embodiments, the central field of view may have an angular range between 40 degrees and 80 degrees.

[0324] In addition, although the advantages of the present disclosure are pointed out, it should be understood that not every embodiment of the present disclosure includes every described advantage. Some embodiments may not implement any feature described as advantageous herein. Therefore, the foregoing description and drawings are only intended as examples.

[0325] The above-mentioned embodiments of the present disclosure can be implemented in any of a variety of ways. For example, the embodiments can be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or processor set, whether these processors or processor sets are provided in a single computer or distributed in multiple computers. These processors can be implemented as integrated circuits, wherein one or more processors are provided in an integrated circuit component, including commercially available integrated circuit components known in the art, such as CPU chips, GPU chips, microprocessors, microcontrollers, or coprocessors. In some embodiments, the processor can be implemented in a custom circuit, such as an ASIC, or a semi-custom circuit generated by configuring a programmable logic device. As another alternative, the processor can be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. As a specific example, some commercially available microprocessors have multiple cores, so that one of these cores or a subset thereof can constitute a processor. However, the processor can be implemented using a circuit of any suitable format.

[0326] Furthermore, it should be understood that a computer may be embodied in any of a variety of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. In addition, a computer may be embedded in a device not generally regarded as a computer but with appropriate processing capabilities, including a personal digital assistant (PDA), a smart phone, or any other suitable portable or fixed electronic device.

[0327] In addition, the computer may have one or more input and output devices. Among other things, these devices can be used to present a user interface. Examples of output devices that can be used to provide a user interface include a printer or display screen for visual presentation of the output and a speaker or other sound generating device for audible presentation of the output. Examples of input devices that can be used for the user interface include keyboards and pointing devices, such as mice, touch pads, and digitized tablets. As another example, the computer can receive input information through voice recognition or other audible formats. In the illustrated embodiment, the input / output device is shown as being physically separated from the computing device. However, in some embodiments, the input and / or output device can be physically integrated into the same unit with the processor or other elements of the computing device. For example, a keyboard may be implemented as a soft keyboard on a touch screen. In some embodiments, the input / output device can be completely disconnected from the computing device and functionally integrated through a wireless connection.

[0328] Such computers may be interconnected in any suitable form via one or more networks, including local or wide area networks, such as an enterprise network or the Internet. These networks may be based on any suitable technology and may operate according to any suitable protocol and may include wireless networks, wired networks, or fiber optic networks.

[0329] In addition, the various methods or processes outlined herein may be encoded as software that can be executed on one or more processors that employ any of a variety of operating systems or platforms. In addition, such software may be written using any of a variety of suitable programming languages ​​and / or programming or scripting tools, and may also be compiled into executable machine language encoding or intermediate encoding that is executed on a framework or virtual machine.

[0330] In this regard, the present disclosure may be embodied as a computer-readable storage medium (or multiple computer-readable media) (e.g., a computer memory, one or more floppy disks, compact disks (CDs), optical disks, digital video disks (DVDs), tapes, flash memory, circuit configurations in field programmable gate arrays or other semiconductor devices, or other tangible computer storage media) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement various embodiments of the above disclosure. As is apparent from the above examples, a computer-readable storage medium may retain information for a sufficient period of time to provide computer-executable instructions in a non-transitory form. Such a computer-readable storage medium or medium may be transportable so that one or more programs stored thereon may be loaded onto one or more different computers or other processors to implement various aspects of the present disclosure as described above. As used herein, the term "computer-readable storage medium" covers only computer-readable media that can be considered as an article of manufacture (i.e., an article of manufacture) or a machine. In some embodiments, the present disclosure may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagating signal.

[0331] The term "program" or "software" is used in this article in a general sense to indicate any type of computer code or computer executable instruction set that can be used to program a computer or other processor to implement the various aspects of the present disclosure described above. In addition, it should be understood that according to one aspect of this embodiment, when performing the method of the present disclosure, one or more computer programs do not need to reside on a single computer or processor, but can be distributed in a modular manner on multiple different computers or processors to implement various aspects of the present disclosure.

[0332] Computer executable instructions can be in various forms, such as program modules, which are executed by one or more computers or other devices. Typically, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Typically, the functions of program modules can be combined or distributed as needed in various embodiments.

[0333] In addition, the data structure can be stored in a computer-readable medium in any suitable form. For simplicity, the data structure can be shown as having fields that are related by position in the data structure. This relationship can also be achieved by allocating storage for the fields, which has positions in the computer-readable medium that convey the relationship between the fields. However, any suitable mechanism can be used to establish a relationship between the information in the fields of the data structure, including by using pointers, tags, or other mechanisms that establish relationships between data elements.

[0334] Various aspects of the present disclosure may be used alone, in combination, or in various arrangements not specifically discussed in the embodiments described above, and therefore their application is not limited to the details and arrangements of the components set forth in the above description or shown in the accompanying drawings. For example, aspects described in one embodiment may be combined in any manner with aspects described in other embodiments.

[0335] In addition, the present disclosure may be embodied as a method, examples of which have been provided. The actions performed as part of the method may be ordered in any suitable manner. Thus, embodiments may be configured to perform actions in an order different from that shown, which may include performing some actions simultaneously, even if the actions are shown as sequential actions in the illustrative embodiments.

[0336] The use of ordinal terms such as "first", "second", "third", etc. to modify claim elements in the claims does not itself imply any priority, precedence or order of one claim element relative to another claim element, nor does it imply a temporal order of actions in performing a method, but is merely used as a label to distinguish one claim element with a specific name from another element with the same name (if no ordinal terms are used) to distinguish the claim elements.

[0337] In addition, the phraseology and terminology used herein are for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," or "having," "containing," "involving," and variations thereof herein is intended to encompass the items listed thereafter and equivalents thereof as well as additional items.

Claims

1. A wearable display system, comprising: Head mounted device; two first cameras mechanically coupled to the head mounted device; a first inertial measurement unit mechanically coupled to a first of the two first cameras; and a second inertial measurement unit mechanically coupled to a second of the two first cameras; as well as a processor operably coupled to the two first cameras and configured to: A calibration routine is executed, the calibration routine configured to determine a relative orientation of the two first cameras using images acquired by the two first cameras and outputs of the first inertial measurement unit and the second inertial measurement unit.

2. The wearable display system according to claim 1, wherein: The calibration routine, when executed, further determines the relative positions of the two first cameras.

3. The wearable display system according to claim 1, wherein: Executing the calibration routine includes: identifying corresponding features in images acquired with each of the first first camera and the second first camera; for each of a plurality of estimated relative orientations of the two first cameras, calculating an error, wherein the error indicates a difference between the corresponding feature appearing in an image acquired with each of the two first cameras and an estimate of the identified feature calculated based on the estimated relative orientations of the two first cameras; and Based on the calculated error, a relative orientation from the plurality of estimated relative orientations is selected as the determined relative orientation.

4. The wearable display system according to claim 2, wherein: The calibration routine also includes selecting at least an initial estimate of the plurality of estimated relative orientations based in part on the outputs of the first and second inertial measurement units.

5. The wearable display system according to claim 1, wherein: The wearable display system further comprises: a color camera coupled to the head mounted device; and The processor is further configured to: creating a world model using the two first cameras and the color camera; and updating the world model using the two first cameras at a first rate; and The world model is updated using the two first cameras and the color camera at a second rate that is slower than the first rate.

6. The wearable display system according to claim 1, wherein: The two first cameras are mechanically coupled to the head mounted device so as to provide a central field of view associated with both of the first cameras and a peripheral field of view associated with a first of the two first cameras; as well as The processor is also configured to track hand movement in the central field of view using depth information determined from images acquired by the two first cameras.

7. The wearable display system according to claim 6, wherein: Using depth information to track hand motion in the central field of view includes: selecting a point in the central field of view; Using the images acquired by the two first cameras, stereoscopically determining depth information of the selected point; generating a depth map using the stereoscopically determined depth information; and Portions of the depth map are matched to corresponding portions of a hand model that includes both shape constraints and motion constraints.

8. The wearable display system according to claim 6, wherein: The processor is further configured to match portions of the image with corresponding portions of a hand model including both shape constraints and motion constraints, thereby tracking hand motion in the peripheral field of view using one or more images acquired by the two first cameras.

9. The wearable display system according to claim 1, wherein: The head mounted device is a lightweight head mounted device weighing between 30 and 300 grams.

10. The wearable display system according to claim 9, wherein: The head mounted device also includes the processor.