Arrangement of imaging and illumination sensors for augmented reality head mounted viewer and system and method of use thereof

Abstract

Arrangements of imaging sensors and illumination sensors for augmented reality head mounted viewers, and systems and methods of use thereof. A housing of the MR head wearable device includes one or more displays located within an interior surface of the housing, the one or more displays configured to cause presentation of an augmented reality environment. The housing includes an object tracking assembly disposed on an outer surface of the housing. The object tracking assembly includes: a plurality of imaging devices aligned along a first axis; and an illumination device aligned along a second axis perpendicular to the first axis, the illumination device disposed at a predetermined intermediate distance between at least two respective imaging devices of the plurality of imaging devices. While the MR head wearable device is performing an operation, the object tracking component is configured to determine, based on imaging data obtained by the plurality of imaging devices while the lighting device generates ambient lighting conditions, that the obtained imaging data satisfies an object tracking threshold such that a tracked object is presented via the one or more displays.

Description

[0001] Cross - Reference to Related Applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 618,853, filed on January 8, 2024, entitled "Arrangements of Imaging and Illumination Sensors for Extended - reality Headset, Physical Button for Passthrough Mode, Drop Protection and Audio Improvements for the Headset, and Systems and Methods of Use Thereof", and to U.S. Non - Provisional Application No. 18 / 885,455, filed on September 13, 2024. The entire contents of these applications are incorporated herein by reference.

[0003] This application also relates to U.S. Application No. 18 / 774,858, filed on July 16, 2024, entitled "Techniques for Using Floodlight LEDs When Imaging Sensors Have an Insufficient Level of Detail for Identifying Hand Gestures, and Mixed - Reality Systems and Methods of Using These Techniques", and to U.S. Application No. 18 / 782,385, filed on July 24, 2024, entitled "Techniques for Guiding Perspiration to Desired Channels to Avoid Negative Impacts to Electrical and Mechanical Functions of Extended - Reality Devices, and Systems and Methods of use thereof". The entire contents of these applications are incorporated herein by reference. Field of the Invention

[0004] This generally relates to mixed - reality (MR) head - mounted viewers and their components, including but not limited to object - tracking components that include imaging devices and illumination devices. Background Art

[0005] An MR head-mounted viewer is capable of presenting MR content to a user, which can be immersive and engagingly interactive. Such a presentation technique presents new opportunities as well as new challenges, especially since such content is suitable for different types of interaction used for more traditional digital content (e.g., desktop computer graphics, smartphones). Additionally, such interaction may require specific conditions (e.g., lighting conditions) to be present in order to be detected with sufficient accuracy for interacting with the MR content. Moreover, such interaction may be detrimentally affected by the auditory performance (e.g., input interaction at a microphone and / or output interaction at a speaker), which can disrupt and / or otherwise affect the sense of immersion provided by the presentation of the MR content.

[0006] Accordingly, there is a need to address one or more of the multiple challenges identified above. A solution to the problems pointed out above is briefly outlined below. SUMMARY OF THE INVENTION

[0007] Embodiments described herein include using a resonant frequency attenuation channel housed within a housing of an MR head-mounted viewer, the channel cyclically exhausting air from a fan cooling component of the MR head-mounted viewer.

[0008] In an example embodiment, an MR head-wearable device is provided. The MR head-wearable device includes one or more displays located within an inner surface of a housing, the one or more displays being configured to cause an extended reality environment to be presented when the user is wearing the MR head-wearable device. The MR head-wearable device includes an object tracking component disposed on an outer surface of the housing. The object tracking component includes: a plurality of imaging devices aligned along a first axis; and a lighting device aligned along a second axis perpendicular to the first axis, the lighting device being disposed at a predetermined intermediate distance between at least two corresponding ones of the plurality of imaging devices. When the MR head-wearable device is performing an operation, the object tracking component is configured to determine that the acquired imaging data satisfies an object tracking threshold based on the imaging data acquired by the plurality of imaging devices when the lighting device generates ambient lighting conditions, so as to cause a tracked object to be presented via the one or more displays.

[0009] The devices and / or systems described herein may be configured to include instructions that cause methods and operations associated with presenting extended reality and / or interacting with extended reality to be performed. These methods and operations may be stored on a non-transitory computer-readable storage medium of the device or system. It should also be noted that the devices and systems described herein may be part of an overall system that includes multiple devices. A non-exhaustive list of the various electronic devices, either alone or in combination (e.g., a system), that may include instructions that cause methods and operations associated with presenting extended reality and / or interacting with extended reality to be performed includes: extended reality head-mounted viewers (e.g., an MR head-mounted viewer or an augmented-reality (AR) head-mounted viewer, as two examples), wrist-wearable devices, intermediate processing devices, smart textile-based apparel, etc. For example, when describing an XR head-mounted viewer, it can be understood that the XR head-mounted viewer may communicate with one or more other devices (e.g., a wrist-wearable device, a server, an intermediate processing device, etc.), and these devices together may include instructions for performing methods and operations associated with presenting the extended reality head-mounted viewer and / or interacting with the extended reality head-mounted viewer (i.e., the XR head-mounted viewer will be part of a system that includes one or more additional devices). A variety of combinations with different associated devices are envisioned, but for the sake of brevity, they are not described further herein.

[0010] The features and advantages described in the specification are not necessarily all encompassing, and in particular, certain additional features and advantages will be apparent to those of ordinary skill in the art from the accompanying drawings, the specification, and the claims. In addition, it should be noted that the language used in the specification has been selected primarily for readability and guidance purposes.

[0011] Having summarized the foregoing example aspects, a brief description of the drawings will now be presented. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] To better understand the described embodiments, reference should be made to the following detailed description in conjunction with the following drawings, in which like reference numerals refer to corresponding parts in all the figures.

[0013] Figures 1A to 1D An example head-wearable device according to some embodiments is shown.

[0014] Figures 2A to 2E Examples of an imaging device and an illumination device of an object tracking component according to some embodiments and data related to these examples are shown.

[0015] Figures 3A to 3E The coverage area of an illumination device of a head-wearable device according to some embodiments is shown.

[0016] Figure 4 Shows an alternative configuration of a head - wearable device according to some embodiments.

[0017] Figure 5A , Figure 5B , Figure 5C-1 and Figure 5C-2 Shows example MR systems and AR systems according to some embodiments.

[0018] Figure 6 Shows an example method of using an object tracking component to determine whether an object tracking criterion is met such that a tracked object is presented by an MR head - wearable device according to some embodiments.

[0019] By convention, the various features shown in the drawings may not be drawn to scale. Accordingly, for clarity, the sizes of the various features may be arbitrarily enlarged or reduced. In addition, some of these drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and the drawings, the same reference numerals may be used to denote the same features. Detailed Description

[0020] Numerous details are described herein to provide a thorough understanding of the example embodiments shown in the drawings. However, some embodiments may be practiced without many of these specific details, and the scope of the claims is limited only by those features and aspects specifically recited in the claims. In addition, well - known processes, components, and materials may not have been described in exhaustive detail to avoid obscuring the relevant aspects of the embodiments described herein.

[0021] Embodiments of the present disclosure may include different types of extended reality (XR), or be implemented in combination with different types of extended reality (XR), such as MR systems and AR systems. As described herein, mixed reality and augmented reality are any superimposed functions and / or sensor-detectable presentations provided by an MR system or an AR system in a user's physical environment. Such mixed reality may include and / or represent virtual reality, in which at least some aspects of the surrounding environment are reconstructed in a virtual environment (e.g., displaying a virtual reconstruction of a physical object in the physical environment to prevent the user from colliding with the physical object in the surrounding physical environment). In the case of mixed reality, the surrounding environment presented via a display is acquired by one or more sensors configured to acquire the surrounding environment (e.g., a camera sensor, a time-of-flight (ToF) sensor). Although the wearer of an MR head-mounted viewer can see all the details of the surrounding environment, in some embodiments, they typically see a reconstruction of the environment reproduced using data from one or more sensors (i.e., the user does not directly view the physical object).

[0022] The MR head-mounted viewer may also forego displaying the reconstruction of the objects in the physical environment, thereby providing the user with a full virtual-reality (VR) experience. On the other hand, an AR system provides an experience in which information is provided, for example, by using waveguides, in combination with directly viewing at least some of the surrounding environment through one or more transparent or translucent waveguides and / or lenses of an AR head-mounted viewer. Throughout this application, the term "XR" is used to encompass both augmented reality and mixed reality. Additionally, this application sometimes also uses the terms "head-wearable device" or "head-mounted viewer device" to describe a head-mounted viewer, such as an AR head-mounted viewer and an MR head-mounted viewer.

[0023] As mentioned above, the MR environment as described herein may include, but is not limited to, a non-immersive VR environment, a semi-immersive VR environment, and a fully immersive VR environment. Also as mentioned above, the AR environment may include a marker-based augmented reality environment, a markerless augmented reality environment, a location-based augmented reality environment, and a projection-based augmented reality environment. The above description is not exhaustive, and any other environment that allows purposeful environmental lighting to reach the user will fall within the scope of augmented reality, while any other environment that does not allow purposeful environmental lighting to reach the user will fall within the scope of mixed reality.

[0024] AR content and MR content can include video, audio, haptic events, or some combination thereof, any of which can be presented in a single channel or in multiple channels (e.g., stereoscopic video that produces a three-dimensional effect for a viewer). Additionally, in some embodiments, AR and MR can be associated with an application, product, accessory, service, or some combination thereof, such as for creating content in an AR environment or an MR environment and / or otherwise using in an AR environment and an MR environment (e.g., performing an activity in an AR environment and an MR environment).

[0025] Interactions with the AR environments and MR environments described herein can be performed using a variety of different modalities, and the resulting output can also be across a variety of modalities. In one example AR system or MR system, a user can perform an air swipe gesture to skip a song via an application programming interface (API) provided by the song: the song-provided API provides playback at, for example, a home speaker.

[0026] As described herein, gestures can include air gestures, surface contact gestures, and / or other gestures that can be detected and determined based on the movement of a single hand (e.g., a single hand gesture performed with the user's hand, which is detected by one or more sensors of a wearable device (e.g., electromyography (EMG) sensors and / or inertial measurement unit (IMU) of a wrist wearable device, and / or one or more sensors included in a smart textile wearable device), and / or detected via image data acquired by an imaging device of the wearable device (e.g., a camera of a head wearable device, an external tracking camera setup in the surrounding environment, etc.)). "Air" means that the user's hand does not touch a surface, an object, or a part of an electronic device (e.g., a head wearable device or another communicatively coupled device, such as a wrist wearable device); in other words, the gesture is performed in an open space in 3D space and does not touch a surface, an object, or an electronic device. More generally, surface contact gestures (contact at a surface, an object, a body part of the user, or an electronic device) are also considered, in which contact (or the intention of contact) is detected at the surface (e.g., a single-finger or double-finger tap on a table, on the user's hand or another finger, on the user's leg, a couch, a steering wheel, etc.). Different gestures disclosed herein can be detected using image data and / or sensor data (e.g., neuromuscular signals sensed by one or more biopotential sensors (e.g., EMG sensors) or other types of data from other sensors (e.g., proximity sensors, time-of-flight (ToF) sensors, sensors of an inertial measurement unit (IMU), capacitance sensors, strain sensors, etc.)), and these image data and / or sensor data are detected by a wearable device worn by the user and / or other electronic devices owned by the user (e.g., a smartphone, a laptop computer, an imaging device, an intermediate device, and / or other devices described herein).

[0027] The input modalities mentioned above can be different and depend on the user's experience. For example, in an interaction using a wrist wearable device, the user can use air gestures or surface contact gestures to provide input, and these air gestures or surface contact gestures are detected using the neuromuscular signal sensors of the wrist wearable device. In the case of not using a wrist wearable device, alternative and fully interchangeable input modalities (e.g., one or more cameras located in a head-mounted viewer or elsewhere) can be used instead to detect air gestures or surface contact gestures or input at an intermediate processing device (e.g., input via physical input components (e.g., buttons and touchpads)). These different input modalities can be interchangeable based on the desired user experience, portability, and / or feature set of the product (e.g., a low-cost product may not include a hand-tracking camera).

[0028] When the input changes, the output generated from the input also changes. For example, an air gesture input detected by a camera of a head-mounted device can cause an output to be generated at the head-mounted device or control another electronic device different from the head-mounted device. In another example, an input detected using data from a neuromuscular signal sensor can also cause an output to be generated at the head-mounted device or control another electronic device different from the head-mounted device. Although only a few examples are described above, those skilled in the art will understand that different input modalities and different output modalities in response to the input are interchangeable.

[0029] The above-described specific operations can occur as a result of specific hardware. The devices described are not restrictive, and features on these devices can be removed or additional features can be added. Different devices can include one or more similar hardware components. For the sake of brevity, only similar devices and components are described herein. Any differences in the devices and components will be described in their respective sections below.

[0030] As described herein, a processor (e.g., a central processing unit (CPU) or a microcontroller unit (MCU)) is an electronic component responsible for executing instructions and controlling the operation of an electronic device (e.g., a wrist-wearable device, a head-wearable device, a handheld intermediate processing device (HIPD), a textile-based smart garment, or other computer system). There are different types of processors, and these different types of processors can be used interchangeably or specifically required by the embodiments described herein. For example, the processor can be: (i) a general-purpose processor, which is designed to perform a wide range of tasks, such as running software applications, managing an operating system, and performing arithmetic and logical operations; (ii) a microcontroller, which is designed for specific tasks, such as controlling electronic devices, sensors, and motors; (iii) a graphics processing unit (GPU), which is designed to accelerate the creation and rendering of images, videos, and animations (e.g., virtual reality animations, such as three-dimensional modeling); (iv) a field-programmable gate array (FPGA), which can be programmed and reconfigured after manufacturing and / or can be customized to perform specific tasks, such as signal processing, encryption, and machine learning; or (v) a digital signal processor (DSP), which is designed to perform mathematical operations on signals (e.g., audio, video, and radio waves). Those skilled in the art will understand that one or more processors of one or more electronic devices can be used in the various embodiments described herein.

[0031] As described herein, a controller is an electronic component that manages and coordinates the operation of other components within an electronic device (e.g., controlling inputs, processing data, and / or generating outputs). Examples of controllers can include: (i) a microcontroller, which includes a small, low-power controller commonly used in embedded systems and Internet of Things (IoT) devices; (ii) a programmable logic controller (PLC), which can be configured to be used in an industrial automation system to control and monitor a manufacturing process; (iii) a system-on-a-chip (SoC) controller, which integrates multiple components (e.g., a processor, memory, I / O interfaces, and other peripherals) into a single chip; and / or (iv) a DSP. As described herein, a graphics module is a component or software module designed to process graphics operations and / or graphical processes, and the graphics module can include a hardware module and / or a software module.

[0032] As described herein, a memory refers to an electronic component in a computer or electronic device that stores data and instructions for access and operation by a processor. Each device described herein may include volatile memory and non-volatile memory. Examples of memory may include: (i) random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid-state memory devices, which are configured to temporarily store data and instructions; (ii) read-only memory (ROM), which is configured to permanently store data and instructions (e.g., one or more portions of system firmware and / or a bootloader); (iii) flash memory, disk storage devices, optical disc storage devices, and other non-volatile solid-state storage devices that can be configured to store data in an electronic device (e.g., universal serial bus (USB) drives, memory cards, and / or solid-state drives (SSD)); and (iv) cache memory, which is configured to temporarily store frequently accessed data and instructions. The memory as described herein may include structured data (e.g., Structured Query Language (SQL) databases, MongoDB databases, GraphQL data, or JSON data). Other examples of memory may include: (i) profile data, including user account data, user settings, and / or other user data stored by a user; (ii) sensor data detected and / or otherwise obtained by one or more sensors; (iii) media content data, including stored image data, audio data, and documents, etc.; (iv) application data, which may include data collected and / or otherwise obtained and stored during the use of an application; and / or any other type of data described herein.

[0033] As described herein, the power system of an electronic device is configured to convert input power into a form that can be used to operate the device. The power system can include various components, including: (i) a power supply, which can be an alternating current (AC) adapter power supply or a direct current (DC) adapter power supply; (ii) a charger input, which can be configured to use a wired connection and / or a wireless connection (which can be part of a peripheral interface, such as a USB, micro-USB interface, near-field magnetic coupling, magnetic induction and magnetic resonance charging, and / or radio frequency (RF) charging); (iii) a power management integrated circuit, which is configured to distribute power to various components of the device and ensure that the device operates within safe limits (e.g., regulate voltage, control current, and / or manage heat dissipation); and / or (iv) a battery, which is configured to store power to provide available power to one or more components of an electronic device.

[0034] As described herein, a peripheral interface is an electronic component (e.g., an electronic component of an electronic device) that allows an electronic device to communicate with other devices or peripherals and can provide a means for inputting and outputting data and signals. Examples of peripheral interfaces can include: (i) a USB and / or micro-USB interface, which is configured to connect a device to an electronic device; (ii) a Bluetooth interface, which is configured to allow devices to communicate with each other, and the Bluetooth interface includes Bluetooth low energy (BLE); (iii) a near-field communication (NFC) interface, which is configured as a short-range wireless interface for operations such as access control; (iv) POGO pins, which can be small spring-loaded pins configured to provide a charging interface; (v) a wireless charging interface; (vi) a global-positioning system (GPS) interface; (vii) a Wi-Fi interface, which is used to provide a connection between a device and a wireless network; and (viii) a sensor interface.

[0035] As described herein, a sensor is an electronic component (e.g., an electronic component in an electronic device (e.g., a wearable device) and / or an electronic component that otherwise electronically communicates with an electronic device) configured to detect physical and environmental changes and generate an electrical signal. Examples of sensors can include: (i) an imaging sensor for collecting imaging data (e.g., including one or more cameras disposed on a corresponding electronic device, such as a camera supporting Simultaneous Localization and Mapping (SLAM)); (ii) a bioelectrical potential signal sensor; (iii) an inertial measurement unit (e.g., IMU) for detecting changes such as angular velocity, force, magnetic field, and / or acceleration; (iv) a heart rate sensor for measuring a user's heart rate; (v) a blood oxygen saturation (SpO2) sensor for measuring a user's blood oxygen saturation and / or other biometric data; (vi) a capacitance sensor for detecting capacitance changes at a location on a user's body and / or near another device or object (e.g., a sensor-skin interface); (vii) a sensor for detecting some input (e.g., a capacitance sensor and a force sensor); and (viii) a light sensor (e.g., a ToF sensor, an infrared light sensor, or a visible light sensor) and / or a sensor for sensing data from a user or a user's environment. As described herein, a bioelectrical potential signal sensing component is a device for measuring electrical activity within a body (e.g., a bioelectrical potential signal sensor). Bioelectrical potential signal sensors include, for example: (i) an electroencephalography (EEG) sensor configured to measure electrical activity in the brain to diagnose neurological disorders; (ii) an electrocardiography (ECG or EKG) sensor configured to measure electrical activity of the heart to diagnose heart problems; (iii) an electromyography (EMG) sensor configured to measure electrical activity of muscles and diagnose neuromuscular disorders; and (iv) an electrooculography (EOG) sensor configured to measure electrical activity of eye muscles to detect eye movements and diagnose eye diseases.

[0036] As described herein, an application (e.g., software) stored in the memory of an electronic device includes instructions stored in the memory. Examples of such applications include: (i) games; (ii) word processors; (iii) messaging applications; (iv) media streaming applications; (v) financial applications; (vi) calendars; (vii) clocks; (viii) web browsers; (ix) social media applications; (x) camera applications; (xi) web-based applications; (xii) health applications; (xiii) AR applications and MR applications; and / or any other application that can be stored in the memory. These applications can operate in conjunction with data and / or one or more components of a device or a device in communication coupling to perform one or more operations and / or functions.

[0037] As described herein, a communication interface module may include hardware and / or software capable of data communication using any of the following various protocols: custom wireless protocols or standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.11a, WirelessHART, or MiWi); custom wired protocols or standard wired protocols (e.g., Ethernet or HomePlug); and / or any other suitable communication protocol, including communication protocols not yet developed as of the filing date of this document. A communication interface is a mechanism that enables different systems or devices to exchange information and data with each other, including hardware, software, or a combination of hardware and software. For example, a communication interface may refer to a physical connector and / or port on a device that enables communication with other devices (e.g., USB, Ethernet, High-Definition Multimedia Interface (HDMI), or Bluetooth). A communication interface may also refer to a software layer that enables different software programs to communicate with each other (e.g., an application programming interface (API) and protocols such as Hypertext Transfer Protocol (HTTP) and Transmission Control Protocol / Internet Protocol (TCP / IP)).

[0038] As described herein, a graphics module is a component or software module designed to process graphics operations and / or graphical processes, and the graphics module may include a hardware module and / or a software module.

[0039] As described herein, a non-transitory computer-readable storage medium is a physical device or storage medium that can be used to store electronic data in a non-transitory form (e.g., such that the data is permanently stored until it is intentionally deleted or modified).

[0040] Figures 1A to 1D An example MR head-mounted viewer according to some embodiments is shown. The MR head-mounted viewer 100 includes a housing 110, one or more displays, one or more object tracking components 120, and one or more processors. Additional components of the MR head-mounted viewer 100 are described below with reference to Figures 5A to 5C-2 Describe additional components of the MR head-mounted viewer 100.

[0041] The housing 110 includes an inner surface and an outer surface opposite the inner surface. The housing 110 occludes the user's field of view when the user wears the MR head-mounted viewer 100 (as Figure 5C-1depicted, where user 502 is wearing an MR head-mounted viewer 532 that may include some or all of the components of the plurality of components of the MR head-mounted viewer 100. In particular, the housing 110 covers the user's eyes to allow for the creation of an immersive environment. One or more displays are disposed within the inner surface of the housing such that the head-mounted device causes the presentation of an extended reality environment when worn by the user. In some embodiments, the housing 110 is a two-part housing configured to have a first part and a second part that are coupled together to form a housing for the electronic and mechanical components for presenting MR content.

[0042] One or more object tracking components 120 are disposed on the outer surface of the housing 110. Each object tracking component 120 includes a plurality of imaging devices 122 (e.g., a first imaging device 122a and a second imaging device 122b) and / or one or more lighting devices 124. In some embodiments, the plurality of imaging devices 122 are composed of different types of imaging devices. For example, the first imaging device 122a may be a red, green, blue (RGB) camera, and the second imaging device 122b may be a camera that supports simultaneous localization and mapping (SLAM). One or more lighting devices 124 may be one or more light-emitting diodes (LEDs), such as floodlight LEDs, infrared (IR) light sources, lamps, etc. In some embodiments, as will be described in more detail below, one or more lighting devices 124 include floodlight LEDs that are configured and arranged to illuminate a volume of a physical space in which the user will perform gestures for interacting with the MR content.

[0043] For each object tracking component 120, a plurality of imaging devices 122 are aligned on a first axis (e.g., the y-axis), and at least one illumination device 124 is aligned on a second axis perpendicular to the first axis. The illumination device 124 is disposed at a predetermined intermediate distance between at least two of the plurality of imaging devices (e.g., intermediate between the first imaging device 122a and the second imaging device 122b). For example, as shown on the MR head-mounted viewer 100, the object tracking components 120 form a triangular arrangement on the outer surface of the housing 110. In some embodiments, the first imaging device 122a (e.g., an RGB camera) is disposed above the second imaging device 122b (e.g., a SLAM camera) such that the first imaging device 122a is as close as possible to the user's actual field of view. In some embodiments, the second imaging device 122b is tilted downward such that the field of view of the second imaging device 122b is focused on tracking the user's hand. Similarly, according to some embodiments, the illumination device 124 is tilted slightly downward in order to illuminate the user's hand, thereby allowing the user's hand to be tracked via the second imaging device 122b during conditions such as: low light conditions; low contrast background conditions; and / or other ambient lighting conditions that negatively impact the detection of objects in the image data. In some embodiments, the MR head-mounted viewer 100 includes at least two object tracking components 120 (e.g., a first object tracking component 120a and a second object tracking component 120b, where the second object tracking component 120b is a mirror image of the first object tracking component 120a).

[0044] One or more of the above processors may be configured to execute one or more programs stored in a memory communicatively coupled to the one or more processors. The one or more programs include instructions for causing the MR head-mounted viewer 100 to generate ambient lighting conditions via the lighting device 124 and receive image data via the plurality of imaging devices 122. The one or more programs further include instructions for causing the MR head-mounted viewer 100 to present a tracked object via the one or more displays 115 based on determining that the image data meets an object tracking threshold. Alternatively or additionally, the one or more programs include instructions for causing the MR head-mounted viewer 100 to detect the execution of a gesture based on determining that the image data meets an object tracking threshold. The above examples are non-limiting; the acquired image data may be used for object detection, face recognition detection, gesture detection, etc. In some embodiments, the one or more programs include instructions for causing the MR head-mounted viewer 100 to perform operations or actions associated with the detected object or gesture in response to detecting the object and / or gesture. As those skilled in the art will recognize upon reading the description provided herein, the above example operations may be performed at the MR head-mounted viewer 100 and / or a device communicatively coupled to the MR head-mounted viewer 100 (e.g., the wrist-wearable device 426, the HIPD 442, the server 430, the computer 440, and / or any other device described below with reference to Figures 5A to 5C-2 ).

[0045] Figure 1B FIG. shows a perspective view of the MR head-mounted viewer 100 according to some embodiments. The perspective view of the MR head-mounted viewer 100 shows an additional imaging device 132a (e.g., a respective floodlight LED of the second set of floodlight LEDs 132) of the MR head-mounted viewer 100. The additional imaging device 132a may be an instance of the second imaging device 122b described above with reference to Figure 1A . Alternatively or additionally, the additional imaging device 132a is an instance of the first imaging device 122a or includes the first imaging device 122a. The additional imaging device 132a may be used in combination with the object tracking assembly 120 to provide full field-of-view coverage (e.g., increasing the field of view illuminated by one or more of the plurality of imaging devices 122). According to some embodiments, another additional imaging device 132b is disposed on an opposite side of the MR head-mounted viewer 100, as shown in Figure 1D .

[0046] Figure 1CShows a bottom view of the MR head-mounted viewer 100 according to some embodiments. The bottom view of the MR head-mounted viewer 100 shows an input device 145 disposed on a portion of the housing. In some embodiments, the input device 145 is a physical (pressable) button. The input device 145, in response to receiving user input (e.g., pressing the button), causes the MR head-mounted viewer 100 to initiate a passthrough mode. When activated, the passthrough mode causes the MR head-mounted viewer 100 to present image data of the real-world environment through the display 115. The image data of the real-world environment is captured by one or more imaging devices (e.g., imaging devices 122 and / or 132) of the MR head-mounted viewer 100. In some embodiments, the image data of the real-world environment replaces the extended reality environment presented by the display 115 (e.g., removes the user from an immersive AR environment so that the user can focus on the real-world environment).

[0047] Figure 1D Shows another perspective view of the MR head-mounted viewer 100 according to some embodiments. This another perspective view of the MR head-mounted viewer 100 shows another additional imaging device 132b of the MR head-mounted viewer 100. As described above, this another additional imaging device 132b is disposed on the opposite side of the MR head-mounted viewer 100. This another additional imaging device 132b can be used in conjunction with the object tracking component 120 to provide full field-of-view coverage. This another perspective view 150 of the MR head-mounted viewer 100 also shows the inner surface of the housing 110 and one or more displays 115 disposed on the inner surface of the housing 110.

[0048] Figures 2A to 2E Illustrates examples of imaging devices and lighting devices of an object tracking component (e.g., of the MR head-mounted viewer 100) according to some embodiments and data related to these examples. In some embodiments, one or more components of the object tracking component shown by Figures 2A to 2E can be positioned in a relative orientation different from the specific structure shown with respect to other components of the object tracking component.

[0049] Figure 2AShows a first embodiment of the object tracking component 205 and a second embodiment of the object tracking component 210. The first embodiment of the object tracking component 205 includes a first imaging device 122a, a second imaging device 122b, and a first exemplary lighting device 204 (similar to the lighting device 124). The second embodiment of the object tracking component 210 includes a first imaging device 122a, a second imaging device 122b, and a second exemplary lighting device 206 (similar to the lighting device 124). The first exemplary lighting device 204 has a flush design that includes a surface matching the outer surface of the housing 110 of the MR head-mounted viewer 100. The second exemplary lighting device 206 has a columnar design that protrudes from the outer surface of the housing 110 of the MR head-mounted viewer 100. The first exemplary lighting device 204 and the second exemplary lighting device 206 include plastic covers to ensure that the light beams (e.g., infrared light beams) emitted by the corresponding lighting devices 204 and 206 are dispersed on the plastic covers so that the light beams are not visible. The first exemplary lighting device 204 and the second exemplary lighting device 206 include a cut-filter plastic resin that reduces red-eye.

[0050] In some embodiments, the first embodiment of the object tracking component 205 and the second embodiment of the object tracking component 210 use a protruding imaging device 122 (or a protruding lighting device, such as the second exemplary lighting device 206) to extend the corresponding device above the outer surface of the housing 110 of the MR head-mounted viewer 100 to allow for a thinner form factor and / or a function for providing sufficient field-of-view illumination for performing interactions. The thinner form factor achieves cost savings by reducing the size of the corresponding covers for each device. In some embodiments, a flash sensor (not shown) is positioned behind the light guide (e.g., rather than behind the front). Refer to the following for Figures 2B to 2E Provide other design considerations.

[0051] Figure 2B Shows a cross-sectional view 215 of the first imaging device 122a according to some embodiments. The cross-sectional view 215 of the first imaging device 122a shows the glass cover 217 (e.g., cover window) of the first imaging device 122a and the first imaging device 122a coupled to the carrier 219. In some embodiments, the glass cover 217 is coupled to the carrier 219 by an adhesive (e.g., a pressure-sensitive adhesive). In some embodiments, the first imaging device 122a is coupled to a carrier bracket on which the carrier 219 can be separately mounted.

[0052] Figure 2C Shows different views of the second imaging device 122b according to some embodiments. The first view 216 shows a top view of the second imaging device 122b, and the second view 218 shows a rear view of the second imaging device 122b.Figure 2C Also shown are multiple parts of the second imaging device 122b. The first part 221 shows the lens assembly of the second imaging device 122b, the second part 222 shows the bonding wires of the second imaging device 122b, and the third part 223 shows the image sensor, (flexible) printed circuit board assembly, and connector of the second imaging device 122b. In some embodiments, the lens assembly of the second imaging device 122b can be used in place of the glass cover. The image sensor is configured to reduce power consumption.

[0053] Figure 2D Different embodiments of a lighting device are shown according to some embodiments. Each embodiment of the lighting device can have an LED field of view 228 and includes a corresponding infrared-transparent cover window, an LED component 227, an opaque front cover window 229, and a stereo bracket 231. The corresponding infrared-transparent cover window is mounted to the stereo bracket by fasteners such as screws. The LED component 227 is part of a surface-mount technology on a (flexible) printed circuit board assembly. The (flexible) printed circuit board assembly can also be coupled to the stereo bracket 231 by an adhesive (e.g., a pressure-sensitive adhesive).

[0054] As shown in the first embodiment of the lighting device 225, the first infrared-transparent cover window 226 is a flat post design. As shown in the second embodiment of the lighting device 230, the second infrared-transparent cover window 232 is a dome design, and as shown in the third embodiment of the lighting device 235, the third infrared-transparent cover window 233 is a dome post design. According to some embodiments, the cover window can be selected based on the desired field of view for illumination by the LED component 227.

[0055] Figure 2E Optical simulations performed on different embodiments of a lighting device are shown according to some embodiments. Chart 260 shows the field of view and throughput performance of lighting devices with different cover windows. Each of the cover window designs in the plurality has a viewing angle (POV) of at least 120° in the horizontal direction and at least 90° in the vertical direction even in the presence of LED placement errors. The lighting coverage angle is calculated using a threshold of 0.022 W / Sr (which corresponds to a lighting requirement of 0.088 mJ / m for a single LED). 2 of a single LED).

[0056] Figure 3A A side view 305 and a top view 310 of the coverage area of the lighting device 124 are shown. For hand tracking, the lighting performance is measured by a critical lighting coverage percentage that describes the amount of the important working volume that is sufficiently illuminated. The critical working volume is defined as an elevation angle of [0°, 75°] and an azimuth angle of [-60°, 60°], as Figure 3AAs shown. The position and specifications of the lighting device 124 (e.g., LED) are designed to optimize the lighting coverage. Therefore, a downward-tilted lighting device 124 is preferred to maximize the lighting working volume below eye level; an ultra-wide viewing angle of the lighting device 124 (e.g., 150 degrees full width at half maximum (FWHM)) is also preferred.

[0057] Figure 3B Shows the frontal illumination using one or more lighting devices according to some embodiments. The first frontal illumination graph 315 shows the illumination produced by a single lighting device 124 configuration implemented in the MR head-mounted viewer 100. The second frontal illumination graph 320 shows the illumination produced by a dual lighting device 124 configuration implemented in the MR head-mounted viewer 100. Although the first frontal illumination graph 315 and the second frontal illumination graph 320 are similar, the power required for the single lighting device 124 configuration is approximately twice the power used in the second frontal illumination graph 320. Although the single lighting device 124 configuration and the dual lighting device 124 configuration can operate similarly with sufficient power, the dual lighting device 124 configuration projects weaker shadows and has improved occlusion robustness (e.g., performs better when one lighting device 124 is occluded and the other lighting device 124 is not occluded). For full POV lighting, hand tracking is designed using three or four lighting devices 124, adding two additional lighting devices 124 to the sides of the MR head-mounted viewer 100 (e.g., the proximal tracking imaging device (e.g., the additional imaging device 132)).

[0058] Figure 3C and Figure 3D Illustrates frontal illumination graphs of different lighting device configurations. Each lighting device configuration in the plurality of lighting device configurations is implemented in a corresponding MR head-mounted viewer 100.

[0059] The third frontal illumination graph 325 shows a third lighting device configuration implemented in the MR head-mounted viewer 100. The third lighting device configuration includes two lighting devices 124. These lighting devices have the following configurations: a peak wavelength of 860 nm and a spectral bandwidth (BW) of 30 nm; the current required for each LED to produce 31.8 mW / Sr is 95.1 mA; the current used for each LED in the simulation is 100 mA; the current used for each LED in the simulation (+IR QE (5%) margin and +CW (17%) margin) is 122 mA; the total electrical power (two LEDs) in the simulation (60 Hz, 1 ms integration time) is 39.52 mW; and the lighting coverage of the critical working volume (at the recommended LED position) is 89.059%.

[0060] The fourth front illuminance diagram 330 shows the fourth lighting device configuration implemented in the MR head-mounted viewer 100. The fourth lighting device configuration includes two lighting devices 124. These lighting devices have the following configurations: a peak wavelength of 840 nm and a spectral bandwidth of 30 nm; the current required for each LED to produce 31.8 mW / Sr is 79.5 mA; the current used for each LED in the simulation is 80 mA; the current used for each LED in the simulation (+IR QE (5%) margin and +CW (17%) margin) is 97.6 mA; the total electrical power (for two LEDs) in the simulation (60 Hz, 1 ms integration time) is 27.98 mW; and the lighting coverage of the critical working volume (at the recommended LED positions) is 98.90%.

[0061] The fifth front illuminance diagram 335 shows the fifth lighting device configuration implemented in the MR head-mounted viewer 100. The fifth lighting device configuration includes two lighting devices 124. These lighting devices have the following configurations: a peak wavelength of 850 nm and a spectral bandwidth of 35 nm; the current required for each LED to produce 31.8 mW / Sr is 94.9 mA; the current used for each LED in the simulation is 100 mA; the current used for each LED in the simulation (+IR QE (5%) margin and +CW (17%) margin) is 122 mA; the total electrical power (for two LEDs) in the simulation (60 Hz, 1 ms integration time) is 39.82 mW; and the lighting coverage of the critical working volume (at the recommended LED positions) is 98.36%.

[0062] The sixth front illuminance diagram 340 shows the sixth lighting device configuration implemented in the MR head-mounted viewer 100. The sixth lighting device configuration includes two lighting devices 124. These lighting devices have the following configurations: a peak wavelength of 845 nm and a spectral bandwidth of 50 nm; the current required for each LED to produce 31.8 mW / Sr is 176.7 mA; the current used for each LED in the simulation is 195 mA; the current used for each LED in the simulation (+IR QE (5%) margin and +CW (17%) margin) is 237.9 mA; the total electrical power (for two LEDs) in the simulation (60 Hz, 1 ms integration time) is 46.22 mW; the lighting coverage of the critical working volume (at the recommended LED positions) is 89.88%.

[0063] The seventh front illumination diagram 345 shows the seventh lighting device configuration implemented in the MR head mounted viewer 100. The seventh lighting device configuration includes six lighting devices 124. These lighting devices have the following configuration: a peak wavelength of 850nm and a spectral bandwidth of 30nm; the current required for each LED to produce 31.8mW / Sr is 141.3mA (multiple LEDs will be required at lower drive); the current used in the simulation for each LED is 150mA; the current used in the simulation for each LED (+IR QE (5%) margin and +CW (17%) margin) is 183mA; the total electrical power (two LEDs) in the simulation (60Hz, 1ms integration time) is 111.98mW; and (at the recommended LED position) the illumination coverage of the critical working volume is 90.13%.

[0064] Figure 3E Example illumination levels required to cover specific use cases according to some embodiments are shown. For example, 80% coverage of social media applications (e.g., 80% of the total time the head wearable device is used using social media applications) requires a total illumination of approximately 16 lux. Alternatively, 90% coverage of shooting games requires a total illumination of approximately 12 lux.

[0065] Figure 4 Alternative configurations of head wearable devices according to some embodiments are shown. A first alternative head wearable device 410 includes one or more object tracking components having at least four frontal sensors. A second alternative head wearable device 420 includes one or more object tracking components having at least four frontal sensors and at least two additional imaging devices and / or lighting devices located on respective sides of the head wearable device. The first alternative head wearable device 410 allows for frontal field of view coverage, while the second alternative head wearable device 420 allows for full field of view coverage.

[0066] Example augmented reality system

[0067] Figure 5A , Figure 5B , Figure 5C-1 and Figure 5C-2 An example XR system including an AR system and an MR system is shown in accordance with some embodiments. Figure 5A A first XR system 500a is shown along with a first example user interaction using a wrist wearable device 526 , a head wearable device (eg, an AR device 528 ), and / or a handheld intermediary processing device (HIPD) 530 . Figure 5BShows a second AR system 500b and a second example user interaction that uses a wrist wearable device 526, an AR device 528, and / or a HIPD 542. Figure 5C-1 and Figure 5C-2 Shows a third MR system 500c and a third example user interaction that uses a wrist wearable device 526, a head wearable device (e.g., a mixed reality device such as a virtual reality (VR) device), and / or a HIPD 542. As those skilled in the art will recognize after reading the description provided herein, the above example AR and MR systems (described in detail below) can perform various functions and / or operations.

[0068] The wrist wearable device 526, the head wearable device, and / or the HIPD 542 can be communicatively coupled via a network 525 (e.g., cellular, near field, Wi-Fi, personal area network, wireless local area network (LAN), etc.). Additionally, the wrist wearable device 526, the head wearable device, and / or the HIPD 542 can also be communicatively coupled via the network 525 (e.g., cellular, near field, Wi-Fi, personal area network, wireless local area network, etc.) to one or more servers 530, a computer 540 (e.g., a laptop computer, a computer, etc.), a mobile device 550 (e.g., a smartphone, a tablet, etc.), and / or other electronic devices. Similarly, the smart textile-based apparel 538 can also be communicatively coupled via the network 525 to the wrist wearable device 526, one or more head wearable devices, the HIPD 542, one or more servers 530, the computer 540, the mobile device 550, and / or other electronic devices when in use to provide input.

[0069] Go to Figure 5A, shows user 502 wearing a wrist-worn device 526 and an AR device 528, and the user places the HIPD 542 on their table. The wrist-worn device 526, the AR device 528, and the HIPD 542 facilitate the user's interaction with the AR environment. In particular, as shown in the first XR system 500a, the wrist-worn device 526, the AR device 528, and / or the HIPD 542 enable the presentation of one or more avatars 504, digital representations 506 of contacts, and virtual objects 508. As discussed below, user 502 can interact with one or more avatars 504, digital representations 506 of contacts, and virtual objects 508 via the wrist-worn device 526, the AR device 528, and / or the HIPD 542. Additionally, user 502 is able to directly view physical objects in the environment, such as physical table 529, through one or more transparent lenses and one or more waveguides of the AR device 528. Alternatively, an MR device can be used in place of the AR device 528, and a similar user experience can occur, but the user will not directly view physical objects in the environment (e.g., table 529), but rather a virtual reconstruction of table 529 generated from one or more sensors of the MR device (e.g., an outward-facing camera capable of recording the surrounding environment).

[0070] User 502 may use any one of the wrist-wearable device 526, the AR device 528 (e.g., via physical input at the AR device and / or built-in motion tracking of the user's limb), smart textile apparel, an externally mounted limb tracking device, and the HIPD 542 to provide user input, etc. For example, user 502 may perform one or more gestures detected by the wrist-wearable device 526 (e.g., using one or more EMG sensors and / or IMUs built into the wrist-wearable device) and / or the AR device 528 (e.g., using one or more image sensors or cameras) to provide user input. Alternatively or additionally, user 502 may provide user input via one or more touch surfaces of the wrist-wearable device 526, the AR device 528, and / or the HIPD 542; and / or voice commands collected by a microphone of the wrist-wearable device 526, the AR device 528, and / or the HIPD 542. The wrist-wearable device 526, the AR device 528, and / or the HIPD 542 include an artificially intelligent (AI) digital assistant to assist the user in providing user input (e.g., completing an operation sequence, suggesting different operations or commands, providing reminders, confirming commands). For example, the digital assistant may be invoked by an input occurring at the AR device 528 (e.g., via an input at the temple of the AR device 528). In some embodiments, user 502 may provide user input via one or more facial gestures and / or facial expressions. For example, cameras of the wrist-wearable device 526, the AR device 528, and / or the HIPD 542 may track the eyes of user 502 for navigating the user interface.

[0071] The wrist-wearable device 526, the AR device 528, and / or the HIPD 542 can operate individually or in combination to allow the user 502 to interact with the AR environment. In some embodiments, the HIPD 542 is configured to operate as a central hub or control center for: the wrist-wearable device 526; the AR device 528; and / or another communicatively coupled device. For example, the user 502 can provide input at any one of the wrist-wearable device 526, the AR device 528, and / or the HIPD 542 to interact with the AR environment, and the HIPD 542 can identify one or more back-end tasks and front-end tasks to effectuate the requested interaction, and can distribute instructions to effectuate the execution of the one or more back-end tasks and front-end tasks at the wrist-wearable device 526, the AR device 528, and / or the HIPD 542. In some embodiments, back-end tasks are background processing tasks not perceptible to the user (e.g., rendering content, decompressing, compressing, etc.), and front-end tasks are user-perceptible user-facing tasks (e.g., presenting information to the user, providing feedback to the user, etc.). The HIPD 542 can execute the back-end tasks and provide operation data corresponding to the executed back-end tasks to the wrist-wearable device 526 and / or the AR device 528 such that the wrist-wearable device 526 and / or the AR device 528 can execute the front-end tasks. In this way, the HIPD 542 (which has more computing resources and greater thermal headroom than the wrist-wearable device 526 and / or the AR device 528) executes computationally intensive tasks and reduces the computer resource utilization and / or power usage of the wrist-wearable device 526 and / or the AR device 528.

[0072] In the example shown in the first XR system 500a, the HIPD 542 identifies one or more back-end tasks and front-end tasks associated with a user request to initiate an AR video call with one or more other users (represented by the avatar 504 and the digital representation 506 of the contact); and the HIPD 542 distributes instructions to effectuate the execution of the one or more back-end tasks and front-end tasks. In particular, the HIPD 542 executes back-end tasks for processing and / or rendering image data (and other data) associated with the AR video call and provides operation data associated with the executed back-end tasks to the AR device 528 such that the AR device 528 executes the front-end task of presenting the AR video call (e.g., presenting the avatar 504 and the digital representation 506 of the contact).

[0073] In some embodiments, the HIPD 542 is used as a focus or anchor point for information presentation. This allows the user 502 to know where the information is presented. For example, as shown in the first XR system 500a, the avatar 504 and the digital representation 506 of the contact are presented above the HIPD 542. In particular, the HIPD 542 and the AR device 528 operate in combination to determine the location for presenting the avatar 504 and the digital representation 506 of the contact. In some embodiments, information can be presented within a predetermined distance from the HIPD 542 (e.g., within five meters). For example, as shown in the first XR system 500a, the virtual object 508 is presented on a table at a certain distance from the HIPD 542. Similar to the above example, the HIPD 542 and the AR device 528 can operate in combination to determine the location for presenting the virtual object 508. Alternatively, in some embodiments, the presentation of information is not restricted by the HIPD 542. More specifically, the avatar 504, the digital representation 506 of the contact, and the virtual object 508 do not have to be presented within a predetermined distance from the HIPD 542. Although the AR device 528 is described as working with the HIPD, the MR head-mounted viewer can interact in the same way as the AR device 528.

[0074] Coordinate the user input provided at the wrist-worn device 526, the AR device 528, and / or the HIPD 542 so that the user can use any device to initiate, continue, and / or complete an operation. For example, the user 502 can provide user input to the AR device 528 to cause the AR device 528 to present the virtual object 508, and while the AR device 528 is presenting the virtual object 508, the user 502 can provide one or more gestures through the wrist-worn device 526 to interact with and / or manipulate the virtual object 508. Although the AR device 528 is described as working with the wrist-worn device 526, the MR head-mounted viewer can interact in the same way as the AR device 528.

[0075] Figure 5A The following interaction is shown: In this interaction, an artificial intelligence (AI) virtual assistant can assist the user 502 in making requests. The AI virtual assistant can be used to complete open-ended requests made by the user 502 through natural language input. For example, in Figure 5A the user 502 issues an auditory request 544 to summarize the conversation and then share the summarized conversation with others in the meeting. In addition, the AI virtual assistant is configured to use the sensors of the extended reality system (e.g., the camera, microphone of the extended reality head-mounted viewer, and various other sensors of any device in the system) to provide context cues to the user for initiating tasks.

[0076] Figure 5AAn example neural network 552 for training artificial intelligence is also shown. The uses of artificial intelligence are diverse and include many different aspects of the devices and systems described herein. AI capabilities cover a wide variety of applications and enhance the interaction between user 502 and user devices (e.g., AR device 528, MR device (also referred to as an MR head-mounted viewer) 532, HIPD 542, wrist-wearable device 526, etc.). The AI discussed herein can be derived using many different training models, including but not limited to artificial neural network (ANN), deep neural network (DNN), convolution neural network (CNN), recurrent neural network (RNN), large language model (LLM), short-term memory network, transformer model, decision tree, random forest, support vector machine, k-nearest neighbor, genetic algorithm, Markov model, Bayesian network, fuzzy logic system, and deep reinforcement learning. For devices and systems that employ multiple AIs herein, different models can be used depending on the task. For example, for a natural language AI virtual assistant, an LLM can be used, while for object detection in a physical environment, a DNN can be used.

[0077] In another example, an AI virtual assistant can include many different AI models and, based on a user's request, can use multiple AI models (concurrently, sequentially, or in combination). For example, an LLM-based AI can provide instructions for helping a user follow a recipe, and the instructions can be partially based on another AI (e.g., object and scene detection) derived from an ANN, DNN, RNN, etc. that can discern which part of the recipe the user is focused on.

[0078] As artificial intelligence training models evolve, the operations and experiences described herein may be performed using different models other than those listed above, and those skilled in the art will understand that the above list is non-limiting.

[0079] User 502 can interact with the artificial intelligence through natural language input collected by a voice sensor, text input, or any other input modality that accepts natural language and / or the corresponding voice sensor module. In another example, the user can provide input by tracking the eye gaze of user 502 via a gaze tracker module. Additionally, the AI can also receive inputs other than those provided by user 502. For example, the AI can also generate its response based on environmental inputs (such as temperature data, image data, video data, ambient light data, audio data, GPS location data, inertial measurements (i.e., user movement) data, pattern recognition data, magnetometer data, depth data, pressure data, force data, neuromuscular data, heart rate data, temperature data, sleep data, etc.) collected by various types of sensors and / or their corresponding sensor modules in response to a user request. The data of the sensors can be retrieved entirely from a single device (such as the AR device 528) or from multiple devices communicating with each other (such as a system including at least two of the following: the AR device 528, the MR device 532, the HIPD 542, the wrist wearable device 526, etc.). The AI can also access additional information (such as one or more servers 530, computers 540, mobile devices 550, and / or other electronic devices) via the network 525.

[0080] A non - restrictive list of AI enhancement functions includes, but is not limited to, image recognition, speech recognition (such as automatic speech recognition), text recognition (such as scene text recognition), pattern recognition, natural language processing and understanding, classification, regression, clustering, anomaly detection, sequence generation, content generation, and optimization. In some embodiments, the AI enhancement functions are executed entirely or partially on a cloud computing platform communicatively coupled to the user device (such as the AR device 528, the MR device 532, the HIPD 542, the wrist wearable device 526, etc.) via one or more networks. The cloud computing platform provides scalable computing resources, distributed computing, hosted AI services, interference acceleration, pre - trained models, application programming interfaces (APIs), and / or other resources to support the comprehensive computing required for the AI enhancement functions.

[0081] Example outputs resulting from using AI can include natural language responses, mathematical calculations, charts displaying information, audio, images, videos, text, meeting summaries, predictive actions based on environmental factors, classification, pattern recognition, suggestions, evaluations, or other actions. In some embodiments, the generated output is stored in the local memory of the user device (such as the AR device 528, the MR device 532, the HIPD 542, the wrist wearable device 526, etc.), the storage device of an external device (server, computer, mobile device, etc.), and / or the storage device of the cloud computing platform.

[0082] AI-based outputs can be presented across different modalities (e.g., audio-based, vision-based, haptic-based, and any combination thereof) and across different devices of the XR systems described herein. Some vision-based outputs can include displaying information on the XR augmentation of an XR head-mounted viewer, user interfaces displayed on a wrist-wearable device, a laptop device, a mobile device, etc. Haptic feedback can provide information to the user 502 on a device with or without a display (e.g., HIPD 542). Artificial intelligence can also use the inputs described above to determine the appropriate modality and one or more devices for presenting content to the user (e.g., an audio output can be presented to a user walking on a busy road instead of a vision output to avoid distracting the user 502).

[0083] Example Augmented Reality Interaction

[0084] Figure 5B Shows the user 502 wearing the wrist-wearable device 526 and the AR device 528 and holding the HIPD 542. In the second AR system 500b, the wrist-wearable device 526, the AR device 528, and / or the HIPD 542 are used to receive one or more messages and / or provide one or more messages to the contacts of the user 502. In particular, the wrist-wearable device 526, the AR device 528, and / or the HIPD 542 detect and coordinate one or more user inputs to initiate a messaging application and prepare a response to a message received via the messaging application.

[0085] In some embodiments, user 502 launches an application on wrist-wearable device 526, AR device 528, and / or HIPD 542 via a user input, which causes the application to launch on at least one device. For example, in the second AR system 500b, user 502 performs a gesture associated with a command to launch a messaging application (represented by messaging user interface 512); wrist-wearable device 526 detects the gesture and, based on determining that user 502 is wearing AR device 528, causes AR device 528 to present messaging user interface 512 of the messaging application. AR device 528 can present messaging user interface 512 to user 502 via its display (e.g., as shown in user 502's field of view 510). In some embodiments, the application is launched and can run on the device (e.g., wrist-wearable device 526, AR device 528, and / or HIPD 542) that detected the user input to launch the application, and that device provides operation data to another device to cause the presentation of the messaging application. For example, wrist-wearable device 526 can detect a user input to launch the messaging application, launch and run the messaging application, and provide operation data to AR device 528 and / or HIPD 542 to cause the presentation of the messaging application. Alternatively, the application can be launched and run on a device other than the device that detected the user input. For example, wrist-wearable device 526 can detect a gesture associated with launching the messaging application and can cause HIPD 542 to run the messaging application and coordinate the presentation of the messaging application.

[0086] Additionally, user 502 can provide a user input provided at wrist-wearable device 526, AR device 528, and / or HIPD 542 to continue and / or complete an operation initiated at another device. For example, after launching the messaging application via wrist-wearable device 526 and while AR device 528 is presenting messaging user interface 512, user 502 can provide an input at HIPD 542 to prepare a response (e.g., as indicated by a swipe gesture performed on HIPD 542). The gesture performed by user 502 on HIPD 542 can be provided and / or displayed on another device. For example, the swipe gesture performed by user 502 on HIPD 542 is displayed on the virtual keyboard of messaging user interface 512 displayed by AR device 528.

[0087] In some embodiments, the wrist-wearable device 526, the AR device 528, the HIPD 542, and / or other communicatively coupled devices may present one or more notifications to the user 502. The notification may be an indication of a new message, an incoming call, an app update, a status update, etc. The user 502 may select the notification via the wrist-wearable device 526, the AR device 528, or the HIPD 542 and cause an app or operation associated with the notification to be presented on at least one device. For example, the user 502 may receive a notification that a message has been received at the wrist-wearable device 526, the AR device 528, the HIPD 542, and / or other communicatively coupled devices, and the user 502 may provide user input at the wrist-wearable device 526, the AR device 528, and / or the HIPD 542 to view the notification, and the device that detects the user input may cause an app associated with the notification to be launched and / or presented at the wrist-wearable device 526, the AR device 528, and / or the HIPD 542.

[0088] Although the example above describes coordinated input for interacting with a messaging app, those skilled in the art will recognize after reading this description that user input may be coordinated to interact with any number of apps, including but not limited to gaming apps, social media apps, camera apps, web-based apps, financial apps, etc. For example, the AR device 528 may present gaming app data to the user 502, and the HIPD 542 may use a controller to provide input to the game. Similarly, the user 502 may use the wrist-wearable device 526 to activate the camera of the AR device 528, and the user may use the wrist-wearable device 526, the AR device 528, and / or the HIPD 542 to manipulate image capture (e.g., zoom in or zoom out, apply filters, etc.) and capture image data.

[0089] Although the AR device 528 is shown as being capable of implementing certain functions, it should be understood that the AR device can be an AR device with different functions based on cost and market demand. For example, the AR device can include a single output modality, such as an audio output modality. In another example, the AR device can include a low-fidelity display as one of the multiple output modalities, where simple information (e.g., text and / or low-fidelity images / videos) can be presented to the user. In yet another example, the AR device can be configured with one or more forward-facing LEDs configured to provide information to the user. For example, the LED around the right lens can be lit when providing a direction to notify the wearer to turn right, or the LED on the left can be lit when providing a direction to notify the wearer to turn left. In another embodiment, the AR device can include an outward-facing projector such that information (e.g., text information, media, etc.) can be displayed on the user's palm or other suitable surface (e.g., a table, a whiteboard, etc.). In yet another embodiment, information can also be provided by locally dimming multiple parts of the lens to highlight the part of the environment to which the user's attention should be directed. These examples are not exhaustive, and the features of one AR device described above can be combined with the features of another AR device described above. Although the features and experiences of the AR device have been described in the previous section, it should be understood that the described functions and experiences can be applied to the MR head-mounted viewer in a similar manner, which will be described in the following section.

[0090] Example Mixed Reality Interactions

[0091] Go to Figure 5C-1 and Figure 5C-2 , shows a user 502 wearing a wrist-wearable device 526 and an MR device 532 (e.g., a device capable of providing a fully virtual reality (VR) experience or a mixed reality experience that displays one or more objects from the physical environment at the display of the device) and holding a HIPD 542. In the third MR system 500c, the wrist-wearable device 526, the MR device 532, and / or the HIPD 542 are used to interact within an MR environment (e.g., a VR game or other MR / AR application). Although the MR device 532 presents a representation of a VR game (e.g., the first MR game environment 520) to the user 502, the wrist-wearable device 526, the MR device 532, and / or the HIPD 542 detect and coordinate one or more user inputs to allow the user 502 to interact with the VR game.

[0092] In some embodiments, the user 502 can provide user inputs that cause actions in the corresponding MR environment through the wrist-wearable device 526, the MR device 532, and / or the HIPD542. For example, the third MR system 500c (as Figure 5C-1User 502 in (as shown) raises HIPD 542 to prepare for a swing in the first MR gaming environment 520. The MR device 532, in response to user 502 raising HIPD 542, causes the user's MR representation 422 to perform a similar action (e.g., raise a virtual object, such as virtual sword 524). In some embodiments, each device uses corresponding sensor data and / or image data to detect user input and provide an accurate representation of the movement of user 502. For example, the imaging sensor of HIPD 542 (e.g., a SLAM camera or other camera) can be used to detect the position of HIPD 542 relative to user 502's body such that the virtual object can be appropriately positioned within the first MR gaming environment 520; the sensor data from the wrist wearable device 526 can be used to detect the speed at which user 502 raises HIPD 542 such that the user's MR representation 422 and virtual sword 524 are synchronized with the movement of user 502; and the image sensor of the MR device 532 can be used to represent user 502's body, boundary conditions, or real-world objects within the first MR gaming environment 520.

[0093] In Figure 5C-2 User 502 performs a downward swing while holding HIPD 542. The wrist wearable device 526, MR device 532, and / or HIPD 542 detect the downward swing of user 502, and the corresponding action is performed in the first MR gaming environment 520. In some embodiments, the data collected by each device is used to enhance the user's experience within the MR environment. For example, the sensor data of the wrist wearable device 526 can be used to determine the speed and / or force of the downward swing, and the image sensors of HIPD 542 and / or the MR device 532 can be used to determine the position of the swing and how it should be represented within the first MR gaming environment 520, which can in turn be used as an input to the MR environment (e.g., a game mechanism that can be used to detect aspects of the speed, force, position, and / or actions of user 502 to classify user input (e.g., user performs a light strike, heavy strike, fatal strike, glancing strike, miss) or calculate an output (e.g., amount of damage)).

[0094] Figure 5C-2It is also shown that a portion of the physical environment is reconstructed and displayed on the display of the MR device 532 when the MR game environment 520 is being displayed. In this case, when one or more objects in the physical environment are potentially in the user's path (e.g., there may be a collision with the user and the objects in the physical environment), the reconstructed physical environment 546 is displayed in place of a portion of the MR game environment 520. Thus, this example MR game environment 420 includes: (i) an immersive virtual reality portion 548 (e.g., an environment that does not have a counterpart in the nearby physical environment) and (ii) the reconstructed physical environment 546 (e.g., the table 550 and the cup 552). Although the example shown here is an MR environment that demonstrates the reconstruction of the physical environment to avoid collisions, other uses of the reconstruction of the physical environment can be used, such as defining the characteristics of the virtual environment based on the surrounding physical environment (e.g., virtual pillars can be placed based on objects (e.g., trees) in the surrounding physical environment).

[0095] Although the wrist-worn device 526, the MR device 532, and / or the HIPD 542 are described as detecting user input, in some embodiments, user input is detected at a single device (where that single device is responsible for distributing the signal to other devices for performing the user input). For example, the HIPD 542 can run an application for generating the first MR game environment 520, provide corresponding data to the MR device 532 for rendering the first MR game environment 520, and detect the movement of the user 502 (when holding the HIPD 542) to cause corresponding actions to be performed within the first MR game environment 520. Additionally or alternatively, in some embodiments, the operation data (e.g., sensor data, image data, application data, device data, and / or other data) of one or more devices is provided to a single device (e.g., the HIPD 542) to process the operation data and cause the corresponding device to perform actions associated with the processed operation data.

[0096] In some embodiments, the user 502 can wear the wrist-worn device 526, wear the MR device 532, wear smart textile-based clothing 538 (e.g., a wearable haptic glove), and / or hold the HIPD 542 device. In this embodiment, the wrist-worn device 526, the MR device 532, and / or the smart textile-based clothing 538 are used in the MR environment (e.g., as referred to above Figure 5A and Figure 5Binteract within any of the described AR or MR systems. Although the MR device 532 presents a representation of an MR game (e.g., the second MR game environment 430) to the user 502, the wrist-wearable device 526, the MR device 532, and / or the smart textile-based garment 538 detect and coordinate one or more user inputs to allow the user 502 to interact with the MR environment.

[0097] In some embodiments, the user 502 may provide user inputs that cause actions in the corresponding MR environment via the wrist-wearable device 542, the MR device 532, and / or the smart textile-based garment 438. In some embodiments, each device uses its respective sensor data and / or image data to detect user inputs and provide an accurate representation of the movement of the user 502. Although four input devices are shown (e.g., the wrist-wearable device 526, the MR device 532, the HIPD 542, and the smart textile-based garment 538), each of these input devices may provide inputs for fully interacting with the MR environment completely independently. For example, the wrist-wearable device alone may provide sufficient inputs for interacting with the MR environment. In some embodiments, if multiple input devices are used (e.g., the wrist-wearable device and the smart textile-based garment 538), sensor fusion may be utilized to ensure that the inputs are correct. Although multiple input devices are described, it should be understood that other input devices may be used in combination or alone, such other devices including but not limited to external motion tracking cameras, other wearable devices adapted to different parts of the user, devices that allow the user to experience walking in the MR while remaining substantially stationary in the physical environment, and the like.

[0098] As described above, the data collected by each device is used to enhance the user's experience within the MR environment. Although not shown, the smart textile-based garment 538 may be used in combination with the MR device and / or the HIPD 542.

[0099] Figure 6 An example method 600 is shown for using an object tracking component to determine whether an object tracking criterion is met such that a tracked object is presented by an MR head-wearable device, according to some embodiments.

[0100] (A1) The method 600 includes: performing (602) an MR operation at the MR head-wearable device, which includes a housing. For example, Figure 5C-1 and Figure 5C-2 displays the MR operation.

[0101] The housing includes one or more displays that are located within an inner surface of the housing and are configured to present an extended reality environment (604) when a user wears the MR head wearable device.

[0102] The housing includes an object tracking component (606) disposed on an outer surface of the housing. For example, Figure 2A the object tracking component 205 shown in is disposed on the outer surface of the housing.

[0103] The object tracking component includes a plurality of imaging devices (608) aligned along a first axis. For example, when a user wears the MR head-mounted viewer, the plurality of imaging devices 122 can be aligned relative to the housing along the y-axis (e.g., the vertical axis).

[0104] The object tracking component includes an illumination device aligned along a second axis perpendicular to the first axis, and the illumination device is disposed at a predetermined intermediate distance between at least two corresponding imaging devices among the plurality of imaging devices (610). For example, the illumination device 124 is located between the imaging devices 122a and 122b with respect to the virtual y-axis.

[0105] The method 600 includes: determining (612) that the imaging data satisfies an object tracking threshold based on the imaging data received by the plurality of imaging devices when the illumination device generates ambient lighting conditions. In some embodiments, the illumination device generates ambient lighting conditions based on the current lighting conditions of the physical environment around the user (e.g., when the external conditions have weak light, the illumination device generates a larger amount of light).

[0106] The method 600 includes: causing (614) a tracked object to be presented via one or more displays according to determining that the imaging data satisfies the object tracking threshold.

[0107] (A2) In some embodiments of A1, at least two corresponding imaging devices and the illumination device of the object tracking component are arranged in a triangular configuration. For example, the first imaging device 122a, the second imaging device 122b, and the illumination device 124 form a triangular configuration, as Figure 1A shown.

[0108] (A3) In some embodiments of A1 or A2, the plurality of imaging devices includes: (i) at least one visible light color imaging sensor, and (ii) at least one SLAM-enabled camera. For example, Figures 1A to 1D the imaging device 122b in can be a camera specifically configured to implement the SLAM process.

[0109] (A4)In some embodiments of A3, each of at least one visible light color imaging sensor, at least one SLAM camera, and the lighting device is covered by a respective cover window, and these respective cover windows are made of different materials (e.g., different combinations of glass and / or polymers). For example, Figure 2B FIG. 215 shows a cross-sectional view of a first imaging device 122a, which includes a glass cover 217 (e.g., a glass cover window) above the first imaging device 122a. And the lighting device 230 can be covered by a different cover window (e.g., an opaque front cover window 229).

[0110] (A5)In some embodiments of A1 to A4, the plurality of imaging devices include a third imaging sensor different from at least two respective imaging devices, and the third imaging sensor is configured to increase the user's field of view. For example, Figure 3B An additional lighting device 124 mounted on the side of the MR head-mounted viewer is shown, where the additional lighting device 124 provides additional imaging data for the field of view from the user's side.

[0111] (A6)In some embodiments of A5, the third imaging sensor is located on a side-facing portion of the outer surface of the housing. For example, an additional imaging device 132b is provided on Figure 1D the side of the housing in

[0112] (A7)In some embodiments of any one of A1 to A6, the lighting device and the respective imaging devices among the plurality of imaging devices are tilted downward. For example, the second imaging device 122b and / or the lighting device 124 can be tilted downward (e.g., focused toward the user's hand).

[0113] (A8)In some embodiments of any one of A1 to A7, the lighting device is configured to extend beyond the outer surface of the housing. For example, Figures 2A to 2D It is shown that the imaging device and the associated glass cover window can extend beyond the edge of the surface of the outer surface of the housing 110.

[0114] (A9)In some embodiments of any one of A1 to A8, when the user wears the MR head-mounted device, the housing blocks the user's field of view. For example, in some embodiments, the housing includes an opaque component that separates the lens from the outer surface, and the camera on the outer surface of the head-mounted viewer can be configured to provide substantially all of the user's field of view beyond the opaque version of the housing (e.g., allowing perspective).

[0115] (B1)In some embodiments, an MR head-mounted viewer is provided, and the MR head-mounted viewer includes a housing for performing the method of any one of A1 to A9.

[0116] Although some experiences are described as occurring on an AR device and other experiences are described as occurring on an MR device, those skilled in the art will recognize that these experiences can be ported from an MR device to an AR device and vice versa.

[0117] For ease of reference, some definitions of the following devices and components are defined herein: These devices and components may be included in some or all of the example devices discussed. Those skilled in the art will recognize that certain types of components described may be more suitable for a particular set of devices and less suitable for a different set of devices. However, subsequent reference to the components defined herein should be considered to be covered by the definitions provided.

[0118] In some embodiments, example devices and systems including electronic devices and systems will be discussed. Such example devices and systems are not intended to be limiting, and those skilled in the art will understand that alternative devices and systems to the example devices and systems described herein may be used to perform the operations described herein and construct the systems and devices described herein.

[0119] As described herein, an electronic device is a device that uses electrical energy to perform a specific function. The electronic device may be any physical object that includes electronic components (such as transistors, resistors, capacitors, diodes, and integrated circuits). Examples of electronic devices include smart phones, laptop computers, digital cameras, televisions, game consoles, and music players, as well as the example electronic devices discussed herein. As described herein, an intermediate electronic device is a device that is located between two other electronic devices and / or is a subset of components of one or more electronic devices, and the intermediate electronic device facilitates communication and / or data processing and / or data transfer between the corresponding electronic devices and / or between electronic components.

[0120] The foregoing description provided above Figures 5A to 5C-2 is intended to supplement the description provided with reference to Figures 1A to 3E Although the terms in the following description may not be the same as those used in the foregoing description, those of ordinary skill in the art will understand that these terms have the same meaning.

[0121] Any data collection performed by the devices described herein and / or any device (hereinafter referred to as a "device") configured to perform or cause to be performed the different embodiments described with reference to any of the plurality of figures above is carried out with the consent of the user and in a manner that complies with all applicable privacy laws. Options are provided to the user for allowing the device to collect data and for restricting or denying the device to collect data. The user is able to select at any time to enable or to select to disable any data collection. In addition, options are provided to the user for requesting deletion of any collected data.

[0122] It will be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0123] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the claims. As used in the description of the embodiments and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are also intended to include the plural forms. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all combinations of one or more of the associated listed items. It will also be understood that "comprising" and / or "including" when used in this specification specify the presence of the stated features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0124] As used herein, depending on the context, "if" can be interpreted to mean: "when the stated condition precedent is true"; or "once" the stated condition precedent is true; or "in response to determining" the stated condition precedent is true; or "in accordance with determining" the stated condition precedent is true; or "in response to detecting" the stated condition precedent is true. Similarly, depending on the context, the phrase "if it is determined [that the stated condition precedent is true]" or "if [the stated condition precedent is true]" or "when [the stated condition precedent is true]" can be interpreted to mean "once determined" or "in response to determining" or "in accordance with determining", "once detected" or "in response to detecting" the stated condition precedent is true.

[0125] For purposes of explanation, the foregoing description has been described with reference to particular embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the claims to the precise forms disclosed. Given the above teachings, many modifications and variations are possible. The embodiments were chosen and described in order to best explain the principles of operation and the practical application, so as to enable others skilled in the art to implement.

Claims

1. A housing of a mixed reality (MR) head - wearable device, comprising: One or more displays, which are located within an inner surface of the housing and are configured to present an extended reality environment when a user wears the MR head - wearable device; And An object - tracking component, which is disposed on an outer surface of the housing and includes: A plurality of imaging devices that are aligned along a first axis, and A lighting device that is aligned along a second axis perpendicular to the first axis, and the lighting device is disposed at a predetermined intermediate distance between at least two corresponding imaging devices among the plurality of imaging devices; Wherein, when the MR head - wearable device is performing an operation, the object - tracking component is configured to determine that the acquired imaging data meets an object - tracking threshold based on the imaging data acquired by the plurality of imaging devices when the lighting device generates ambient lighting conditions, so as to present a tracked object via the one or more displays.

2. The housing according to claim 1, wherein, The at least two corresponding imaging devices and the lighting device of the object - tracking component are arranged in a triangular configuration.

3. The housing according to claim 1, wherein, The plurality of imaging devices includes: at least one visible - light color imaging sensor; and at least one simultaneous localization and mapping (SLAM) camera.

4. The housing according to claim 3, wherein, Each of the at least one visible - light color imaging sensor, the at least one SLAM camera, and the lighting device is covered by a corresponding cover window made of different materials.

5. The housing according to claim 1, wherein: The plurality of imaging devices includes a third imaging sensor that is different from the at least two corresponding imaging devices, and The third imaging sensor is configured to increase the field of view of the user.

6. The housing according to claim 5, wherein, The third imaging sensor is located at a side - facing portion of the outer surface of the housing.

7. The housing according to claim 1, wherein, The lighting device and the corresponding imaging device among the plurality of imaging devices are inclined downward.

8. The housing according to claim 1, wherein, The lighting device is configured to extend beyond the outer surface of the housing.

9. The housing according to claim 1, wherein, When a user wears the MR head - wearable device, the housing blocks the field of view of the user.

10. A mixed reality (MR) head - wearable device, comprising: A housing; One or more displays, which are located within an inner surface of the housing and are configured to present an extended reality environment when a user wears the MR head - wearable device; And An object - tracking component, which is disposed on an outer surface of the housing and includes: A plurality of imaging devices that are aligned along a first axis, and A lighting device that is aligned along a second axis perpendicular to the first axis, and the lighting device is disposed at a predetermined intermediate distance between at least two corresponding imaging devices among the plurality of imaging devices; Wherein, when the MR head wearable device is performing an operation, the object tracking component is configured to: determine that the acquired imaging data meets an object tracking threshold based on the imaging data acquired by the plurality of imaging devices when the lighting device generates ambient lighting conditions, so as to present a tracked object via the one or more displays.

11. The MR head wearable device according to claim 10, wherein, The at least two corresponding imaging devices and the lighting device of the object tracking component are arranged in a triangular configuration.

12. The MR head wearable device according to claim 10, wherein, The plurality of imaging devices includes: at least one visible light color imaging sensor; and at least one simultaneous localization and mapping (SLAM) camera.

13. The MR head-wearable device according to claim 12, wherein, Each of the at least one visible light color imaging sensor, the at least one SLAM camera, and the lighting device is covered by a corresponding cover window, and the corresponding cover windows are made of different materials.

14. The MR head wearable device according to claim 10, wherein: The plurality of imaging devices includes a third imaging sensor, the third imaging sensor is different from the at least two corresponding imaging devices, and The third imaging sensor is configured to increase the field of view of the user.

15. The MR head wearable device according to claim 14, wherein, The third imaging sensor is located on a side-facing portion of the outer surface of the housing.

16. The MR head-wearable device according to claim 10, wherein, The lighting device and the corresponding imaging devices among the plurality of imaging devices are tilted downward.

17. The MR head wearable device according to claim 10, wherein, The lighting device is configured to extend beyond the outer surface of the housing.

18. The MR head wearable device according to claim 10, wherein, When the user wears the MR head wearable device, the housing blocks the field of view of the user.

19. A method includes: Performing an MR operation on an MR head wearable device, the MR head wearable device including a housing, the housing including: One or more displays, the one or more displays are located inside the inner surface of the housing, and the one or more displays are configured to: when the user wears the MR head wearable device, present an extended reality environment; and An object tracking component, the object tracking component is disposed on the outer surface of the housing, and the object tracking component includes: A plurality of imaging devices, the plurality of imaging devices are aligned along a first axis, and A lighting device, the lighting device is aligned along a second axis, the second axis is perpendicular to the first axis, and the lighting device is disposed at a predetermined intermediate distance between at least two corresponding imaging devices among the plurality of imaging devices; Determining that the acquired imaging data meets an object tracking threshold based on the imaging data acquired by the plurality of imaging devices when the lighting device generates ambient lighting conditions; and According to the determination that the acquired imaging data meets the object tracking threshold, causing a tracked object to be presented via the one or more displays.

20. The method according to claim 19, wherein The at least two corresponding imaging devices and the lighting device of the object tracking component are arranged in a triangular configuration.

Images