Content Sharing in Extended Reality

Abstract

A system, method, and non-transitory computer-readable medium configured to enable content sharing between users of wearable extended reality devices are provided. In one implementation, the computer-readable medium may be configured to include instructions for causing at least one processor to establish a link between a first wearable extended reality device and a second wearable extended reality device. The first wearable extended reality device may display a first virtual content. The second wearable extended reality device may obtain a command to display the first virtual content via the second wearable extended reality device, and in response, the content may be transmitted and displayed via the second extended reality device. Additionally, the first wearable extended reality device may receive a second virtual content from the second wearable extended reality device and display the second virtual content via the first wearable extended reality device.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Provisional Patent Application No. 63 / 147,051 filed February 8, 2021, U.S. Provisional Patent Application No. 63 / 157,768 filed March 7, 2021, U.S. Provisional Patent Application No. 63 / 173,095 filed April 9, 2021, U.S. Provisional Patent Application No. 63 / 213,019 filed June 21, 2021, U.S. Provisional Patent Application No. 63 / 215,500 filed June 27, 2021, U.S. Provisional Patent Application No. 63 / 216,335 filed June 29, 2021, and U.S. Provisional Patent Application No. 63 / 216,335 filed June 29, 2021. This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 226,977, filed July 29, U.S. Provisional Patent Application No. 63 / 300,005, filed January 16, 2022, U.S. Provisional Patent Application No. 63 / 307,207, filed February 7, 2022, U.S. Provisional Patent Application No. 63 / 307,203, filed February 7, 2022, and U.S. Provisional Patent Application No. 63 / 307,217, filed February 7, 2022, all of which are incorporated by reference herein in their entireties.

[0002] Technical Field The present disclosure relates generally to the field of extended reality, and more particularly to systems, methods, and devices for providing productivity applications using extended reality environments. [Background technology]

[0003] For many years, PC users have faced a productivity dilemma: whether to limit their mobility (when choosing a desktop computer) or their screen size (when choosing a laptop computer). One partial solution to this dilemma is the use of a docking station. A docking station is an interface device that connects a laptop computer with other devices. By plugging a laptop computer into a docking station, laptop users can enjoy the improved visibility provided by a larger monitor. However, because the large monitor is fixed, the user's mobility, while improved, is still limited. For example, even laptop users with docking stations do not have the freedom to use two 32-inch screens wherever they wish.

[0004] Some of the disclosed embodiments are directed to providing a new approach to solving productivity dilemmas, which uses extended reality (XR) to provide a mobile environment that allows users to experience the comfort of a stationary workspace wherever they want by providing a virtual desktop-like screen. Summary of the Invention [Problem to be solved by the invention]

[0005] SUMMARY OF THE INVENTION Embodiments consistent with the present disclosure provide systems, methods, and devices for providing and supporting productivity applications using extended reality environments. [Means for solving the problem]

[0006] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for enabling content sharing between users of wearable extended reality devices. Some of these embodiments may include establishing a link between a first wearable extended reality device and a second wearable extended reality device, presenting first virtual content through the first wearable extended reality device, obtaining a first command to display the first virtual content via the second wearable extended reality device, causing the first virtual content to be transmitted for display to the second wearable extended reality device in response to the first command, receiving second virtual content from the second wearable extended reality device for display via the first wearable extended reality device, and presenting the second virtual content received from the second wearable extended reality device via the first wearable extended reality device.

[0007] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for providing situational awareness to a user of a wearable extended reality device, some of which may include displaying virtual content through a first wearable extended reality device, detecting a second wearable extended reality device in proximity to the first wearable extended reality device, establishing a link between the first wearable extended reality device and the second wearable extended reality device, and transmitting data representing at least a portion of the virtual content in an obscured form to the second wearable extended reality device, the obscured form providing an indication of a location of the at least a portion of the virtual content in three-dimensional space without revealing the entity of the virtual content in the obscured form.

[0008] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for linking a virtual whiteboard to a physical space, including receiving, via a wireless network, an indication of a location of a first wearable extended reality device; performing a lookup in a repository of virtual whiteboards and locations of the virtual whiteboards to determine that the location of the first wearable extended reality device corresponds to the location of a particular virtual whiteboard; transmitting data corresponding to content of the particular virtual whiteboard to the first wearable extended reality device, thereby enabling a first user of the first wearable extended reality device to virtually view the content of the particular virtual whiteboard and add virtual content to the particular virtual whiteboard; and, during a first period of time, , receiving virtual content added by the first user; receiving an indication via the wireless network that a second wearable extended reality device is at the location of the particular virtual whiteboard at a second time period after the first wearable extended reality device is no longer at the location of the particular virtual whiteboard; and transmitting data corresponding to the content of the particular virtual whiteboard and the added content to the second wearable extended reality device, thereby enabling a second user of the second wearable extended reality device to view the content and the added content while the first user was not at the location of the particular virtual whiteboard.

[0009] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for transferring virtual content to a physical display device. Some of these embodiments may include presenting an extended reality environment in a room via a wearable extended reality device, the wearable extended reality device configured to be paired with multiple display devices located in the room, each display device associated with a unique network identifier; receiving input associated with the wearable extended reality device to cause a target display device to present a particular virtual object in the extended reality environment; receiving image data from an image sensor associated with the wearable extended reality device, the image data being indicative of the target display device; analyzing the image data to identify the target display device; upon identifying the target display device, determining a network identifier for the target display device; establishing a communications link with the target display device using the determined network identifier for the target display device; and transmitting data representing the particular virtual object to the target display device, the transmitted data enabling the target display device to present the particular virtual object.

[0010] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for simulating user interactions with shared content. Some of these embodiments may include establishing a communication channel for sharing content and user interactions between a first wearable extended reality device and at least one second wearable extended reality device, transmitting first data representing an object associated with the first wearable extended reality device to the at least one second wearable extended reality device, the first data enabling a virtual representation of the object to be displayed through the at least one second wearable extended reality device, receiving image data from an image sensor associated with the first wearable extended reality device, detecting at least one user interaction associated with the object in the image data, the at least one user interaction including a human hand pointing to a particular portion of the object, and transmitting second data indicating an area of ​​the particular portion of the object to the at least one second wearable extended reality device based on the detection of the at least one user interaction in the image data.

[0011] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for managing content placement in an extended reality environment. Some of these embodiments may include receiving a request from an entity to place virtual content at a specific geographic location within at least one shared extended reality environment including a plurality of virtual objects, wherein the plurality of virtual objects are viewable by a plurality of wearable extended reality devices registered to the at least one shared extended reality environment; obtaining information associated with the request; accessing a plurality of content placement rules that define geographic restrictions on extended reality environment content placement; determining, based on the obtained information, that the request from the entity to place the virtual content at the specific geographic location corresponds to the specific content placement rule; implementing the specific content placement rule to prevent display of the virtual content at the specific geographic location within the at least one shared extended reality environment by at least some of the plurality of wearable extended reality devices when a condition of the specific content placement rule is not met; and implementing the specific content placement rule to enable display of the virtual content at the specific geographic location within the at least one shared extended reality environment by at least some of the plurality of wearable extended reality devices when a condition of the specific content placement rule is met.

[0012] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for presenting virtual content to multiple viewers. Some of these embodiments include receiving sensor data indicative of multiple wearable extended reality devices located in a room, receiving a command to share a virtual object with the multiple wearable extended reality devices, analyzing the sensor data to determine a first location within the room for a first wearable extended reality device, a second location within the room for a second wearable extended reality device, and a third location within the room for a third wearable extended reality device, determining a location within the room for displaying the virtual object based on the determined first location, the determined second location, and the determined third location, and displaying the virtual object with the first wearable extended reality device. The method may include generating a first representation of the virtual object at a location determined through a wearable extended reality device, the first representation rendered from a first perspective; generating a second representation of the virtual object at a location determined through a second wearable extended reality device, the second representation rendered from a second perspective different from the first perspective; and generating a third representation of the virtual object at a location determined through a third wearable extended reality device, the third representation rendered from a third perspective different from the first perspective and the second perspective.

[0013] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for creating virtual colored markings on an object, some of which may include receiving an indication of the object, receiving from an image sensor an image of an individual's hand holding a physical marking implement, detecting a color associated with the marking implement in the image, receiving from the image sensor image data indicative of movement and location of the tip of the marking implement, determining from the image data when the location of the tip corresponds to a location on the object, and generating a virtual marking on the object at the corresponding location in the detected color.

[0014] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media for enabling wearable extended reality devices to share virtual content. Some of these embodiments may include generating a visual code that reflects a first physical location of a mobile device, where the visual code is configured to be read by multiple wearable extended reality devices; presenting the visual code on a display of the mobile device for detection by the multiple wearable extended reality devices, thereby enabling the multiple wearable extended reality devices to share content in a common coordinate system upon detection of the visual code; detecting movement of the mobile device to a second physical location different from the first physical location; and altering the presentation of the visual code such that the visual code is unavailable for use in content sharing upon detection of the mobile device movement.

[0015] Consistent with other disclosed embodiments, a non-transitory computer-readable storage medium can store program instructions that, when executed by at least one processing device, perform any of the methods described herein.

[0016] The foregoing summary and the following detailed description are exemplary and explanatory only and are not limiting of the scope of the claims.

[0017] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 1 is a schematic diagram of a user using an exemplary extended reality system consistent with this disclosure. [Figure 2] FIG. 2 is a schematic diagram of the major components of the exemplary extended reality system of FIG. 1 consistent with this disclosure. [Figure 3] 3 is a block diagram illustrating some of the components of the input unit consistent with this disclosure. [Figure 4] FIG. 4 is a block diagram illustrating some of the components of an extended reality unit consistent with this disclosure. [Figure 5] FIG. 5 is a block diagram illustrating some of the components of a remote processing unit consistent with this disclosure. [Figure 6] FIG. 6 is an example illustration of a shared movement between a first user and a second user of a wearable extended reality device, consistent with some disclosed embodiments. [Figure 7] 7A and 7B illustrate exemplary embodiments of a shared action via a share bubble between a first user and a second user of a wearable extended reality device. [Figure 8]8A shows an example embodiment of a sharing action in which a first extended reality device user may lean towards a second user to share content, and FIG. 8B shows an example embodiment of a sharing action in which a first extended reality device user may lean towards a second user to share content. [Figure 9] FIG. 9 illustrates an exemplary embodiment of a shared action via a shared ring. [Figure 10] Figure 10A illustrates an example embodiment of sharing content to be presented later via a first wearable extended reality device. Figure 10B illustrates an example embodiment of sharing content to be presented later via a first wearable extended reality device. [Figure 11] 11A and 11B illustrate exemplary embodiments of shared actions, showing the use of gestures to send content to a physical screen. [Figure 12] 12A and 12B illustrate exemplary embodiments of shared actions, illustrating the use of gestures to share virtual content with one or more physical screens. [Figure 13] Figure 13A illustrates an example embodiment of sharing and presenting content via a second wearable extended reality device. Figure 13B illustrates an example embodiment of sharing and presenting content via a second wearable extended reality device. [Figure 14] FIG. 14 is a flowchart illustrating an example method of coordinating between a first wearable extended reality device and a second wearable extended reality device to display virtual content, consistent with disclosed embodiments. [Figure 15]FIG. 15 is a diagram of an exemplary environment including multiple users using multiple wearable extended reality devices, consistent with some embodiments of the present disclosure. [Figure 16] FIG. 16 is a diagram of an example virtual display including a portion of the virtual display provided in obscured form, consistent with certain embodiments of the present disclosure. [Figure 17] FIG. 17 is another example illustration of a virtual display provided in obscured form, consistent with certain embodiments of the present disclosure. [Figure 18] FIG. 18 is a flowchart of an exemplary method for providing situational awareness to a user of a wearable extended reality device, consistent with certain embodiments of the present disclosure. [Figure 19] FIG. 19 is a flowchart illustrating an example process for linking a virtual whiteboard to a physical space, consistent with certain embodiments of the present disclosure. [Figure 20] FIG. 20 is a diagram illustrating an example of linking a virtual whiteboard to a physical space, consistent with certain embodiments of the present disclosure. [Figure 21] FIG. 21 is a schematic diagram illustrating an example of linking a virtual whiteboard to a physical space, consistent with certain embodiments of the present disclosure. [Figure 22] FIG. 22 is a flowchart illustrating an example process for adding content to a virtual whiteboard, consistent with certain embodiments of the present disclosure. [Figure 23] FIG. 23 is a flowchart illustrating an example process for deleting content from a virtual whiteboard, consistent with certain embodiments of the present disclosure. [Figure 24] FIG. 24 is a flowchart illustrating an example process for uploading content to a virtual whiteboard based on content type, consistent with certain embodiments of the present disclosure. [Figure 25]FIG. 25 is a flowchart illustrating an example process for viewing content from a virtual whiteboard, consistent with certain embodiments of the present disclosure. [Figure 26] FIG. 26 is a flowchart illustrating an example process for viewing virtual whiteboard content of interest to a user, consistent with certain embodiments of the disclosure. [Figure 27] FIG. 27 shows examples of various physical display devices consistent with certain embodiments of the present disclosure. [Figure 28] FIG. 28 illustrates an example of transferring virtual content to a physical display device, consistent with certain embodiments of the present disclosure. [Figure 29] FIG. 29 illustrates example user gestures for transferring virtual content to a physical display device, consistent with certain embodiments of the present disclosure. [Figure 30] FIG. 30 illustrates an example of a pointing device input for transferring virtual content to a physical display device, consistent with certain embodiments of the present disclosure. [Figure 31] FIG. 31 is a flowchart of an example method 3100 for transferring virtual content to a physical display device, consistent with certain embodiments of the present disclosure. [Figure 32] FIG. 32 is a flowchart illustrating an example process for simulating user interaction with shared content, consistent with certain embodiments of the present disclosure. [Figure 33] FIG. 33 is a flowchart illustrating an example process for simulating user interaction with shared content, consistent with certain embodiments of the present disclosure. [Figure 34] FIG. 34 is a schematic diagram illustrating various usage snapshots of an exemplary system for simulating user interaction with shared content, consistent with certain embodiments of the present disclosure. [Figure 35]FIG. 35 is a schematic diagram illustrating various usage snapshots of an exemplary system for simulating user interaction with shared content, consistent with certain embodiments of the present disclosure. [Figure 36] FIG. 36 is a schematic diagram illustrating various usage snapshots of an exemplary system for simulating user interaction with shared content, consistent with certain embodiments of the present disclosure. [Figure 37-1] Figure 37A shows an example geographic location that has not yet been registered with any extended reality environment, and Figure 37B shows the example geographic location described in Figure 37A after it has been registered with a public extended reality environment, consistent with some embodiments of the present disclosure. [Figure 37-2] Figure 37C illustrates an example geographic location described in Figure 37B where virtual content is displayed within an extended reality environment, consistent with certain embodiments of the present disclosure. Figure 37D illustrates an example geographic location described in Figure 37B where a user can gesture to a geographic location where virtual content is desired for display, consistent with certain embodiments of the present disclosure. [Figure 38] FIG. 38 is a diagram illustrating content placement rules implemented based on various input parameters, consistent with certain embodiments of the present disclosure. [Figure 39] FIG. 39 is a flowchart of a process for determining whether to display content in a particular geographic location, consistent with certain embodiments of the present disclosure. [Figure 40] FIG. 40 is a flowchart illustrating an example method for managing content placement in an extended reality environment, consistent with certain embodiments of the present disclosure. [Figure 41] FIG. 41 illustrates one exemplary presentation of virtual content to multiple viewers, consistent with certain embodiments of the present disclosure. [Figure 42A] FIG. 42A illustrates another exemplary presentation of virtual content to multiple viewers, consistent with certain embodiments of the present disclosure. [Figure 42B] FIG. 42B illustrates an exemplary display of a virtual object through a wearable extended reality device rendered from various perspectives, consistent with some embodiments of the present disclosure. [Figure 43] FIG. 43 illustrates a further exemplary presentation of virtual content to multiple viewers, consistent with certain embodiments of the present disclosure. [Figure 44] FIG. 44 illustrates yet another exemplary presentation of virtual content to multiple viewers, consistent with certain embodiments of the present disclosure. [Figure 45] FIG. 45 illustrates an exemplary method consistent with certain embodiments of the present disclosure. [Figure 46] FIG. 46 illustrates an exemplary method consistent with certain embodiments of the present disclosure. [Figure 47] FIG. 47 illustrates an exemplary method consistent with certain embodiments of the present disclosure. [Figure 48] FIG. 48 illustrates an exemplary method consistent with certain embodiments of the present disclosure. [Figure 49] FIG. 49 is a schematic illustration of an individual applying virtual color markings to an object using an exemplary extended reality system, consistent with certain embodiments of the present disclosure. [Figure 50] FIG. 50 is a schematic diagram illustrating an object and a hand holding a physical marking implement, consistent with certain embodiments of the present disclosure. [Figure 51]Figure 51A is an example of several images taken from a wearable extended reality device at different times showing movement of a physical marking tool relative to an object, consistent with certain embodiments of the present disclosure. Figure 51B is an example of several images taken from a wearable extended reality device at different times showing movement of a physical marking tool relative to an object, consistent with certain embodiments of the present disclosure. Figure 51C is an example of several images taken from a wearable extended reality device at different times showing movement of a physical marking tool relative to an object, consistent with certain embodiments of the present disclosure. Figure 51D is an example of several images taken from a wearable extended reality device at different times showing movement of a physical marking tool relative to an object, consistent with certain embodiments of the present disclosure. [Figure 52] FIG. 52 is a schematic illustration of two individuals viewing and interacting with the same virtual color marking on the same object, consistent with certain embodiments of the present disclosure. [Figure 53] FIG. 53 is a schematic diagram of an individual holding a physical marking implement and multiple individuals at remote locations interacting with the same virtual color marking on the same object, consistent with some embodiments of the present disclosure. [Figure 54] FIG. 54 is a schematic illustration of an individual viewing virtual color markings on an object made by an individual holding a physical marking implement that is then removed from the object's environment, consistent with some embodiments of the present disclosure. [Figure 55] FIG. 55 is a schematic diagram illustrating an extended reality display object adjacent to a hand holding a physical marking implement and a virtual color palette, consistent with certain embodiments of the present disclosure. [Figure 56] FIG. 56 is a flowchart of a process for applying virtual color markings to an object, consistent with certain embodiments of the present disclosure. [Figure 57]FIG. 57 illustrates an example of multiple wearable extended reality devices with individual, unaligned coordinate systems, consistent with some embodiments of the present disclosure. [Figure 58] FIG. 58 illustrates an example of multiple wearable extended reality devices having a common coordinate system, consistent with some embodiments of the present disclosure. [Figure 59] FIG. 59 illustrates an example of an additional wearable extended reality device that combines multiple wearable extended reality devices, consistent with some embodiments of the present disclosure, where the additional device has a coordinate system that is not aligned with the common coordinate system. [Figure 60] FIG. 60 illustrates an example of an additional wearable extended reality device having a common coordinate system with multiple wearable extended reality devices, consistent with some embodiments of the present disclosure. [Figure 61] FIG. 61 shows a flowchart of an example method that may be executed by a processor to perform operations for sharing virtual content. DETAILED DESCRIPTION OF THE INVENTION

[0019] The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. While several exemplary embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or modifications may be made to the components shown in the drawings, and the exemplary methods described herein may be modified by substituting, rearranging, removing, or adding steps to the disclosed methods. The following detailed description, therefore, is not limited to the particular embodiments and examples, but includes the general principles described herein and illustrated in the figures, in addition to the general principles encompassed by the appended claims.

[0020] The present disclosure is directed to systems and methods for providing an extended reality environment to a user. The term "extended reality environment," which may also be referred to as "extended reality," "extended reality space," or "extended environment," refers to any type of combined real and virtual environment and human-machine interaction generated at least in part by computer technology. An extended reality environment may be a fully simulated virtual environment or a combined real and virtual environment that a user can perceive from various perspectives. In some examples, a user can interact with elements of the extended reality environment. One non-limiting example of an extended reality environment may be a virtual reality environment, also known as "virtual reality" or "virtual environment." An immersive virtual reality environment may be a simulated non-physical environment that provides a user with the perception of being in a virtual environment. Another non-limiting example of an extended reality environment may be an augmented reality environment, also known as "augmented reality" or "augmented environment." An augmented reality environment may include a live, direct or indirect view of a physical, real-world environment augmented with virtual, computer-generated sensory information, such as virtual objects with which a user can interact. Another non-limiting example of an extended reality environment is a mixed reality environment, also known as "mixed reality" or "composite environment." A mixed reality environment may be a hybrid of a physical, real-world and virtual environment, where physical and virtual objects coexist and may interact in real time. In some examples, both augmented reality and mixed reality environments may include a combination of real and virtual worlds, real-time interaction, and precise 3D registration of virtual and real objects. In some examples, both augmented reality and mixed reality environments may include structural, overlaid sensory information that may be added to the physical environment. In other examples, both augmented reality and mixed reality environments may include disruptive virtual content that may mask at least a portion of the physical environment.

[0021] In some embodiments, the present systems and methods can provide an extended reality environment using extended reality devices. The term extended reality device may include any type of device or system that allows a user to perceive and / or interact with an extended reality environment. An extended reality device may allow a user to perceive and / or interact with the extended reality environment through one or more sensory modalities. Some non-limiting examples of such sensory modalities may include vision, hearing, touch, somatosensation, and smell. One example of an extended reality device is a virtual reality device that allows a user to perceive and / or interact with a virtual reality environment. Another example of an extended reality device is an augmented reality device that allows a user to perceive and / or interact with an augmented reality environment. Yet another example of an extended reality device is a mixed reality device that allows a user to perceive and / or interact with a mixed reality environment.

[0022] Consistent with one aspect of the present disclosure, the extended reality device may be a wearable device, such as a head-mounted device, e.g., smart glasses, smart contact lenses, a headset, or any other device worn by a human for the purpose of presenting extended reality to the human. Other extended reality devices may include a holographic projector or any other device or system capable of providing augmented reality (AR), virtual reality (VR), mixed reality (MR), or any immersive experience. Typical components of a wearable extended reality device may include a stereoscopic head-mounted display, a stereoscopic head-mounted sound system, a head-motion tracking sensor (such as a gyroscope, accelerometer, magnetometer, image sensor, structured light sensor), a head-mounted projector, an eye-tracking sensor, and at least one of the additional components described below. Consistent with another aspect of the present disclosure, the extended reality device may be a non-wearable extended reality device. Specifically, the non-wearable extended reality device may include a multi-projection environment device. In some embodiments, the extended reality device may be configured to change the viewing perspective of the extended reality environment in response to user movement, and in particular in response to user head movement. In one example, a wearable extended reality device may change the field of view of the extended reality environment in response to a change in the user's head pose, such as by changing the spatial orientation without changing the user's spatial position within the extended reality environment. In another example, a non-wearable extended reality device may change the user's spatial position within the extended reality environment in response to a change in the user's position in the real world, for example, by changing the user's spatial position within the extended reality environment without changing the field of view orientation relative to the spatial position.

[0023] According to some embodiments, the extended reality equipment may include a digital communication device configured to at least one of receive virtual content data configured to enable presentation of virtual content, transmit virtual content for sharing with at least one external device, receive contextual data from the at least one external device, transmit contextual data to the at least one external device, transmit usage data indicative of use of the extended reality equipment, and transmit data based on information captured using at least one sensor included in the extended reality equipment. In additional embodiments, the extended reality equipment may include a memory for storing at least one of virtual data configured to enable presentation of virtual content, contextual data, usage data indicative of use of the extended reality equipment, sensor data based on information captured using at least one sensor included in the extended reality equipment, software instructions configured to cause a processing device to present the virtual content, software instructions configured to cause the processing device to collect and analyze the contextual data, software instructions configured to cause the processing device to collect and analyze the usage data, and software instructions configured to cause the processing device to collect and analyze the sensor data. In additional embodiments, the extended reality equipment may include a processing device configured to at least one of render virtual content, collect and analyze contextual data, collect and analyze usage data, and collect and analyze sensor data. In additional embodiments, the extended reality equipment may include one or more sensors.The one or more sensors may include one or more image sensors (e.g., configured to capture images and / or video of a user of the device or the user's environment), one or more motion sensors (such as accelerometers, gyroscopes, magnetometers), one or more positioning sensors (such as GPS, outdoor positioning sensors, indoor positioning sensors), one or more temperature sensors (e.g., configured to measure the temperature of at least a portion of the device and / or environment), one or more contact sensors, one or more proximity sensors (e.g., configured to detect whether the device is currently being worn), one or more electrical impedance sensors (e.g., configured to measure the user's electrical impedance), one or more eye tracking sensors such as a gaze detector, an optical tracker, an electric potential tracker (e.g., an electro-oculogram (EOG) sensor), a video-based eye tracker, an infrared / near-infrared sensor, a passive optical sensor, or any other technology capable of determining where a human is looking or gazing.

[0024] In some embodiments, the present systems and methods can interact with the extended reality equipment using input devices. The term input device may include any physical device configured to receive input from a user or the user's environment and provide data to a computing device. The data provided to the computing device may be in digital and / or analog format. In one embodiment, the input device can store input received from a user in a memory device accessible by the processing device, and the processing device can access the stored data for analysis. In another embodiment, the input device can provide data directly to the processing device, for example, via a bus or another communication system configured to transfer data from the input device to the processing device. In some examples, input received by the input device may include key presses, tactile input data, movement data, position data, gesture-based input data, directional data, or any other data to feed into a computation. Some examples of input devices may include buttons, keys, keyboards, computer mice, touchpads, touchscreens, joysticks, or other mechanisms by which input may be received. Another example of an input device may include an integrated computing interface device that includes at least one physical component for receiving input from a user. The integrated computing interface device may include at least a memory, a processing device, and at least one physical component for receiving input from a user. In one example, the integrated computing interface device may further include a digital network interface that enables digital communication with other computing devices. In one example, the integrated computing interface device may further include a physical component for outputting information to a user. In some examples, all components of the integrated computing interface device may be contained in a single housing, while in other examples, the components may be distributed among two or more housings.Some non-limiting examples of physical components for receiving input from a user that may be included in an integrated computing interface device may include at least one of buttons, keys, a keyboard, a touchpad, a touchscreen, a joystick, or any other mechanism or sensor from which computational information may be received. Some non-limiting examples of physical components for outputting information to a user may include at least one of a light indicator (such as an LED indicator), a screen, a touchscreen, a beeper, an audio speaker, or any other audio, video, or tactile device that provides a human-perceptible output.

[0025] In some embodiments, the image data may be captured using one or more image sensors. In some examples, the image sensors may be included in an extended reality device, a wearable device, a wearable extended reality device, an input device, the user's environment, etc. In some examples, the image data may be read from memory, received from an external device, generated (e.g., using a generative model), etc. Some non-limiting examples of image data may include images, grayscale images, color images, 2D images, 3D images, videos, 2D videos, 3D videos, frames, scenes, data derived from other image data, etc. In some examples, the image data may be encoded in any analog or digital format. Some non-limiting examples of such formats may include raw formats, compressed formats, uncompressed formats, lossy formats, lossless formats, JPEG, GIF, PNG, TIFF, BMP, NTSC, PAL, SECAM, MPEG, MPEG-4 Part 14, MOV, WMV, FLV, AVI, AVCHD, WebM, MKV, etc.

[0026] In some embodiments, the extended reality device may receive a digital signal, for example, from an input device. The term digital signal refers to a series of digital values ​​that are discrete in time. The digital signal may represent, for example, sensor data, text data, audio data, video data, virtual data, or any other form of data that provides perceptible information. Consistent with the present disclosure, the digital signal may be configured to cause the extended reality device to present virtual content. In one embodiment, the virtual content may be presented in a selected orientation. In this embodiment, the digital signal may indicate the position and angle of a viewpoint in an environment, such as an extended reality environment. Specifically, the digital signal may include encoding of position and angle in six degrees of freedom (e.g., forward / backward, up / down, left / right, yaw, pitch, and roll). In another embodiment, the digital signal may include encoding of position as three-dimensional coordinates (e.g., x, y, and z) and encoding of angle as a vector resulting from the encoded position. Specifically, the digital signal may indicate the orientation and angle of the presented virtual content in absolute coordinates of the environment, for example, by encoding the yaw, pitch, and roll of the virtual content relative to a standard default angle. In another embodiment, the digital signal may indicate the orientation and angle of the presented virtual content relative to the viewpoint of another object (e.g., a virtual object, a physical object, etc.), for example, by encoding the yaw, pitch, and roll of the virtual content relative to a direction corresponding to the viewpoint or a direction corresponding to another object. In another embodiment, such a digital signal may include one or more projections of the virtual content, for example, in a presentation-ready format (e.g., an image, a video, etc.). For example, each such projection may correspond to a particular orientation or a particular angle. In another embodiment, the digital signal may include a representation of the virtual content, for example, by encoding the object as a three-dimensional array of voxels, a polygonal mesh, or any other format in which the virtual content may be presented.

[0027] In some embodiments, the digital signal may be configured to cause the extended reality device to present virtual content. The term virtual content may include any type of data representation that can be displayed to a user by the extended reality device. Virtual content may include virtual objects, non-animated virtual content, animated virtual content configured to change over time or in response to a trigger, virtual two-dimensional content, virtual three-dimensional content, virtual overlays on portions of a physical environment or on physical objects, virtual additions to a physical environment or object, virtual promotional content, virtual representations of physical objects, virtual representations of a physical environment, virtual documents, virtual characters or personas, virtual computer screens, virtual widgets, or any other format for virtually displaying information. Consistent with this disclosure, virtual content may include any visual presentation rendered by a computer or processing device. In one embodiment, virtual content may include virtual objects, which are visual presentations rendered by a computer within a limited area and configured to represent a particular type of object (such as a non-animated virtual object, an animated virtual object, a virtual fixture, a virtual decorative object, a virtual widget, or other virtual representation). The rendered visual presentation may be changed to reflect changes to a status object or a change in the viewing angle of the object, for example, to mimic a change in the appearance of a physical object. In another embodiment, the virtual content may include a virtual display (also referred to herein as a "virtual display screen" or "virtual screen"), such as a virtual computer screen, a virtual tablet screen, or a virtual smartphone screen, configured to display information generated by an operating system, where the operating system may be configured to receive text data from a physical keyboard and / or a virtual keyboard and cause the text content to be displayed within the virtual display screen. In one example shown in FIG. 1, the virtual content may include a virtual environment including a virtual computer screen and a plurality of virtual objects.In some examples, the virtual display may be a virtual object that mimics and / or extends the functionality of a physical display screen. For example, the virtual display may be presented within an extended reality environment (such as a mixed reality environment, an augmented reality environment, or a virtual reality environment) using an extended reality device. In one example, the virtual display may present content generated by a conventional operating system that may be presented equivalently on a physical display screen. In one example, text content entered using a keyboard (e.g., using a physical keyboard, using a virtual keyboard, etc.) may be presented on the virtual display in real time as the text content is typed. In one example, a virtual cursor may be presented on the virtual display, and the virtual cursor may be controlled by a pointing device (such as a physical pointing device, a virtual pointing device, a computer mouse, a joystick, a touchpad, or a physical touch controller). In one example, one or more windows of a graphical user interface operating system may be presented on the virtual display. In another example, the content presented on the virtual display may be interactive, i.e., may change in response to a user's actions. In yet another example, the presentation of the virtual display may or may not include the presentation of a screen frame.

[0028] Some disclosed embodiments may include and / or access a data structure or database. The terms data structure and database consistent with this disclosure may include any collection of data values ​​and the relationships between them. Data may be stored linearly, horizontally, hierarchically, relationally, non-relationally, unidimensionally, multidimensionally, operationally, in an ordered manner, in an unordered manner, in an object-oriented manner, in a centralized manner, in a decentralized manner, in a distributed manner, in a custom manner, or in any manner that enables data access. As non-limiting examples, data structures may include arrays, associative arrays, linked lists, binary trees, balanced trees, heaps, stacks, queues, sets, hash tables, records, tagged unions, entity-relationship models, graphs, hypergraphs, matrices, tensors, etc. For example, the data structure may include an XML database, an RDBMS database, an SQL database, or a NoSQL alternative for data storage / retrieval, such as, for example, MongoDB, Redis, Couchbase, Datastax Enterprise Graph, Elastic Search, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, and Neo4J. The data structure may be a component of the disclosed system or a remote computing component (e.g., a cloud-based data structure). The data in the data structure may be stored in contiguous or non-contiguous memory. Furthermore, the data structure does not require the information to be co-located. It may be distributed, for example, across multiple servers, which may be owned or operated by the same or different entities. Thus, the term "data structure" in the singular includes multiple data structures.

[0029] In some embodiments, the system can determine a confidence level in a received input or any determined value. The term confidence level refers to any indication, numerical value, or otherwise, of a level (e.g., within a predetermined range) that indicates the amount of confidence the system has in the determined data. For example, the confidence level may have a value between 1 and 10. Alternatively, the confidence level may be expressed as a percentage or any other numerical or non-numerical indication. In some cases, the system may compare the confidence level to a threshold. The term threshold may refer to a reference value, level, point, or range of values. In operation, when the confidence level of the determined data exceeds (or falls below, depending on the particular use case) the threshold, the system can follow a first course of action, and when the confidence level falls below (or exceeds, depending on the particular use case), the system can follow a second course of action. The value of the threshold may be predetermined for each type of object being inspected or may be dynamically selected based on various considerations.

[0030] System Overview Referring now to FIG. 1 , a user is shown using an exemplary extended reality system consistent with various embodiments of the present disclosure. It should be understood that FIG. 1 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements added within the scope of the present disclosure. As shown, a user 100 is seated behind a table 102 supporting a keyboard 104 and a mouse 106. The keyboard 104 is connected by wires 108 to a wearable extended reality device 110 that displays virtual content to the user 100. As an alternative to, or in addition to, the wires 108, the keyboard 104 may be wirelessly connected to the wearable extended reality device 110. For purposes of illustration, the wearable extended reality device is shown as a pair of smart glasses, but as noted above, the wearable extended reality device 110 may be any type of head-mounted device used to present extended reality to the user 100. The virtual content displayed by the wearable extended reality device 110 includes a virtual screen 112 (also referred to herein as a “virtual display screen” or “virtual display”) and a plurality of virtual widgets 114. Virtual widgets 114A-114D are displayed adjacent to the virtual screen 112, and virtual widget 114E is displayed on the table 102. The user 100 can use the keyboard 104 to input text into a document 116 displayed on the virtual screen 112 and can use the mouse 106 to control a virtual cursor 118. In one example, the virtual cursor 118 can move anywhere within the virtual screen 112. In another example, the virtual cursor 118 can move anywhere within the virtual screen 112 and can also move to any one of the virtual widgets 114A-114D, but cannot move to the virtual widget 114E. In yet another example, the virtual cursor 118 can move anywhere within the virtual screen 112 and can also move to any one of the virtual widgets 114A-114E.In an additional example, the virtual cursor 118 can move anywhere within the extended reality environment, including the virtual screen 112 and the virtual widgets 114A-114E. In yet another example, the virtual cursor can move over all available surfaces (i.e., virtual or physical surfaces) or only over selected surfaces within the extended reality environment. Alternatively, or in addition, the user 100 can interact with any one of the virtual widgets 114A-114E, or with a selected virtual widget, using hand gestures recognized by the wearable extended reality device 110. For example, the virtual widget 114E may be an interactive widget (e.g., a virtual slider controller) that can be manipulated with hand gestures.

[0031] FIG. 2 illustrates an example of a system 200 for providing an extended reality (XR) experience to a user, such as user 100. It should be understood that FIG. 2 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of this disclosure. System 200 may be computer-based and may include computer system components, wearable devices, workstations, tablets, handheld computing devices, memory devices, and / or an internal network connecting the components. System 200 may include or be connected to various network computing resources (e.g., servers, routers, switches, network connections, storage devices, etc.) to support the services provided by system 200. Consistent with this disclosure, system 200 may include an input unit 202, an XR unit 204, a mobile communication device 206, and a remote processing unit 208. Remote processing unit 208 may include a server 210 coupled to one or more physical or virtual storage devices, such as a data structure 212. The system 200 may also include or be connected to a communications network 214 that facilitates communication and data exchange between various system components and various entities associated with the system 200 .

[0032] Consistent with the present disclosure, the input unit 202 may include one or more devices capable of receiving input from the user 100. In one embodiment, the input unit 202 may include a text input device such as the keyboard 104. The text input device may include all possible types of devices and mechanisms for inputting text information into the system 200. Examples of text input devices may include a mechanical keyboard, a membrane keyboard, a flexible keyboard, a QWERTY keyboard, a Dvorak keyboard, a Colemak keyboard, a corded keyboard, a wireless keyboard, a keypad, a key-based control panel, or other arrangement of control keys, a visual input device, or any other mechanism for inputting text, whether the mechanism is provided in physical form or presented virtually. In one embodiment, the input unit 202 may also include a pointing input device such as the mouse 106. The pointing input device may include all possible types of devices and mechanisms for inputting two-dimensional or three-dimensional information into the system 200. In one example, two-dimensional input from the pointing input device may be used to interact with virtual content presented via the XR unit 204. Examples of pointing input devices may include a computer mouse, trackball, touchpad, trackpad, touchscreen, joystick, pointing stick, stylus, light pen, or any other physical or virtual input mechanism. In one embodiment, the input unit 202 may also include a graphical input device such as a touchscreen configured to detect contact, movement, or interruption of movement. The graphical input device may use any of a number of touch-sensitivity technologies, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact. In one embodiment, the input unit 202 may also include one or more audio input devices, such as a microphone.The audio input device may include all possible types of devices and mechanisms for inputting audio data to facilitate voice-enabled functions such as voice recognition, audio replication, digital recording, and telephone functions. In one embodiment, the input unit 202 may also include one or more image input devices, such as image sensors, configured to capture image data. In one embodiment, the input unit 202 may also include one or more haptic gloves configured to capture hand movement and posture data. In one embodiment, the input unit 202 may also include one or more proximity sensors configured to detect the presence and / or movement of objects within a selected area near the sensors.

[0033] According to some embodiments, the system may include at least one sensor configured to detect and / or measure characteristics associated with a user, the user's actions, or the user's environment. One example of the at least one sensor is the sensor 216 included in the input unit 202. The sensor 216 may be a motion sensor, a touch sensor, a light sensor, an infrared sensor, an audio sensor, an image sensor, a proximity sensor, a positioning sensor, a gyroscope, a temperature sensor, a biometric sensor, or any other sensing device for facilitating related functions. The sensor 216 may be integrated with or connected to the input device, or may be separate from the input device. In one example, a thermometer may be included in the mouse 106 to measure the user's 100 body temperature. In another example, a positioning sensor may be integrated with the keyboard 104 to determine the user's 100 movement relative to the keyboard 104. Such a positioning sensor may be implemented using one of the following technologies: Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Galileo Global Navigation System, BeiDou Navigation System, other Global Navigation Satellite Systems (GNSS), Indian Regional Navigation Satellite System (IRNSS), Local Positioning System (LPS), Real-Time Location System (RTLS), Indoor Positioning System (IPS), Wi-Fi based positioning system, cellular triangulation, image-based positioning technology, indoor positioning technology, outdoor positioning technology, or any other positioning technology.

[0034] According to some embodiments, the system may include one or more sensors for identifying the position and / or movement of a physical device (e.g., a physical input device, a physical computing device, a keyboard 104, a mouse 106, a wearable extended reality device 110, etc.). The one or more sensors may be included in the physical device or may be external to the physical device. In some examples, an image sensor external to the physical device (e.g., an image sensor included in another physical device) may be used to capture image data of the physical device, and the image data may be analyzed to identify the position and / or movement of the physical device. For example, the image data may be analyzed using a visual object tracking algorithm to identify movement of the physical device, a visual object detection algorithm to identify the position of the physical device (e.g., position relative to the image sensor, position in a global coordinate system, etc.), etc. In some examples, an image sensor included in the physical device may be used to capture image data, and the image data may be analyzed to identify the position and / or movement of the physical device. For example, the image data may be analyzed using a visual odometry algorithm to identify the position of the physical device, an egomotion algorithm to identify movement of the physical device, etc. In some examples, a positioning sensor, such as an indoor positioning sensor or an outdoor positioning sensor, may be included in a physical device and used to determine the location of the physical device. In some examples, a motion sensor, such as an accelerometer or gyroscope, may be included in a physical device and used to determine movement of the physical device. In some examples, a physical device, such as a keyboard or mouse, may be configured to be positioned on a physical surface. Such a physical device may include an optical mouse sensor (also known as a non-mechanical tracking engine) pointed at the physical surface, and the output of the optical mouse sensor may be analyzed to determine movement of the physical device relative to the physical surface.

[0035] Consistent with this disclosure, the XR unit 204 may include a wearable extended reality device configured to present virtual content to the user 100. One example of a wearable extended reality device is the wearable extended reality device 110. Additional examples of wearable extended reality devices may include virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, or any other device capable of generating extended reality content. Some non-limiting examples of such devices may include Nreal Light, Magic Leap One, Varjo, Quest 1 / 2, Vive, etc. In some embodiments, the XR unit 204 may present virtual content to the user 100. In general, extended reality devices may include all real-virtual composite environments and human-machine interactions generated by computer technology and wearables. As discussed above, the term “extended reality” (XR) refers to a superset that includes all real-virtual and virtual-virtual environments. This includes representative forms such as augmented reality (AR), mixed reality (MR), and virtual reality (VR), as well as areas interpolated therebetween. Accordingly, it should be noted that the terms "XR device," "AR device," "VR device," and "MR device" may be used interchangeably herein and may refer to any device from the various devices listed above.

[0036] Consistent with the present disclosure, the system can exchange data with various communication devices associated with a user, such as a mobile communication device 206. The term “communication device” is intended to include all possible types of devices capable of exchanging data using a digital communication network, an analog communication network, or any other communication network configured to transmit data. In some examples, the communication device may include a smartphone, a tablet, a smartwatch, a personal digital assistant, a desktop computer, a laptop computer, an IoT device, a dedicated terminal, a wearable communication device, and any other device that enables data communication. In some cases, the mobile communication device 206 can complement or replace the input unit 202. Specifically, the mobile communication device 206 may be associated with a physical touch controller that can function as a pointing input device. Furthermore, the mobile communication device 206 may also be used, for example, to implement a virtual keyboard and replace a text input device. For example, if the user 100 leaves the table 102 and walks to the break room with their smart glasses, the user may receive an email that requires a quick response. In this case, the user can choose to use their smartwatch as an input device and type a response to the email while it is virtually presented by the smartglasses.

[0037] Consistent with the present disclosure, embodiments of the present system may include the use of a cloud server. The term “cloud server” refers to a computer platform that provides services over a network, such as the Internet. In the exemplary embodiment shown in FIG. 2, the server 210 may use a virtual machine that does not need to correspond to individual hardware. For example, computing and / or storage capabilities may be implemented by allocating appropriate portions of desired computing / storage capabilities from a scalable repository, such as a data center or a distributed computing environment. Specifically, in one embodiment, the remote processing unit 208 may be used in conjunction with the XR unit 204 to provide virtual content to the user 100. In one exemplary configuration, the server 210 may be a cloud server that functions as an operating system (OS) for a wearable extended reality device. In one example, the server 210 may implement the methods described herein using customized hardwired logic, one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), firmware, and / or program logic that, in combination with a computer system, makes the server 210 a dedicated machine.

[0038] In some embodiments, server 210 can access data structure 212 to, for example, determine virtual content to display to user 100. Data structure 212 can utilize volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, or other types of storage devices or tangible or non-transitory computer-readable media, or any medium or mechanism for storing information. Data structure 212 may be part of server 210, as shown, or may be separate from server 210. If data structure 212 is not part of server 210, server 210 can exchange data with data structure 212 via a communications link. Data structure 212 may include one or more memory devices that store data and instructions used to execute one or more features of the disclosed methods. In one embodiment, data structure 212 may include any of several suitable data structures, ranging from small data structures hosted on a workstation to large data structures distributed across a data center. Data structure 212 may also include any combination of one or more data structures controlled by a memory controller device (e.g., a server) or software.

[0039] Consistent with this disclosure, a communications network may be any type of network (including infrastructure) that supports communications, exchanges information, and / or facilitates the exchange of information between components of a system. For example, communications network 214 in system 200 may include, for example, a telephone network, an extranet, an intranet, the Internet, satellite communications, offline communications, radio communications, transponder communications, a local area network (LAN), a wireless network (e.g., a Wi-Fi / 302.11 network), a wide area network (WAN), a virtual private network (VPN), a digital communications network, an analog communications network, or any other mechanism or combination of mechanisms that enables data transmission.

[0040] The components and arrangement of system 200 shown in FIG. 2 are intended to be illustrative only and not limiting of any embodiment, as the system components used to implement the disclosed processes and features may vary.

[0041] FIG. 3 is a block diagram of an exemplary configuration of input unit 202. It should be understood that FIG. 3 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. In the embodiment of FIG. 3, input unit 202 may have direct or indirect access to a bus 300 (or other communication mechanism) that interconnects subsystems and components for transferring information within input unit 202. For example, bus 300 may interconnect a memory interface 310, a network interface 320, an input interface 330, a power supply 340, an output interface 350, a processing device 360, a sensor interface 370, and a database 380.

[0042] The memory interface 310 shown in FIG. 3 can be used to access software products and / or data stored on a non-transitory computer-readable medium. Generally, a non-transitory computer-readable storage medium refers to any type of physical memory capable of storing information or data readable by at least one processor. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD-ROMs, DVDs, flash drives, disks, any other optical data storage media, any physical media with a pattern of holes, PROMs, EPROMs, flash EPROMs or any other flash memory, NVRAM, cache, registers, any other memory chips or cartridges, and networked versions thereof. The terms "memory" and "computer-readable storage medium" can refer to multiple structures, such as multiple memories or computer-readable storage media located within an input unit or at a remote location. In addition, one or more computer-readable storage media can be utilized in practicing a computer-implemented method. Thus, the term computer-readable storage medium should be understood to include tangible items and exclude carrier waves and transient signals. 3, memory interface 310 may be used to access software products and / or data stored in a memory device, such as memory device 311. Memory device 311 may include high-speed random access memory and / or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and / or flash memory (e.g., NAND, NOR). Consistent with the present disclosure, components of memory device 311 may be distributed across two or more units and / or two or more memory devices of system 200.

[0043] The memory device 311 shown in FIG. 3 may include software modules for executing processes consistent with the present disclosure. In particular, the memory device 311 may include an input determination module 312, an output determination module 313, a sensor communication module 314, a virtual content determination module 315, a virtual content communication module 316, and a database access module 317. Modules 312-317 may include software instructions for execution by at least one processor (e.g., processing device 360) associated with the input unit 202. The input determination module 312, the output determination module 313, the sensor communication module 314, the virtual content determination module 315, the virtual content communication module 316, and the database access module 317 may cooperate to perform various operations. For example, the input determination module 312 may determine text using data received, for example, from the keyboard 104. The output determination module 313 may then cause the recently entered text to be presented, for example, on a dedicated display 352 physically or wirelessly coupled to the keyboard 104. In this way, as the user 100 types, they can see a preview of the typed text without having to constantly move their head up and down to look at the virtual screen 112. The sensor communication module 314 can receive data from various sensors to determine the status of the user 100. The virtual content determination module 315 can then determine virtual content to display based on the received input and the determined status of the user 100. For example, the determined virtual content may be a virtual presentation of recently entered text on a virtual screen virtually positioned adjacent to the keyboard 104. The virtual content communication module 316 can retrieve virtual content not determined by the virtual content determination module 315 (e.g., another user's avatar). The retrieval of virtual content may be from the database 380, from the remote processing unit 208, or from any other source.

[0044] In some embodiments, the input determination module 312 can adjust the operation of the input interface 330 to receive pointer input 331, text input 332, audio input 333, and XR-related input 334. Pointer input, text input, and audio input are described in more detail above. The term “XR-related input” may include any type of data that may cause a change in virtual content displayed to the user 100. In one embodiment, the XR-related input 334 may include image data of the user 100, a wearable extended reality device (e.g., a detected hand gesture of the user 100), or a wireless communication indicating the presence of another user in proximity to the user 100. Consistent with the present disclosure, the input determination module 312 can simultaneously receive different types of input data. The input determination module 312 can then further apply various rules based on the type of input detected. For example, pointer input may be prioritized over voice input.

[0045] In some embodiments, the output determination module 313 can coordinate operation of the output interface 350 to generate output using light indicators 351, display 352, and / or speaker 353. Generally, the output generated by the output determination module 313 does not include virtual content presented by the wearable extended reality device. Instead, the output generated by the output determination module 313 includes various outputs related to operation of the input unit 202 and / or operation of the XR unit 204. In one embodiment, the light indicator 351 may include a light indicator that indicates the status of the wearable extended reality device. For example, the light indicator may display a green light when the wearable extended reality device 110 is connected to the keyboard 104 and flash when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 352 may be used to display operational information. For example, the display may present an error message when the wearable extended reality device is inoperable. In another embodiment, the speaker 353 may be used to output sound, for example, when the user 100 wants to play some music for other users.

[0046] In some embodiments, the sensor communications module 314 can coordinate the operation of the sensor interface 370 to receive sensor data from one or more sensors integrated with or connected to the input device. The one or more sensors may include an audio sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374 (e.g., a temperature sensor, an ambient light detector, etc.), and other sensors 375. In one embodiment, the data received from the sensor communications module 314 can be used to determine the physical orientation of the input device. The physical orientation of the input device can indicate the user's state and can be determined based on a combination of tilt motion, roll motion, and lateral motion. The physical orientation of the input device can then be used by the virtual content determination module 315 to modify the display parameters of the virtual content to match the user's state (e.g., alert, sleepy, active, sitting, standing, leaning back, leaning forward, walking, moving, riding, etc.).

[0047] In some embodiments, the virtual content determination module 315 can determine virtual content to be displayed by the wearable extended reality device. The virtual content can be determined based on data from the input determination module 312, the sensor communication module 314, and other sources (e.g., database 380). In some embodiments, determining the virtual content can include determining the distance, size, and orientation of a virtual object. The position of the virtual object can be determined based on the type of the virtual object. Specifically, with respect to the example shown in FIG. 1 , the virtual content determination module 315 can determine that the four virtual widgets 114A-114D should be placed on both sides of the virtual screen 112 and that the virtual widget 114E should be placed on the table 102 because the virtual widget 114E is a virtual controller (e.g., a volume bar). The position of the virtual object can be further determined based on a user's preference. For example, for a left-handed user, the virtual content determination module 315 may determine to place the virtual volume bar to the left of the keyboard 104, and for a right-handed user, the virtual content determination module 315 may determine to place the virtual volume bar to the right of the keyboard 104.

[0048] In some embodiments, the virtual content communication module 316 can coordinate the operation of the network interface 320 to obtain data from one or more sources to be presented as virtual content to the user 100. The one or more sources may include other XR units 204, the user's mobile communication device 206, the remote processing unit 208, publicly available information, etc. In one embodiment, the virtual content communication module 316 can communicate with the mobile communication device 206 to provide a virtual representation of the mobile communication device 206. For example, the virtual representation may allow the user 100 to read messages and interact with applications installed on the mobile communication device 206. The virtual content communication module 316 can also coordinate the operation of the network interface 320 to share virtual content with other users. In one example, the virtual content communication module 316 can use data from the input determination module to identify triggers (e.g., the triggers may include user gestures) and transfer content from a virtual display to a physical display (e.g., a television) or to a different user's virtual display.

[0049] In some embodiments, the database access module 317 can cooperate with the database 380 to retrieve stored data. The retrieved data may include, for example, privacy levels associated with various virtual objects, relationships between virtual and physical objects, user preferences, the user's past behavior, etc. As described above, the virtual content determination module 315 can determine virtual content using data stored in the database 380. The database 380 may include separate databases, including, for example, a vector database, a raster database, a tile database, a viewport database, and / or a user input database. The data stored in the database 380 may be received from modules 314-317 or other components of the system 200. Additionally, the data stored in the database 380 may be provided as input using data entry, data transfer, or data upload.

[0050] The modules 312-317 may be implemented in software, hardware, firmware, any combination thereof, etc. In some embodiments, data associated with any one or more of the modules 312-317 and the database 380 may be stored in the XR unit 204, the mobile communication device 206, or the remote processing unit 208. A processing device of the system 200 may be configured to execute instructions of the modules 312-317. In some embodiments, aspects of the modules 312-317 may be implemented in hardware, software (including one or more signal processing and / or application specific integrated circuits), firmware, or any combination thereof, executable by one or more processors, alone or in various combinations with each other. In particular, the modules 312-317 may be configured to interact with each other and / or with other modules of the system 200 to perform functions consistent with some disclosed embodiments. For example, the input unit 202 may execute instructions including image processing algorithms on data from the XR unit 204 to determine head movement of the user 100. Furthermore, each function described throughout this specification with respect to input unit 202 or with respect to components of input unit 202 may correspond to a set of instructions for performing said function. These instructions need not be implemented as a separate software program, procedure, or module. Memory device 311 may include additional or fewer modules and instructions. For example, memory device 311 may store an operating system such as ANDROID, iOS, UNIX, OSX, WINDOWS, DARWIN, RTXC, LINUX, or an embedded operating system such as VXWorkS. The operating system may include instructions for handling basic system services and performing hardware-dependent tasks.

[0051] The network interface 320 shown in FIG. 3 can provide bidirectional data communication to a network, such as the communications network 214. In one embodiment, the network interface 320 may include an Integrated Services Digital Network (ISDN) card, a cellular modem, a satellite modem, or a modem for providing a data communication connection via the Internet. As another example, the network interface 320 may include a Wireless Local Area Network (WLAN) card. In another embodiment, the network interface 320 may include an Ethernet port connected to a radio frequency receiver and transmitter and / or an optical (e.g., infrared) receiver and transmitter. The specific design and implementation of the network interface 320 may depend on the communications network or networks over which the input unit 202 is intended to operate. For example, in some embodiments, the input unit 202 may include a network interface 320 designed to operate on a GSM network, a GPRS network, an EDGE network, a Wi-Fi or WiMax network, and a Bluetooth network. In any such implementation, the network interface 320 may be configured to transmit and receive electrical, electromagnetic, or optical signals that carry digital data streams or signals representing various types of information.

[0052] The input interface 330 shown in FIG. 3 can receive input from a variety of input devices, such as a keyboard, a mouse, a touchpad, a touchscreen, one or more buttons, a joystick, a microphone, an image sensor, and any other device configured to detect physical or virtual input. The received input can be in the form of at least one of text, sound, speech, hand gestures, body gestures, tactile information, and any other type of physical or virtual input generated by a user. In the illustrated embodiment, the input interface 330 can receive pointer input 331, text input 332, audio input 333, and XR-related input 334. In additional embodiments, the input interface 330 can be an integrated circuit that can function as a bridge between the processing device 360 ​​and any of the input devices listed above.

[0053] The power source 340 shown in FIG. 3 provides electrical energy to the input unit 202 and can optionally also power the XR unit 204. In general, the power source included in any device or system in this disclosure may be any device that can repeatedly store, distribute, or transmit electrical power, including, but not limited to, one or more batteries (e.g., lead-acid batteries, lithium-ion batteries, nickel-metal hydride batteries, nickel-cadmium batteries), one or more capacitors, one or more connections to external power sources, one or more power converters, or any combination thereof. Referring to the example shown in FIG. 3 , the power source may be portable, meaning that the input unit 202 can be easily carried by hand (e.g., the total weight of the power source 340 may be less than 1 pound). The mobility of the power source allows the user 100 to use the input unit 202 in a variety of situations. In other embodiments, the power source 340 may be associated with a connection to an external power source (such as a power grid) that can be used to charge the power source 340. Additionally, the power source 340 may be configured to charge one or more batteries included in the XR unit 204, for example, so that extended reality glasses (e.g., wearable extended reality device 110) can be charged (e.g., wirelessly or not) when the glasses are placed on or in proximity to the input unit 202.

[0054] 3 can be output through various output devices using, for example, light indicators 351, display 352, and / or speaker 353. In one embodiment, output interface 350 may be an integrated circuit that can act as a bridge between processing device 360 ​​and at least one of the output devices listed above. Light indicator 351 may include one or more light sources, for example, an LED array associated with various colors. Display 352 may include a screen (e.g., an LCD or dot-matrix screen) or a touchscreen. Speaker 353 may include audio headphones, a hearing aid-type device, a speaker, bone conduction headphones, an interface that provides tactile cues, a vibrotactile stimulator, etc.

[0055] The processing device 360 ​​shown in FIG. 3 may include at least one processor configured to execute computer programs, applications, methods, processes, or other software to perform the embodiments described in this disclosure. Generally, the processing device included in any device or system in this disclosure may include one or more integrated circuits, microchips, microcontrollers, microprocessors, all or part of a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), field programmable gate array (FPGA), or other circuitry suitable for executing instructions or performing logical operations. The processing device may include at least one processor configured to perform the functions of the disclosed methods, such as a microprocessor manufactured by Intel™. The processing device may include a single-core or multi-core processor that simultaneously executes parallel processes. In one example, the processing device may be a single-core processor configured with virtual processing technology. The processing device may implement virtual machine technology or other technologies to provide the ability to run, control, operate, manipulate, store, etc., multiple software processes, applications, programs, etc. In another example, the processing device may include a multi-core processor configuration (e.g., dual-core, quad-core, etc.) configured to provide parallel processing capabilities to enable devices associated with the processing device to execute multiple processes simultaneously. It should be understood that other types of processor configurations may be implemented to provide the capabilities disclosed herein.

[0056] 3 can acquire sensor data from a variety of sensors, such as audio sensor 371, image sensor 372, motion sensor 373, environmental sensor 374, and other sensors 375. In one embodiment, sensor interface 370 can be an integrated circuit that can act as a bridge between processing device 360 ​​and at least one of the sensors listed above.

[0057] Audio sensor 371 may include one or more audio sensors configured to capture audio by converting sound into digital information. Some examples of audio sensors may include a microphone, a unidirectional microphone, a bidirectional microphone, a cardioid microphone, an omnidirectional microphone, an on-board microphone, a wired microphone, a wireless microphone, or any combination of the above. Consistent with the present disclosure, processing device 360 ​​can modify the presentation of virtual content based on data (e.g., voice commands) received from audio sensor 371.

[0058] Image sensor 372 may include one or more image sensors configured to capture visual information by converting light into image data. Consistent with this disclosure, the image sensor may be included in any device or system herein and may be any device capable of detecting and converting optical signals in the near-infrared, infrared, visible, and ultraviolet spectrums into electrical signals. Examples of image sensors may include digital cameras, phone cameras, semiconductor charge-coupled devices (CCDs), active pixel sensors in complementary metal-oxide semiconductor (CMOS), or N-type metal-oxide semiconductor (NMOS). The electrical signals may be used to generate image data. Consistent with this disclosure, the image data may include pixel data streams, digital images, digital video streams, data obtained from captured images, and data that can be used to construct one or more 3D images, a sequence of 3D images, a 3D video, or a virtual 3D representation. Image data acquired by image sensor 372 may be transmitted to any processing device of system 200 via wired or wireless transmission. For example, the image data may be processed to detect objects, detect events, detect actions, detect faces, detect people, recognize known persons, or any other information that may be used by system 200. Consistent with the present disclosure, processing device 360 ​​may modify the presentation of virtual content based on the image data received from image sensor 372.

[0059] The motion sensor 373 may include one or more motion sensors configured to measure the movement of the input unit 202 or the movement of objects in the environment of the input unit 202. Specifically, the motion sensor may perform at least one of: detecting the movement of objects in the environment of the input unit 202, measuring the velocity of objects in the environment of the input unit 202, measuring the acceleration of objects in the environment of the input unit 202, detecting the movement of the input unit 202, measuring the velocity of the input unit 202, measuring the acceleration of the input unit 202, etc. In some embodiments, the motion sensor 373 may include one or more accelerometers configured to detect changes in the appropriate acceleration and / or measure the appropriate acceleration of the input unit 202. In other embodiments, the motion sensor 373 may include one or more gyroscopes configured to detect changes in the orientation of the input unit 202 and / or measure information related to the orientation of the input unit 202. In other embodiments, the motion sensor 373 may include one or more using image sensors, LIDAR sensors, radar sensors, or proximity sensors. For example, by analyzing the captured images, the processing device can determine the movement of the input unit 202, e.g., using an egomotion algorithm. Additionally, the processing device can determine the movement of objects in the input unit 202's environment, e.g., using an object tracking algorithm. Consistent with this disclosure, the processing device 360 ​​can modify the presentation of virtual content based on the determined movement of the input unit 202 or the determined movement of objects in the input unit 202's environment, e.g., by causing a virtual display to follow the movement of the input unit 202.

[0060] The environmental sensors 374 may include one or more sensors of different types configured to capture data reflecting the environment of the input unit 202. In some embodiments, the environmental sensors 374 may include one or more chemical sensors configured to measure at least one of: measuring a chemical property in the environment of the input unit 202; measuring a change in a chemical property in the environment of the input unit 202; detecting the presence of a chemical in the environment of the input unit 202; or measuring the concentration of a chemical in the environment of the input unit 202. Examples of such chemical properties may include pH level, toxicity, and temperature. Examples of such chemicals may include electrolytes, certain enzymes, certain hormones, certain proteins, smoke, carbon dioxide, carbon monoxide, oxygen, ozone, hydrogen, and hydrogen sulfide. In other embodiments, the environmental sensors 374 may include one or more temperature sensors configured to detect a change in temperature of the environment of the input unit 202 and / or to measure the temperature of the environment of the input unit 202. In other embodiments, the environmental sensor 374 may include one or more barometers configured to detect changes in atmospheric pressure in the input unit 202's environment and / or measure the atmospheric pressure in the input unit 202's environment. In other embodiments, the environmental sensor 374 may include one or more light sensors configured to detect changes in ambient light in the input unit 202's environment. Consistent with the present disclosure, the processing device 360 ​​may modify the presentation of the virtual content based on input from the environmental sensor 374. For example, automatically reducing the brightness of the virtual content when the user 100's environment becomes dark.

[0061] The other sensors 375 may include weight sensors, optical sensors, resistive sensors, ultrasonic sensors, proximity sensors, biometric sensors, or other sensing devices to facilitate related functions. In particular embodiments, the other sensors 375 may include one or more positioning sensors configured to obtain positioning information of the input unit 202, detect changes in the position of the input unit 202, and / or measure the position of the input unit 202. Alternatively, the GPS software may enable the input unit 202 to access an external GPS receiver (e.g., connect via a serial port or Bluetooth). Consistent with the present disclosure, the processing device 360 ​​may modify the presentation of virtual content based on input from the other sensors 375. For example, presenting personal information only after identifying the user 100 using data from a biometric sensor.

[0062] The components and arrangement shown in FIG. 3 are not intended to limit any embodiment. As will be understood by those skilled in the art with the benefit of this disclosure, numerous variations and / or modifications may be made to the illustrated configuration of input unit 202. For example, not all components are essential to the operation of the input unit in all cases. Any component may be located in any suitable portion of the input unit, and components may be rearranged in various configurations while still providing the functionality of various embodiments. For example, some input units may not include all of the elements shown in input unit 202.

[0063] FIG. 4 is a block diagram of an exemplary configuration of the XR unit 204. It should be understood that FIG. 4 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. In the embodiment of FIG. 4, the XR unit 204 may have direct or indirect access to a bus 400 (or other communication mechanism) that interconnects subsystems and components for transferring information within the XR unit 204. For example, the bus 400 may interconnect a memory interface 410, a network interface 420, an input interface 430, a power supply 440, an output interface 450, a processing device 460, a sensor interface 470, and a database 480.

[0064] The memory interface 410 shown in FIG. 4 is assumed to have functionality similar to that of the memory interface 310 described in detail above. The memory interface 410 may be used to access software products and / or data stored on a non-transitory computer-readable medium or in a memory device, such as the memory device 411. The memory device 411 may include software modules for performing processes consistent with the present disclosure. In particular, the memory device 411 may include an input determination module 412, an output determination module 413, a sensor communication module 414, a virtual content determination module 415, a virtual content communication module 416, and a database access module 417. The modules 412-417 may include software instructions for execution by at least one processor (e.g., the processing device 460) associated with the XR unit 204. The input determination module 412, the output determination module 413, the sensor communication module 414, the virtual content determination module 415, the virtual content communication module 416, and the database access module 417 may cooperate to perform various operations. For example, the input determination module 412 can determine user interface (UI) input received from the input unit 202. Simultaneously, the sensor communication module 414 can receive data from various sensors to determine the status of the user 100. The virtual content determination module 415 can determine virtual content to display based on the received input and the determined status of the user 100. The virtual content communication module 416 can search for virtual content not determined by the virtual content determination module 415. The search for virtual content may be from the database 380, the database 480, the mobile communication device 206, or the remote processing unit 208. Based on the output of the virtual content determination module 415, the output determination module 413 can cause a change in the virtual content displayed to the user 100 by the projector 454.

[0065] In some embodiments, the input determination module 412 can coordinate operation of the input interface 430 to receive gesture input 431, virtual input 432, audio input 433, and UI input 434. Consistent with the present disclosure, the input determination module 412 can simultaneously receive different types of input data. In one embodiment, the input determination module 412 can apply various rules based on the type of input detected. For example, gesture input may be prioritized over virtual input. In some embodiments, the output determination module 413 can coordinate operation of the output interface 450 to generate output using a light indicator 451, a display 452, a speaker 453, and a projector 454. In one embodiment, the light indicator 451 may include a light indicator that indicates the status of the wearable extended reality device. For example, the light indicator may display a green light when the wearable extended reality device 110 is connected to the input unit 202 and flash when the battery of the wearable extended reality device 110 is low. In another embodiment, display 452 may be used to display operational information. In another embodiment, speaker 453 may include bone conduction headphones used to output audio to user 100. In another embodiment, projector 454 may present virtual content to user 100.

[0066] The operation of the sensor communication module, virtual content determination module, virtual content communication module, and database access module has been described above with reference to Figure 3, and the details thereof will not be repeated here. Modules 412-417 may be implemented in software, hardware, firmware, any mixture thereof, etc.

[0067] 4 is assumed to have functionality similar to that of the network interface 320 described in detail above. The specific design and implementation of the network interface 420 may depend on the communication network in which the XR unit 204 is intended to operate. For example, in some embodiments, the XR unit 204 is configured to be selectively connectable to the input unit 202 via a wire. When connected via a wire, the network interface 420 may enable communication with the input unit 202, and when not connected via a wire, the network interface 420 may enable communication with the mobile communication device 206.

[0068] 4 is assumed to have functionality similar to that of the input interface 330 described in detail above. In this case, the input interface 430 can communicate with an image sensor to obtain gesture input 431 (e.g., a finger of the user 100 pointing at a virtual object), communicate with other XR units 204 to obtain virtual input 432 (e.g., a virtual object shared with the XR unit 204, or a gesture of an avatar detected in the virtual environment), communicate with a microphone to obtain audio input 433 (e.g., a voice command), and communicate with the input unit 202 to obtain UI input 434 (e.g., virtual content determined by the virtual content determination module 315).

[0069] 4 is assumed to have functionality similar to that of the power supply 340 described above, and simply provides electrical energy to power the XR unit 204. In some embodiments, the power supply 440 may be charged by the power supply 340. For example, the power supply 440 may be changed wirelessly when the XR unit 204 is placed on or near the input unit 202.

[0070] The output interface 450 shown in FIG. 4 is assumed to have functionality similar to that of the output interface 350 described in detail above. In this case, the output interface 450 can provide output from a light indicator 451, a display 452, a speaker 453, and a projector 454. The projector 454 can be any device, apparatus, fixture, etc. that can project (or direct) light to display virtual content onto a surface. This surface can be part of the XR unit 204, part of the eye of the user 100, or part of an object in close proximity to the user 100. In one embodiment, the projector 454 can include a lighting unit that focuses light within a limited solid angle by one or more mirrors and lenses to provide a high value of luminous intensity in a defined direction.

[0071] 4 is assumed to have functionality similar to that of output processing device 360, described in detail above. When XR unit 204 is connected to input unit 202, processing device 460 can cooperate with processing device 360. In particular, processing device 460 can implement virtual machine technology or other technology to provide the capability to run, control, operate, manipulate, store, etc., multiple software processes, applications, programs, etc. It should be understood that other types of processor configurations can be implemented to provide the capabilities disclosed herein.

[0072] The sensor interface 470 shown in Figure 4 is assumed to have functionality similar to that of the sensor interface 370 described in detail above. Specifically, the sensor interface 470 can communicate with an audio sensor 471, an image sensor 472, a motion sensor 473, an environmental sensor 474, and other sensors 475. The operation of the audio, image, motion, environmental, and other sensors was described above with reference to Figure 3, and the details will not be repeated here. It should be understood that other types and combinations of sensors may be used to provide the capabilities disclosed herein.

[0073] The components and arrangement shown in FIG. 4 are not intended to limit any embodiment. As will be understood by one of ordinary skill in the art having the benefit of this disclosure, numerous variations and / or modifications may be made to the illustrated configuration of the XR unit 204. For example, not all components are essential to the operation of the XR unit 204 in all cases. Any component may be located in any suitable part of the system 200, and components may be rearranged in various configurations while still providing functionality of various embodiments. For example, some XR units may not include all of the elements in the XR unit 204 (e.g., the wearable extended reality device 110 may not have a light indicator 451).

[0074] FIG. 5 is a block diagram of an exemplary configuration of remote processing unit 208. It should be understood that FIG. 5 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. In the embodiment of FIG. 5, remote processing unit 208 may include server 210, directly or indirectly accessing bus 500 (or other communication mechanism) that interconnects subsystems and components for transferring information within server 210. For example, bus 500 may interconnect memory interface 510, network interface 520, power supply 540, processing device 560, and database 580. Remote processing unit 208 may also include one or more data structures, such as data structures 212A, 212B, and 212C.

[0075] The memory interface 510 shown in FIG. 5 is assumed to have functionality similar to that of the memory interface 310 described in detail above. The memory interface 510 may be used to access software products and / or data stored on a non-transitory computer-readable medium or in other memory devices, such as memory devices 311, 411, 511, or data structures 212A, 212B, and 212C. The memory device 511 may include software modules for performing processes consistent with the present disclosure. In particular, the memory device 511 may include a shared memory module 512, a node registration module 513, a load balancing module 514, one or more compute nodes 515, an internal communications module 516, an external communications module 517, and a database access module (not shown). The modules 512-517 may include software instructions for execution by at least one processor (e.g., processing device 560) associated with the remote processing unit 208. The shared memory module 512, the node registration module 513, the load balancing module 514, the computation module 515, and the external communication module 517 may cooperate to perform various operations.

[0076] The shared memory module 512 may enable information sharing between the remote processing unit 208 and other components of the system 200. In some embodiments, the shared memory module 512 may be configured to enable the processing device 560 (and other processing devices in the system 200) to access, retrieve, and store data. For example, using the shared memory module 512, the processing device 560 may at least one of execute software programs stored in the memory device 511, the database 580, or the data structures 212A-C, store information in the memory device 511, the database 580, or the data structures 212A-C, or retrieve information from the memory device 511, the database 580, or the data structures 212A-C.

[0077] The node registration module 513 may be configured to track the availability of one or more computational nodes 515. In some examples, the node registration module 513 may be implemented as a software program, such as a software program executed by one or more computational nodes 515, a hardware solution, or a combined software and hardware solution. In some implementations, the node registration module 513 may communicate with one or more computational nodes 515, for example, using the internal communication module 516. In some examples, one or more computational nodes 515 may inform the node registration module 513 of their status by sending a message, for example, upon startup, upon shutdown, at regular intervals, at selected times, in response to a query received from the node registration module 513, or at any other determined time. In some examples, the node registration module 513 may inquire about the status of one or more computational nodes 515 by sending a message, for example, upon startup, at regular intervals, at selected times, or at any other determined time.

[0078] The load balancing module 514 may be configured to divide the workload among one or more computing nodes 515. In some examples, the load balancing module 514 may be implemented as a software program, such as a software program executed by one or more of the computing nodes 515, a hardware solution, or a combined software and hardware solution. In some implementations, the load balancing module 514 may interact with the node registration module 513 to obtain information regarding the availability of one or more computing nodes 515. In some implementations, the load balancing module 514 may communicate with one or more computing nodes 515 using, for example, the internal communications module 516. In some examples, one or more computing nodes 515 may inform the load balancing module 514 of their status by sending a message, for example, upon startup, upon shutdown, at regular intervals, at selected times, in response to a query received from the load balancing module 514, or at any other determined time. In some examples, the load balancing module 514 may inquire about the status of one or more compute nodes 515 by sending a message, for example, upon startup, at regular intervals, at preselected times, or at any other determined time.

[0079] Internal communications module 516 may be configured to receive and / or transmit information from one or more components of remote processing unit 208. For example, control and / or synchronization signals may be transmitted and / or received through internal communications module 516. In one embodiment, input information for a computer program, output information for a computer program, and / or intermediate information for a computer program may be transmitted and / or received through internal communications module 516. In another embodiment, information received through internal communications module 516 may be stored in memory device 511, database 580, data structures 212A-C, or other memory devices within system 200. For example, information retrieved from data structure 212A may be transmitted using internal communications module 516. In another example, input data may be received using internal communications module 516 and stored in data structure 212B.

[0080] The external communication module 517 may be configured to receive and / or transmit information from one or more components of the system 200. For example, control signals may be transmitted and / or received through the external communication module 517. In one embodiment, information received through the external communication module 517 may be stored in the memory device 511, the database 580, the data structures 212A-C, and / or any memory device within the system 200. In another embodiment, information retrieved from any of the data structures 212A-C may be transmitted to the XR unit 204 using the external communication module 517. In another embodiment, input data may be transmitted and / or received using the external communication module 517. Examples of such input data may include data received from the input unit 202, information captured from the environment of the user 100 using one or more sensors (e.g., audio sensor 471, image sensor 472, motion sensor 473, environmental sensor 474, other sensors 475), etc.

[0081] In some embodiments, aspects of modules 512-517 may be implemented in hardware executable by one or more processors, in software (including one or more signal processing and / or application specific integrated circuits), in firmware, or any combination thereof, alone or in various combinations with one another. In particular, modules 512-517 may be configured to interact with each other and / or with other modules of system 200 to perform functions consistent with embodiments of the present disclosure. Memory device 511 may include additional modules and instructions or fewer modules and instructions.

[0082] 5 assume functionality similar to that of similar elements described above with reference to FIGS. 4 and 5. The specific design and implementation of the above-described components may vary based on the implementation of system 200. Additionally, remote processing unit 208 may include more or fewer components. For example, remote processing unit 208 may include an input interface configured to receive input directly from one or more input devices.

[0083] Consistent with the present disclosure, a processing device of system 200 (e.g., a processor in mobile communication device 206, a processor in server 210, a processor in a wearable extended reality device such as wearable extended reality device 110, and / or a processor in an input device associated with wearable extended reality device 110, such as keyboard 104) may use machine learning algorithms to implement any of the methods disclosed herein. In some embodiments, the machine learning algorithms (also referred to in this disclosure as machine learning models) may be trained using training examples, for example, as described below. Some non-limiting examples of such machine learning algorithms may include classification algorithms, data regression algorithms, image segmentation algorithms, visual detection algorithms (such as object detectors, face detectors, person detectors, motion detectors, edge detectors, etc.), visual recognition algorithms (such as face recognition, person recognition, object recognition, etc.), speech recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbor algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recurrent neural network algorithms, linear machine learning models, nonlinear machine learning models, ensemble algorithms, etc. For example, the trained machine learning algorithms may include predictive models, classification models, data regression models, clustering models, segmentation models, inference models such as artificial neural networks (such as deep neural networks, convolutional neural networks, recurrent neural networks), random forests, support vector machines, etc. In some examples, the training examples may include example inputs along with desired outputs corresponding to the example inputs. Further, in some examples, training a machine learning algorithm using training examples can produce a trained machine learning algorithm, which can be used to estimate outputs for inputs not included in the training examples.In some examples, engineers, scientists, processes, and machines that train machine learning algorithms may further use validation examples and / or test examples. For example, the validation examples and / or test examples may include example inputs along with desired outputs corresponding to the example inputs, and the trained machine learning algorithm and / or intermediate trained machine learning algorithm may be used to estimate outputs for the example inputs of the validation examples and / or test examples, and the estimated outputs may be compared with the corresponding desired outputs, and the trained machine learning algorithm and / or intermediate trained machine learning algorithm may be evaluated based on the results of the comparison. In some examples, the machine learning algorithm may have parameters and hyperparameters, and the hyperparameters may be set manually by a person or automatically by a process external to the machine learning algorithm (such as a hyperparameter search algorithm), and the parameters of the machine learning algorithm may be set by the machine learning algorithm based on training examples. In some implementations, the hyperparameters may be set based on training examples and validation examples, and the parameters may be set based on training examples and selected hyperparameters. For example, given hyperparameters, the parameters may be conditionally independent of the validation examples.

[0084] In some embodiments, a trained machine learning algorithm (also referred to in this disclosure as a machine learning model and a trained machine learning model) may be used to analyze inputs and generate outputs, for example, as described below. In some examples, a trained machine learning algorithm may be used as an inference model that generates an inferred output when provided with an input. For example, the trained machine learning algorithm may include a classification algorithm, where the input may include samples and the inferred output may include a classification of the samples (an inferred label, an inferred tag, etc.). In another example, the trained machine learning algorithm may include a regression model, where the input may include samples and the inferred output may include an inferred value corresponding to the samples. In yet another example, the trained machine learning algorithm may include a clustering model, where the input may include samples and the inferred output may include an assignment of the samples to at least one cluster. In an additional example, the trained machine learning algorithm may include a classification algorithm, where the input may include images and the inferred output may include a classification of an item shown in the image. In yet another example, the trained machine learning algorithm may include a regression model, where the input may include an image, and the inferred output may include an inferred value corresponding to an item shown in the image (such as an estimated property of the item, such as size, volume, age of a person shown in the image, distance from the item shown in the image, etc.). In an additional example, the trained machine learning algorithm may include an image segmentation model, where the input may include an image, and the inferred output may include a segmentation of the image. In yet another example, the trained machine learning algorithm may include an object detector, where the input may include an image, and the inferred output may include one or more detected objects in the image and / or one or more locations of the objects in the image.In some examples, the trained machine learning algorithm may include one or more formulas, and / or one or more functions, and / or one or more rules, and / or one or more procedures, the inputs may be used as inputs to the formulas and / or functions, and / or rules, and / or procedures, and the inferred output may be based on the output of the formulas and / or functions, and / or rules, and / or procedures (e.g., selecting one of the outputs of the formulas and / or functions, and / or rules, and / or procedures, using statistical measures of the output of the formulas and / or functions, and / or rules, and / or procedures, etc.).

[0085] Consistent with the present disclosure, the processing device of system 200 can analyze image data captured by an image sensor (e.g., image sensor 372, image sensor 472, or any other image sensor) to implement any of the methods disclosed herein. In some embodiments, analyzing the image data may include analyzing the image data to obtain preprocessed image data and then analyzing the image data and / or the preprocessed image data to obtain the desired result. Those skilled in the art will recognize that the following are examples and that other types of preprocessing methods may be used to preprocess the image data. In some examples, the image data may be preprocessed by transforming the image data using a transformation function to obtain transformed image data, and the preprocessed image data may include the transformed image data. For example, the transformed image data may include one or more convolutions of the image data. For example, the transformation function may include one or more image filters, such as a low-pass filter, a high-pass filter, a band-pass filter, an all-pass filter, etc. In some examples, the transformation function may include a nonlinear function. In some examples, the image data may be preprocessed by smoothing at least a portion of the image data, for example, using a Gaussian convolution, using a median filter, etc. In some examples, the image data may be preprocessed to obtain different representations of the image data. For example, the preprocessed image data may include a representation of at least a portion of the image data in the frequency domain, a discrete Fourier transform of at least a portion of the image data, a discrete wavelet transform of at least a portion of the image data, a time / frequency representation of at least a portion of the image data, a representation of at least a portion of the image data in a reduced dimension, a lossy representation of at least a portion of the image data, a lossless representation of at least a portion of the image data, a time series of any of the above, any combination of the above, etc. In some examples, the image data may be preprocessed to extract edges, and the preprocessed image data may include information based on and / or related to the extracted edges. In some examples, the image data may be preprocessed to extract image features from the image data.Some non-limiting examples of such image features may include information based on and / or related to edges, corners, blobs, ridges, Scale Invariant Feature Transform (SIFT) features, temporal features, etc. In some examples, analyzing the image data may include computing at least one convolution of at least a portion of the image data and using the computed at least one convolution to compute at least one resultant value and / or to determine, identify, recognize, classify, etc.

[0086] Consistent with another aspect of the present disclosure, the processing device of system 200 can analyze image data to implement any of the methods disclosed herein. In some embodiments, analyzing the image may include analyzing the image data and / or preprocessed image data using one or more rules, functions, procedures, artificial neural networks, object detection algorithms, face detection algorithms, visual event detection algorithms, action detection algorithms, motion detection algorithms, background subtraction algorithms, inference models, etc. Some non-limiting examples of such inference models may include manually pre-programmed inference models, classification models, regression models, results of training algorithms such as machine learning algorithms and / or deep learning algorithms on training examples, etc. The training examples may include example data instances, and in some cases, the data instances may be labeled with corresponding desired labels and / or results. In some embodiments, analyzing the image data (e.g., by the methods, steps, and modules described herein) may include analyzing pixels, voxels, point clouds, range data, etc. included in the image data.

[0087] Convolutions may include convolutions of any dimension. A one-dimensional convolution is a function that transforms an original sequence into a transformed sequence. A one-dimensional convolution may be defined by a sequence of scalars. Each particular value of the transformed sequence may be determined by calculating a linear combination of values ​​of subsequences of the original sequence that correspond to a particular value. The resulting value of the calculated convolution may include any value of the transformed sequence. Similarly, an n-dimensional convolution is a function that transforms an original n-dimensional array into a transformed array. An n-dimensional convolution may be defined by an n-dimensional array of scalars (known as the kernel of the n-dimensional convolution). Each particular value of the transformed array may be determined by calculating a linear combination of values ​​in an n-dimensional region of the original array that correspond to a particular value. The resulting value of the calculated convolution may include any value in the transformed array. In some examples, an image may include one or more components (such as a color component, a depth component, etc.), and each component may include a two-dimensional array of pixel values. In one example, calculating the convolution of an image may include calculating a two-dimensional convolution for one or more components of the image. In another example, computing the convolution of an image may include stacking arrays from various components to create a three-dimensional array and computing the three-dimensional convolution on the resulting three-dimensional array. In some examples, a video may include one or more components (such as a color component, a depth component, etc.), and each component may include a three-dimensional array of pixel values ​​(having two spatial axes and one temporal axis). In one example, computing the convolution of a video may include computing the three-dimensional convolution on one or more components of the video. In another example, computing the convolution of a video may include stacking arrays from various components to create a four-dimensional array and computing the four-dimensional convolution on the resulting four-dimensional array.

[0088] The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. While several exemplary embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or modifications may be made to the components shown in the drawings, and the exemplary methods described herein may be modified by substituting, rearranging, removing, or adding steps to the disclosed methods. Therefore, the following detailed description is not limited to the disclosed embodiments and examples, but includes the general principles described herein and illustrated in the figures, in addition to the general principles encompassed by the appended claims.

[0089] The present disclosure is directed to systems and methods for providing an extended reality environment to a user. The term "extended reality environment," which may also be referred to as "extended reality," "extended reality space," or "extended environment," refers to any type of combined real and virtual environment and human-machine interaction generated at least in part by computer technology. An extended reality environment may be a fully simulated virtual environment or a combined real and virtual environment that a user can perceive from various perspectives. In some examples, a user can interact with elements of the extended reality environment. One non-limiting example of an extended reality environment may be a virtual reality environment, also known as "virtual reality" or "virtual environment." An immersive virtual reality environment may be a simulated non-physical environment that provides a user with the perception of being in a virtual environment. Another non-limiting example of an extended reality environment may be an augmented reality environment, also known as "augmented reality" or "augmented environment." An augmented reality environment may include a live, direct or indirect view of a physical, real-world environment augmented with virtual, computer-generated sensory information, such as virtual objects with which a user can interact. Another non-limiting example of an extended reality environment is a mixed reality environment, also known as "mixed reality" or "composite environment." A mixed reality environment may be a hybrid of a physical, real-world and virtual environment, where physical and virtual objects coexist and may interact in real time. In some examples, both augmented reality and mixed reality environments may include a combination of real and virtual worlds, real-time interaction, and precise 3D registration of virtual and real objects. In some examples, both augmented reality and mixed reality environments may include structural, overlaid sensory information that may be added to the physical environment. In other examples, both augmented reality and mixed reality environments may include disruptive virtual content that may mask at least a portion of the physical environment.

[0090] In some embodiments, the present systems and methods can provide an extended reality environment using extended reality devices. The term extended reality device may include any type of device or system that allows a user to perceive and / or interact with an extended reality environment. An extended reality device may allow a user to perceive and / or interact with the extended reality environment through one or more sensory modalities. Some non-limiting examples of such sensory modalities may include vision, hearing, touch, somatosensation, and smell. One example of an extended reality device is a virtual reality device that allows a user to perceive and / or interact with a virtual reality environment. Another example of an extended reality device is an augmented reality device that allows a user to perceive and / or interact with an augmented reality environment. Yet another example of an extended reality device is a mixed reality device that allows a user to perceive and / or interact with a mixed reality environment.

[0091] Consistent with one aspect of the present disclosure, the extended reality device may be a wearable device, such as a head-mounted device, e.g., smart glasses, smart contact lenses, a headset, or any other device worn by a human for the purpose of presenting extended reality to the human. Other extended reality devices may include a holographic projector or any other device or system capable of providing augmented reality (AR), virtual reality (VR), mixed reality (MR), or any immersive experience. Typical components of a wearable extended reality device may include a stereoscopic head-mounted display, a stereoscopic head-mounted sound system, a head-motion tracking sensor (such as a gyroscope, accelerometer, magnetometer, image sensor, structured light sensor), a head-mounted projector, an eye-tracking sensor, and at least one of the additional components described below. Consistent with another aspect of the present disclosure, the extended reality device may be a non-wearable extended reality device. Specifically, the non-wearable extended reality device may include a multi-projection environment device. In some embodiments, the extended reality device may be configured to change the viewing perspective of the extended reality environment in response to user movement, and in particular in response to user head movement. In one example, a wearable extended reality device may change the field of view of the extended reality environment in response to a change in the user's head pose, such as by changing the spatial orientation without changing the user's spatial position within the extended reality environment. In another example, a non-wearable extended reality device may change the user's spatial position within the extended reality environment in response to a change in the user's position in the real world, for example, by changing the user's spatial position within the extended reality environment without changing the field of view orientation relative to the spatial position.

[0092] According to some embodiments, the extended reality equipment may include a digital communication device configured to at least one of receive virtual content data configured to enable presentation of virtual content, transmit virtual content for sharing with at least one external device, receive contextual data from the at least one external device, transmit contextual data to the at least one external device, transmit usage data indicative of use of the extended reality equipment, and transmit data based on information captured using at least one sensor included in the extended reality equipment. In additional embodiments, the extended reality equipment may include a memory for storing at least one of virtual data configured to enable presentation of virtual content, contextual data, usage data indicative of use of the extended reality equipment, sensor data based on information captured using at least one sensor included in the wearable extended reality equipment, software instructions configured to cause a processing device to present the virtual content, software instructions configured to cause the processing device to collect and analyze the contextual data, software instructions configured to cause the processing device to collect and analyze the usage data, and software instructions configured to cause the processing device to collect and analyze the sensor data. In additional embodiments, the extended reality equipment may include a processing device configured to at least one of render virtual content, collect and analyze contextual data, collect and analyze usage data, and collect and analyze sensor data. In additional embodiments, the extended reality equipment may include one or more sensors.The one or more sensors may include one or more image sensors (e.g., configured to capture images and / or video of a user of the device or the user's environment), one or more motion sensors (such as accelerometers, gyroscopes, magnetometers), one or more positioning sensors (such as GPS, outdoor positioning sensors, indoor positioning sensors), one or more temperature sensors (e.g., configured to measure the temperature of at least a portion of the device and / or environment), one or more contact sensors, one or more proximity sensors (e.g., configured to detect whether the device is currently being worn), one or more electrical impedance sensors (e.g., configured to measure the user's electrical impedance), one or more eye tracking sensors such as a gaze detector, an optical tracker, an electric potential tracker (e.g., an electro-oculogram (EOG) sensor), a video-based eye tracker, an infrared / near-infrared sensor, a passive optical sensor, or any other technology capable of determining where a human is looking or gazing.

[0093] In some embodiments, the present systems and methods can interact with the extended reality equipment using input devices. The term input device may include any physical device configured to receive input from a user or the user's environment and provide data to a computing device. The data provided to the computing device may be in digital and / or analog format. In one embodiment, the input device can store input received from a user in a memory device accessible by the processing device, and the processing device can access the stored data for analysis. In another embodiment, the input device can provide data directly to the processing device, for example, via a bus or another communication system configured to transfer data from the input device to the processing device. In some examples, input received by the input device may include key presses, tactile input data, movement data, position data, gesture-based input data, directional data, or any other data to feed into a computation. Some examples of input devices may include buttons, keys, keyboards, computer mice, touchpads, touchscreens, joysticks, or other mechanisms by which input may be received. Another example of an input device may include an integrated computing interface device that includes at least one physical component for receiving input from a user. The integrated computing interface device may include at least a memory, a processing device, and at least one physical component for receiving input from a user. In one example, the integrated computing interface device may further include a digital network interface that enables digital communication with other computing devices. In one example, the integrated computing interface device may further include a physical component for outputting information to a user. In some examples, all components of the integrated computing interface device may be contained in a single housing, while in other examples, the components may be distributed among two or more housings.Some non-limiting examples of physical components for receiving input from a user that may be included in an integrated computing interface device may include at least one of buttons, keys, a keyboard, a touchpad, a touchscreen, a joystick, or any other mechanism or sensor from which computational information may be received. Some non-limiting examples of physical components for outputting information to a user may include at least one of a light indicator (such as an LED indicator), a screen, a touchscreen, a beeper, an audio speaker, or any other audio, video, or tactile device that provides a human-perceptible output.

[0094] In some embodiments, the image data may be captured using one or more image sensors. In some examples, the image sensors may be included in an extended reality device, a wearable device, a wearable extended reality device, an input device, the user's environment, etc. In some examples, the image data may be read from memory, received from an external device, generated (e.g., using a generative model), etc. Some non-limiting examples of image data may include images, grayscale images, color images, 2D images, 3D images, videos, 2D videos, 3D videos, frames, scenes, data derived from other image data, etc. In some examples, the image data may be encoded in any analog or digital format. Some non-limiting examples of such formats may include raw formats, compressed formats, uncompressed formats, lossy formats, lossless formats, JPEG, GIF, PNG, TIFF, BMP, NTSC, PAL, SECAM, MPEG, MPEG-4 Part 14, MOV, WMV, FLV, AVI, AVCHD, WebM, MKV, etc.

[0095] In some embodiments, the extended reality device may receive a digital signal, for example, from an input device. The term digital signal may refer to a series of digital values ​​that are discrete in time. The digital signal may represent, for example, sensor data, text data, audio data, video data, virtual data, or any other form of data that provides perceptible information. Consistent with the present disclosure, the digital signal may be configured to cause the extended reality device to present virtual content. In one embodiment, the virtual content may be presented in a selected orientation. In this embodiment, the digital signal may indicate the position and angle of a viewpoint in an environment, such as an extended reality environment. Specifically, the digital signal may include encoding of position and angle in six degrees of freedom (e.g., forward / backward, up / down, left / right, yaw, pitch, and roll). In another embodiment, the digital signal may include encoding of position as three-dimensional coordinates (e.g., x, y, and z) and encoding of angle as a vector resulting from the encoded position. Specifically, the digital signal may indicate the orientation and angle of the presented virtual content in absolute coordinates of the environment, for example, by encoding the yaw, pitch, and roll of the virtual content relative to a standard default angle. In another embodiment, the digital signal may indicate the orientation and angle of the presented virtual content relative to the viewpoint of another object (e.g., a virtual object, a physical object, etc.), for example, by encoding the yaw, pitch, and roll of the virtual content relative to a direction corresponding to the viewpoint or a direction corresponding to another object. In another embodiment, such a digital signal may include one or more projections of the virtual content, for example, in a presentation-ready format (e.g., an image, a video, etc.). For example, each such projection may correspond to a particular orientation or a particular angle. In another embodiment, the digital signal may include a representation of the virtual content, for example, by encoding the object as a three-dimensional array of voxels, a polygonal mesh, or any other format in which the virtual content may be presented.

[0096] In some embodiments, the digital signal may be configured to cause the extended reality device to present virtual content. The term virtual content may include any type of data representation that can be displayed to a user by the extended reality device. Virtual content may include virtual objects, non-animated virtual content, animated virtual content configured to change over time or in response to a trigger, virtual two-dimensional content, virtual three-dimensional content, virtual overlays on portions of a physical environment or on physical objects, virtual additions to a physical environment or object, virtual promotional content, virtual representations of physical objects, virtual representations of a physical environment, virtual documents, virtual characters or personas, virtual computer screens, virtual widgets, or any other format for virtually displaying information. Consistent with this disclosure, virtual content may include any visual presentation rendered by a computer or processing device. In one embodiment, virtual content may include virtual objects, which are visual presentations rendered by a computer within a limited area and configured to represent a particular type of object (such as a non-animated virtual object, an animated virtual object, a virtual fixture, a virtual decorative object, a virtual widget, or other virtual representation). The rendered visual presentation may be changed to reflect changes to a status object or a change in the viewing angle of the object, for example, to mimic a change in the appearance of a physical object. In another embodiment, the virtual content may include a virtual display (also referred to herein as a "virtual display screen" or "virtual screen"), such as a virtual computer screen, a virtual tablet screen, or a virtual smartphone screen, configured to display information generated by an operating system, where the operating system may be configured to receive text data from a physical keyboard and / or a virtual keyboard and cause the text content to be displayed within the virtual display screen. In one example shown in FIG. 1, the virtual content may include a virtual environment including a virtual computer screen and a plurality of virtual objects.In some examples, the virtual display may be a virtual object that mimics and / or extends the functionality of a physical display screen. For example, the virtual display may be presented within an extended reality environment (such as a mixed reality environment, an augmented reality environment, or a virtual reality environment) using an extended reality device. In one example, the virtual display may present content generated by a conventional operating system that may be presented equivalently on a physical display screen. In one example, text content entered using a keyboard (e.g., using a physical keyboard, using a virtual keyboard, etc.) may be presented on the virtual display in real time as the text content is typed. In one example, a virtual cursor may be presented on the virtual display, and the virtual cursor may be controlled by a pointing device (such as a physical pointing device, a virtual pointing device, a computer mouse, a joystick, a touchpad, or a physical touch controller). In one example, one or more windows of a graphical user interface operating system may be presented on the virtual display. In another example, the content presented on the virtual display may be interactive, i.e., may change in response to a user's actions. In yet another example, the presentation of the virtual display may or may not include the presentation of a screen frame.

[0097] Some disclosed embodiments may include and / or access a data structure or database. The terms data structure and database consistent with this disclosure may include any collection of data values ​​and the relationships between them. Data may be stored linearly, horizontally, hierarchically, relationally, non-relationally, unidimensionally, multidimensionally, operationally, in an ordered manner, in an unordered manner, in an object-oriented manner, in a centralized manner, in a decentralized manner, in a distributed manner, in a custom manner, or in any manner that enables data access. As non-limiting examples, data structures may include arrays, associative arrays, linked lists, binary trees, balanced trees, heaps, stacks, queues, sets, hash tables, records, tagged unions, entity-relationship models, graphs, hypergraphs, matrices, tensors, etc. For example, the data structure may include an XML database, an RDBMS database, an SQL database, or a NoSQL alternative for data storage / retrieval, such as, for example, MongoDB, Redis, Couchbase, Datastax Enterprise Graph, Elastic Search, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, and Neo4J. The data structure may be a component of the disclosed system or a remote computing component (e.g., a cloud-based data structure). The data in the data structure may be stored in contiguous or non-contiguous memory. Furthermore, the data structure does not require the information to be co-located. It may be distributed, for example, across multiple servers, which may be owned or operated by the same or different entities. Thus, the term "data structure" in the singular includes multiple data structures.

[0098] In some embodiments, the system can determine a confidence level in a received input or any determined value. The term confidence level refers to any indication, numerical value, or otherwise, of a level (e.g., within a predetermined range) that indicates the amount of confidence the system has in the determined data. For example, the confidence level may have a value between 1 and 10. Alternatively, the confidence level may be expressed as a percentage or any other numerical or non-numerical indication. In some cases, the system may compare the confidence level to a threshold. The term threshold may refer to a reference value, level, point, or range of values. In operation, when the confidence level of the determined data exceeds (or falls below, depending on the particular use case) the threshold, the system can follow a first course of action, and when the confidence level falls below (or exceeds, depending on the particular use case), the system can follow a second course of action. The value of the threshold may be predetermined for each type of object being inspected or may be dynamically selected based on various considerations.

[0099] Referring now to FIG. 1 , a user using an exemplary extended reality system consistent with embodiments of the present disclosure is shown; it should be understood that FIG. 1 is merely an exemplary representation of one embodiment, and that some depicted elements may be omitted and other elements may be added within the scope of the present disclosure. As shown, a user 100 is seated behind a table 102 supporting a keyboard 104 and a mouse 106. The keyboard 104 is connected by wires 108 to a wearable extended reality device 110 that displays virtual content to the user 100. As an alternative to, or in addition to, the wires 108, the keyboard 104 may be wirelessly connected to the wearable extended reality device 110. For purposes of illustration, the wearable extended reality device is shown as a pair of smart glasses; however, as noted above, the wearable extended reality device 110 may be any type of head-mounted device used to present extended reality to the user 100. The virtual content displayed by the wearable extended reality device 110 includes a virtual screen 112 (also referred to herein as a “virtual display screen” or “virtual display”) and a plurality of virtual widgets 114. Virtual widgets 114A-114D are displayed adjacent to the virtual screen 112, and virtual widget 114E is displayed on the table 102. The user 100 can use the keyboard 104 to input text into a document 116 displayed on the virtual screen 112 and can use the mouse 106 to control a virtual cursor 118. In one example, the virtual cursor 118 can move anywhere within the virtual screen 112. In another example, the virtual cursor 118 can move anywhere within the virtual screen 112 and can also move to any one of the virtual widgets 114A-114D, but cannot move to the virtual widget 114E. In yet another example, the virtual cursor 118 can move anywhere within the virtual screen 112 and can also move to any one of the virtual widgets 114A-114E.In an additional example, the virtual cursor 118 can move anywhere within the extended reality environment, including the virtual screen 112 and the virtual widgets 114A-114E. In yet another example, the virtual cursor can move over all available surfaces (i.e., virtual or physical surfaces) or only over selected surfaces within the extended reality environment. Alternatively, or in addition, the user 100 can interact with any one of the virtual widgets 114A-114E, or with a selected virtual widget, using hand gestures recognized by the wearable extended reality device 110. For example, the virtual widget 114E may be an interactive widget (e.g., a virtual slider controller) that can be manipulated with hand gestures.

[0100] FIG. 2 illustrates an example of a system 200 for providing an extended reality (XR) experience to a user, such as user 100. It should be understood that FIG. 2 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of this disclosure. System 200 may be computer-based and may include computer system components, wearable devices, workstations, tablets, handheld computing devices, memory devices, and / or an internal network connecting the components. System 200 may include or be connected to various network computing resources (e.g., servers, routers, switches, network connections, storage devices, etc.) to support the services provided by system 200. Consistent with this disclosure, system 200 may include an input unit 202, an XR unit 204, a mobile communication device 206, and a remote processing unit 208. Remote processing unit 208 may include a server 210 coupled to one or more physical or virtual storage devices, such as a data structure 212. The system 200 may also include or be connected to a communications network 214 that facilitates communication and data exchange between various system components and various entities associated with the system 200 .

[0101] Consistent with the present disclosure, the input unit 202 may include one or more devices capable of receiving input from the user 100. In one embodiment, the input unit 202 may include a text input device such as the keyboard 104. The text input device may include all possible types of devices and mechanisms for inputting text information into the system 200. Examples of text input devices may include a mechanical keyboard, a membrane keyboard, a flexible keyboard, a QWERTY keyboard, a Dvorak keyboard, a Colemak keyboard, a corded keyboard, a wireless keyboard, a keypad, a key-based control panel, or other arrangement of control keys, a visual input device, or any other mechanism for inputting text, whether the mechanism is provided in physical form or presented virtually. In one embodiment, the input unit 202 may also include a pointing input device such as the mouse 106. The pointing input device may include all possible types of devices and mechanisms for inputting two-dimensional or three-dimensional information into the system 200. In one example, two-dimensional input from the pointing input device may be used to interact with virtual content presented via the XR unit 204. Examples of pointing input devices may include a computer mouse, trackball, touchpad, trackpad, touchscreen, joystick, pointing stick, stylus, light pen, or any other physical or virtual input mechanism. In one embodiment, the input unit 202 may also include a graphical input device such as a touchscreen configured to detect contact, movement, or interruption of movement. The graphical input device may use any of a number of touch-sensitivity technologies, including, but not limited to, capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more points of contact. In one embodiment, the input unit 202 may also include one or more audio input devices, such as a microphone.The audio input device may include all possible types of devices and mechanisms for inputting audio data to facilitate voice-enabled functions such as voice recognition, audio replication, digital recording, and telephone functions. In one embodiment, the input unit 202 may also include one or more image input devices, such as image sensors, configured to capture image data. In one embodiment, the input unit 202 may also include one or more haptic gloves configured to capture hand movement and posture data. In one embodiment, the input unit 202 may also include one or more proximity sensors configured to detect the presence and / or movement of objects within a selected area near the sensors.

[0102] According to some embodiments, the system may include at least one sensor configured to detect and / or measure characteristics associated with a user, the user's actions, or the user's environment. One example of the at least one sensor is the sensor 216 included in the input unit 202. The sensor 216 may be a motion sensor, a touch sensor, a light sensor, an infrared sensor, an audio sensor, an image sensor, a proximity sensor, a positioning sensor, a gyroscope, a temperature sensor, a biometric sensor, or any other sensing device for facilitating related functions. The sensor 216 may be integrated with or connected to the input device, or may be separate from the input device. In one example, a thermometer may be included in the mouse 106 to measure the user's 100 body temperature. In another example, a positioning sensor may be integrated with the keyboard 104 to determine the user's 100 movement relative to the keyboard 104. Such a positioning sensor may be implemented using one of the following technologies: Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Galileo Global Navigation System, BeiDou Navigation System, other Global Navigation Satellite Systems (GNSS), Indian Regional Navigation Satellite System (IRNSS), Local Positioning System (LPS), Real-Time Location System (RTLS), Indoor Positioning System (IPS), Wi-Fi based positioning system, cellular triangulation, image-based positioning technology, indoor positioning technology, outdoor positioning technology, or any other positioning technology.

[0103] According to some embodiments, the system may include one or more sensors for identifying the position and / or movement of a physical device (e.g., a physical input device, a physical computing device, a keyboard 104, a mouse 106, a wearable extended reality device 110, etc.). The one or more sensors may be included in the physical device or may be external to the physical device. In some examples, an image sensor external to the physical device (e.g., an image sensor included in another physical device) may be used to capture image data of the physical device, and the image data may be analyzed to identify the position and / or movement of the physical device. For example, the image data may be analyzed using a visual object tracking algorithm to identify movement of the physical device, a visual object detection algorithm to identify the position of the physical device (e.g., position relative to the image sensor, position in a global coordinate system, etc.), etc. In some examples, an image sensor included in the physical device may be used to capture image data, and the image data may be analyzed to identify the position and / or movement of the physical device. For example, the image data may be analyzed using a visual odometry algorithm to identify the position of the physical device, an egomotion algorithm to identify movement of the physical device, etc. In some examples, a positioning sensor, such as an indoor positioning sensor or an outdoor positioning sensor, may be included in a physical device and used to determine the location of the physical device. In some examples, a motion sensor, such as an accelerometer or gyroscope, may be included in a physical device and used to determine movement of the physical device. In some examples, a physical device, such as a keyboard or mouse, may be configured to be positioned on a physical surface. Such a physical device may include an optical mouse sensor (also known as a non-mechanical tracking engine) pointed at the physical surface, and the output of the optical mouse sensor may be analyzed to determine movement of the physical device relative to the physical surface.

[0104] Consistent with this disclosure, the XR unit 204 may include a wearable extended reality device configured to present virtual content to the user 100. One example of a wearable extended reality device is the wearable extended reality device 110. Additional examples of wearable extended reality devices may include virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, or any other device capable of generating extended reality content. Some non-limiting examples of such devices may include Nreal Light, Magic Leap One, Varjo, Quest 1 / 2, Vive, etc. In some embodiments, the XR unit 204 may present virtual content to the user 100. In general, extended reality devices may include all real-virtual composite environments and human-machine interactions generated by computer technology and wearables. As discussed above, the term “extended reality” (XR) refers to a superset that includes all real-virtual and virtual-virtual environments. This includes representative forms such as augmented reality (AR), mixed reality (MR), and virtual reality (VR), as well as areas interpolated therebetween. Accordingly, it should be noted that the terms "XR device," "AR device," "VR device," and "MR device" may be used interchangeably herein and may refer to any device from the various devices listed above.

[0105] Consistent with the present disclosure, the system can exchange data with various communication devices associated with a user, such as a mobile communication device 206. The term “communication device” is intended to include all possible types of devices capable of exchanging data using a digital communication network, an analog communication network, or any other communication network configured to transmit data. In some examples, the communication device may include a smartphone, a tablet, a smartwatch, a personal digital assistant, a desktop computer, a laptop computer, an IoT device, a dedicated terminal, a wearable communication device, and any other device that enables data communication. In some cases, the mobile communication device 206 can complement or replace the input unit 202. Specifically, the mobile communication device 206 may be associated with a physical touch controller that can function as a pointing input device. Furthermore, the mobile communication device 206 may also be used, for example, to implement a virtual keyboard and replace a text input device. For example, if the user 100 leaves the table 102 and walks to the break room with their smart glasses, the user may receive an email that requires a quick response. In this case, the user can choose to use their smartwatch as an input device and type a response to the email while it is virtually presented by the smartglasses.

[0106] Consistent with the present disclosure, embodiments of the present system may include the use of a cloud server. The term “cloud server” refers to a computer platform that provides services over a network, such as the Internet. In the exemplary embodiment shown in FIG. 2, the server 210 may use a virtual machine that does not need to correspond to individual hardware. For example, computing and / or storage capabilities may be implemented by allocating appropriate portions of desired computing / storage capabilities from a scalable repository, such as a data center or a distributed computing environment. Specifically, in one embodiment, the remote processing unit 208 may be used in conjunction with the XR unit 204 to provide virtual content to the user 100. In one exemplary configuration, the server 210 may be a cloud server that functions as an operating system (OS) for a wearable extended reality device. In one example, the server 210 may implement the methods described herein using customized hardwired logic, one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), firmware, and / or program logic that, in combination with a computer system, makes the server 210 a dedicated machine.

[0107] In some embodiments, server 210 can access data structure 212 to, for example, determine virtual content to display to user 100. Data structure 212 can utilize volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, or other types of storage devices or tangible or non-transitory computer-readable media, or any medium or mechanism for storing information. Data structure 212 may be part of server 210, as shown, or may be separate from server 210. If data structure 212 is not part of server 210, server 210 can exchange data with data structure 212 via a communications link. Data structure 212 may include one or more memory devices that store data and instructions used to execute one or more features of the disclosed methods. In one embodiment, data structure 212 may include any of several suitable data structures, ranging from small data structures hosted on a workstation to large data structures distributed across a data center. Data structure 212 may also include any combination of one or more data structures controlled by a memory controller device (e.g., a server) or software.

[0108] Consistent with this disclosure, a communications network may be any type of network (including infrastructure) that supports communications, exchanges information, and / or facilitates the exchange of information between components of a system. For example, communications network 214 in system 200 may include, for example, a telephone network, an extranet, an intranet, the Internet, satellite communications, offline communications, radio communications, transponder communications, a local area network (LAN), a wireless network (e.g., a Wi-Fi / 302.11 network), a wide area network (WAN), a virtual private network (VPN), a digital communications network, an analog communications network, or any other mechanism or combination of mechanisms that enables data transmission.

[0109] The components and arrangement of system 200 shown in FIG. 2 are intended to be illustrative only and not limiting of the disclosed embodiments, as the system components used to implement the disclosed processes and features may vary.

[0110] FIG. 3 is a block diagram of an exemplary configuration of input unit 202. It should be understood that FIG. 3 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. In the embodiment of FIG. 3, input unit 202 may have direct or indirect access to a bus 300 (or other communication mechanism) that interconnects subsystems and components for transferring information within input unit 202. For example, bus 300 may interconnect a memory interface 310, a network interface 320, an input interface 330, a power supply 340, an output interface 350, a processing device 360, a sensor interface 370, and a database 380.

[0111] The memory interface 310 shown in FIG. 3 can be used to access software products and / or data stored on a non-transitory computer-readable medium. Generally, a non-transitory computer-readable storage medium refers to any type of physical memory capable of storing information or data readable by at least one processor. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD-ROMs, DVDs, flash drives, disks, any other optical data storage media, any physical media with a pattern of holes, PROMs, EPROMs, flash EPROMs or any other flash memory, NVRAM, cache, registers, any other memory chips or cartridges, and networked versions thereof. The terms "memory" and "computer-readable storage medium" can refer to multiple structures, such as multiple memories or computer-readable storage media located within an input unit or at a remote location. In addition, one or more computer-readable storage media can be utilized in practicing a computer-implemented method. Thus, the term computer-readable storage medium should be understood to include tangible items and exclude carrier waves and transient signals. 3, memory interface 310 may be used to access software products and / or data stored in a memory device, such as memory device 311. Memory device 311 may include high-speed random access memory and / or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and / or flash memory (e.g., NAND, NOR). Consistent with the present disclosure, components of memory device 311 may be distributed across two or more units and / or two or more memory devices of system 200.

[0112] The memory device 311 shown in FIG. 3 may include software modules for executing processes consistent with the present disclosure. In particular, the memory device 311 may include an input determination module 312, an output determination module 313, a sensor communication module 314, a virtual content determination module 315, a virtual content communication module 316, and a database access module 317. Modules 312-317 may include software instructions for execution by at least one processor (e.g., processing device 360) associated with the input unit 202. The input determination module 312, the output determination module 313, the sensor communication module 314, the virtual content determination module 315, the virtual content communication module 316, and the database access module 317 may cooperate to perform various operations. For example, the input determination module 312 may determine text using data received, for example, from the keyboard 104. The output determination module 313 may then cause the recently entered text to be presented, for example, on a dedicated display 352 physically or wirelessly coupled to the keyboard 104. In this way, as the user 100 types, they can see a preview of the typed text without having to constantly move their head up and down to look at the virtual screen 112. The sensor communication module 314 can receive data from various sensors to determine the status of the user 100. The virtual content determination module 315 can then determine virtual content to display based on the received input and the determined status of the user 100. For example, the determined virtual content may be a virtual presentation of recently entered text on a virtual screen virtually positioned adjacent to the keyboard 104. The virtual content communication module 316 can retrieve virtual content not determined by the virtual content determination module 315 (e.g., another user's avatar). The retrieval of virtual content may be from the database 380, from the remote processing unit 208, or from any other source.

[0113] In some embodiments, the input determination module 312 can adjust the operation of the input interface 330 to receive pointer input 331, text input 332, audio input 333, and XR-related input 334. Pointer input, text input, and audio input are described in more detail above. The term “XR-related input” may include any type of data that may cause a change in virtual content displayed to the user 100. In one embodiment, the XR-related input 334 may include image data of the user 100, a wearable extended reality device (e.g., a detected hand gesture of the user 100), or a wireless communication indicating the presence of another user in proximity to the user 100. Consistent with the present disclosure, the input determination module 312 can simultaneously receive different types of input data. The input determination module 312 can then further apply various rules based on the type of input detected. For example, pointer input may be prioritized over voice input.

[0114] In some embodiments, the output determination module 313 can coordinate operation of the output interface 350 to generate output using light indicators 351, display 352, and / or speaker 353. Generally, the output generated by the output determination module 313 does not include virtual content presented by the wearable extended reality device. Instead, the output generated by the output determination module 313 includes various outputs related to operation of the input unit 202 and / or operation of the XR unit 204. In one embodiment, the light indicator 351 may include a light indicator that indicates the status of the wearable extended reality device. For example, the light indicator may display a green light when the wearable extended reality device 110 is connected to the keyboard 104 and flash when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 352 may be used to display operational information. For example, the display may present an error message when the wearable extended reality device is inoperable. In another embodiment, the speaker 353 may be used to output sound, for example, when the user 100 wants to play some music for other users.

[0115] In some embodiments, the sensor communications module 314 can coordinate the operation of the sensor interface 370 to receive sensor data from one or more sensors integrated with or connected to the input device. The one or more sensors may include an audio sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374 (e.g., a temperature sensor, an ambient light detector, etc.), and other sensors 375. In one embodiment, the data received from the sensor communications module 314 can be used to determine the physical orientation of the input device. The physical orientation of the input device can indicate the user's state and can be determined based on a combination of tilt motion, roll motion, and lateral motion. The physical orientation of the input device can then be used by the virtual content determination module 315 to modify the display parameters of the virtual content to match the user's state (e.g., alert, sleepy, active, sitting, standing, leaning back, leaning forward, walking, moving, riding, etc.).

[0116] In some embodiments, the virtual content determination module 315 can determine virtual content to be displayed by the wearable extended reality device. The virtual content can be determined based on data from the input determination module 312, the sensor communication module 314, and other sources (e.g., database 380). In some embodiments, determining the virtual content can include determining the distance, size, and orientation of a virtual object. The position of the virtual object can be determined based on the type of the virtual object. Specifically, with respect to the example shown in FIG. 1 , the virtual content determination module 315 can determine that the four virtual widgets 114A-114D should be placed on both sides of the virtual screen 112 and that the virtual widget 114E should be placed on the table 102 because the virtual widget 114E is a virtual controller (e.g., a volume bar). The position of the virtual object can be further determined based on a user's preference. For example, for a left-handed user, the virtual content determination module 315 may determine to place the virtual volume bar to the left of the keyboard 104, and for a right-handed user, the virtual content determination module 315 may determine to place the virtual volume bar to the right of the keyboard 104.

[0117] In some embodiments, the virtual content communication module 316 can coordinate the operation of the network interface 320 to obtain data from one or more sources to be presented as virtual content to the user 100. The one or more sources may include other XR units 204, the user's mobile communication device 206, the remote processing unit 208, publicly available information, etc. In one embodiment, the virtual content communication module 316 can communicate with the mobile communication device 206 to provide a virtual representation of the mobile communication device 206. For example, the virtual representation may allow the user 100 to read messages and interact with applications installed on the mobile communication device 206. The virtual content communication module 316 can also coordinate the operation of the network interface 320 to share virtual content with other users. In one example, the virtual content communication module 316 can use data from the input determination module to identify triggers (e.g., the triggers may include user gestures) and transfer content from a virtual display to a physical display (e.g., a television) or to a different user's virtual display.

[0118] In some embodiments, the database access module 317 can cooperate with the database 380 to retrieve stored data. The retrieved data may include, for example, privacy levels associated with various virtual objects, relationships between virtual and physical objects, user preferences, the user's past behavior, etc. As described above, the virtual content determination module 315 can determine virtual content using data stored in the database 380. The database 380 may include separate databases, including, for example, a vector database, a raster database, a tile database, a viewport database, and / or a user input database. The data stored in the database 380 may be received from modules 314-317 or other components of the system 200. Additionally, the data stored in the database 380 may be provided as input using data entry, data transfer, or data upload.

[0119] The modules 312-317 may be implemented in software, hardware, firmware, any combination thereof, etc. In some embodiments, data associated with any one or more of the modules 312-317 and the database 380 may be stored in the XR unit 204, the mobile communication device 206, or the remote processing unit 208. A processing device of the system 200 may be configured to execute instructions of the modules 312-317. In some embodiments, aspects of the modules 312-317 may be implemented in hardware, software (including one or more signal processing and / or application specific integrated circuits), firmware, or any combination thereof, executable by one or more processors, alone or in various combinations with each other. In particular, the modules 312-317 may be configured to interact with each other and / or with other modules of the system 200 to perform functions consistent with disclosed embodiments. For example, the input unit 202 may execute instructions including image processing algorithms on data from the XR unit 204 to determine head movement of the user 100. Furthermore, each function described throughout this specification with respect to input unit 202 or with respect to components of input unit 202 may correspond to a set of instructions for performing said function. These instructions need not be implemented as a separate software program, procedure, or module. Memory device 311 may include additional or fewer modules and instructions. For example, memory device 311 may store an operating system such as ANDROID, iOS, UNIX, OSX, WINDOWS, DARWIN, RTXC, LINUX, or an embedded operating system such as VXWorkS. The operating system may include instructions for handling basic system services and performing hardware-dependent tasks.

[0120] The network interface 320 shown in FIG. 3 can provide bidirectional data communication to a network, such as the communications network 214. In one embodiment, the network interface 320 may include an Integrated Services Digital Network (ISDN) card, a cellular modem, a satellite modem, or a modem for providing a data communication connection via the Internet. As another example, the network interface 320 may include a Wireless Local Area Network (WLAN) card. In another embodiment, the network interface 320 may include an Ethernet port connected to a radio frequency receiver and transmitter and / or an optical (e.g., infrared) receiver and transmitter. The specific design and implementation of the network interface 320 may depend on the communications network or networks over which the input unit 202 is intended to operate. For example, in some embodiments, the input unit 202 may include a network interface 320 designed to operate on a GSM network, a GPRS network, an EDGE network, a Wi-Fi or WiMax network, and a Bluetooth network. In any such implementation, the network interface 320 may be configured to transmit and receive electrical, electromagnetic, or optical signals that carry digital data streams or signals representing various types of information.

[0121] The input interface 330 shown in FIG. 3 can receive input from a variety of input devices, such as a keyboard, a mouse, a touchpad, a touchscreen, one or more buttons, a joystick, a microphone, an image sensor, and any other device configured to detect physical or virtual input. The received input can be in the form of at least one of text, sound, speech, hand gestures, body gestures, tactile information, and any other type of physical or virtual input generated by a user. In the illustrated embodiment, the input interface 330 can receive pointer input 331, text input 332, audio input 333, and XR-related input 334. In additional embodiments, the input interface 330 can be an integrated circuit that can function as a bridge between the processing device 360 ​​and any of the input devices listed above.

[0122] The power source 340 shown in FIG. 3 provides electrical energy to the input unit 202 and can optionally also power the XR unit 204. In general, the power source included in any device or system in this disclosure may be any device that can repeatedly store, distribute, or transmit electrical power, including, but not limited to, one or more batteries (e.g., lead-acid batteries, lithium-ion batteries, nickel-metal hydride batteries, nickel-cadmium batteries), one or more capacitors, one or more connections to external power sources, one or more power converters, or any combination thereof. Referring to the example shown in FIG. 3 , the power source may be portable, meaning that the input unit 202 can be easily carried by hand (e.g., the total weight of the power source 340 may be less than 1 pound). The mobility of the power source allows the user 100 to use the input unit 202 in a variety of situations. In other embodiments, the power source 340 may be associated with a connection to an external power source (such as a power grid) that can be used to charge the power source 340. Additionally, the power source 340 may be configured to charge one or more batteries included in the XR unit 204, for example, so that extended reality glasses (e.g., wearable extended reality device 110) can be charged (e.g., wirelessly or not) when the glasses are placed on or in proximity to the input unit 202.

[0123] 3 can be output through various output devices using, for example, light indicators 351, display 352, and / or speaker 353. In one embodiment, output interface 350 may be an integrated circuit that can act as a bridge between processing device 360 ​​and at least one of the output devices listed above. Light indicator 351 may include one or more light sources, for example, an LED array associated with various colors. Display 352 may include a screen (e.g., an LCD or dot-matrix screen) or a touchscreen. Speaker 353 may include audio headphones, a hearing aid-type device, a speaker, bone conduction headphones, an interface that provides tactile cues, a vibrotactile stimulator, etc.

[0124] The processing device 360 ​​shown in FIG. 3 may include at least one processor configured to execute computer programs, applications, methods, processes, or other software to perform the embodiments described in this disclosure. Generally, the processing device included in any device or system in this disclosure may include one or more integrated circuits, microchips, microcontrollers, microprocessors, all or part of a central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), field programmable gate array (FPGA), or other circuitry suitable for executing instructions or performing logical operations. The processing device may include at least one processor configured to perform the functions of the disclosed methods, such as a microprocessor manufactured by Intel™. The processing device may include a single-core or multi-core processor that simultaneously executes parallel processes. In one example, the processing device may be a single-core processor configured with virtual processing technology. The processing device may implement virtual machine technology or other technologies to provide the ability to run, control, operate, manipulate, store, etc., multiple software processes, applications, programs, etc. In another example, the processing device may include a multi-core processor configuration (e.g., dual-core, quad-core, etc.) configured to provide parallel processing capabilities to enable devices associated with the processing device to execute multiple processes simultaneously. It should be understood that other types of processor configurations may be implemented to provide the capabilities disclosed herein.

[0125] 3 can acquire sensor data from a variety of sensors, such as audio sensor 371, image sensor 372, motion sensor 373, environmental sensor 374, and other sensors 375. In one embodiment, sensor interface 370 can be an integrated circuit that can act as a bridge between processing device 360 ​​and at least one of the sensors listed above.

[0126] Audio sensor 371 may include one or more audio sensors configured to capture audio by converting sound into digital information. Some examples of audio sensors may include a microphone, a unidirectional microphone, a bidirectional microphone, a cardioid microphone, an omnidirectional microphone, an on-board microphone, a wired microphone, a wireless microphone, or any combination of the above. Consistent with the present disclosure, processing device 360 ​​can modify the presentation of virtual content based on data (e.g., voice commands) received from audio sensor 371.

[0127] Image sensor 372 may include one or more image sensors configured to capture visual information by converting light into image data. Consistent with this disclosure, the image sensor may be included in any device or system herein and may be any device capable of detecting and converting optical signals in the near-infrared, infrared, visible, and ultraviolet spectrums into electrical signals. Examples of image sensors may include digital cameras, phone cameras, semiconductor charge-coupled devices (CCDs), active pixel sensors in complementary metal-oxide semiconductor (CMOS), or N-type metal-oxide semiconductor (NMOS). The electrical signals may be used to generate image data. Consistent with this disclosure, the image data may include pixel data streams, digital images, digital video streams, data obtained from captured images, and data that can be used to construct one or more 3D images, a sequence of 3D images, a 3D video, or a virtual 3D representation. Image data acquired by image sensor 372 may be transmitted to any processing device of system 200 via wired or wireless transmission. For example, the image data may be processed to detect objects, detect events, detect actions, detect faces, detect people, recognize known persons, or any other information that may be used by system 200. Consistent with the present disclosure, processing device 360 ​​may modify the presentation of virtual content based on the image data received from image sensor 372.

[0128] The motion sensor 373 may include one or more motion sensors configured to measure the movement of the input unit 202 or the movement of objects in the environment of the input unit 202. Specifically, the motion sensor may perform at least one of: detecting the movement of objects in the environment of the input unit 202, measuring the velocity of objects in the environment of the input unit 202, measuring the acceleration of objects in the environment of the input unit 202, detecting the movement of the input unit 202, measuring the velocity of the input unit 202, measuring the acceleration of the input unit 202, etc. In some embodiments, the motion sensor 373 may include one or more accelerometers configured to detect changes in the appropriate acceleration and / or measure the appropriate acceleration of the input unit 202. In other embodiments, the motion sensor 373 may include one or more gyroscopes configured to detect changes in the orientation of the input unit 202 and / or measure information related to the orientation of the input unit 202. In other embodiments, the motion sensor 373 may include one or more using image sensors, LIDAR sensors, radar sensors, or proximity sensors. For example, by analyzing the captured images, the processing device can determine the movement of the input unit 202, e.g., using an egomotion algorithm. Additionally, the processing device can determine the movement of objects in the input unit 202's environment, e.g., using an object tracking algorithm. Consistent with this disclosure, the processing device 360 ​​can modify the presentation of virtual content based on the determined movement of the input unit 202 or the determined movement of objects in the input unit 202's environment, e.g., by causing a virtual display to follow the movement of the input unit 202.

[0129] The environmental sensors 374 may include one or more sensors of different types configured to capture data reflecting the environment of the input unit 202. In some embodiments, the environmental sensors 374 may include one or more chemical sensors configured to measure at least one of: measuring a chemical property in the environment of the input unit 202; measuring a change in a chemical property in the environment of the input unit 202; detecting the presence of a chemical in the environment of the input unit 202; or measuring the concentration of a chemical in the environment of the input unit 202. Examples of such chemical properties may include pH level, toxicity, and temperature. Examples of such chemicals may include electrolytes, certain enzymes, certain hormones, certain proteins, smoke, carbon dioxide, carbon monoxide, oxygen, ozone, hydrogen, and hydrogen sulfide. In other embodiments, the environmental sensors 374 may include one or more temperature sensors configured to detect a change in temperature of the environment of the input unit 202 and / or to measure the temperature of the environment of the input unit 202. In other embodiments, the environmental sensor 374 may include one or more barometers configured to detect changes in atmospheric pressure in the input unit 202's environment and / or measure the atmospheric pressure in the input unit 202's environment. In other embodiments, the environmental sensor 374 may include one or more light sensors configured to detect changes in ambient light in the input unit 202's environment. Consistent with the present disclosure, the processing device 360 ​​may modify the presentation of the virtual content based on input from the environmental sensor 374. For example, automatically reducing the brightness of the virtual content when the user 100's environment becomes dark.

[0130] The other sensors 375 may include weight sensors, optical sensors, resistive sensors, ultrasonic sensors, proximity sensors, biometric sensors, or other sensing devices to facilitate related functions. In particular embodiments, the other sensors 375 may include one or more positioning sensors configured to obtain positioning information of the input unit 202, detect changes in the position of the input unit 202, and / or measure the position of the input unit 202. Alternatively, the GPS software may enable the input unit 202 to access an external GPS receiver (e.g., connect via a serial port or Bluetooth). Consistent with the present disclosure, the processing device 360 ​​may modify the presentation of virtual content based on input from the other sensors 375. For example, presenting personal information only after identifying the user 100 using data from a biometric sensor.

[0131] The components and arrangement shown in FIG. 3 are not intended to limit the disclosed embodiments. Numerous variations and / or modifications may be made to the illustrated configuration of input unit 202, as would be understood by one of ordinary skill in the art having the benefit of this disclosure. For example, not all components may be essential to the operation of the input unit in all cases. Any component may be located in any suitable portion of the input unit, and components may be rearranged in various configurations while still providing the functionality of the disclosed embodiments. For example, some input units may not include all of the elements shown in input unit 202.

[0132] FIG. 4 is a block diagram of an exemplary configuration of the XR unit 204. It should be understood that FIG. 4 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. In the embodiment of FIG. 4, the XR unit 204 may have direct or indirect access to a bus 400 (or other communication mechanism) that interconnects subsystems and components for transferring information within the XR unit 204. For example, the bus 400 may interconnect a memory interface 410, a network interface 420, an input interface 430, a power supply 440, an output interface 450, a processing device 460, a sensor interface 470, and a database 480.

[0133] The memory interface 410 shown in FIG. 4 is assumed to have functionality similar to that of the memory interface 310 described in detail above. The memory interface 410 may be used to access software products and / or data stored on a non-transitory computer-readable medium or in a memory device, such as the memory device 411. The memory device 411 may include software modules for performing processes consistent with the present disclosure. In particular, the memory device 411 may include an input determination module 412, an output determination module 413, a sensor communication module 414, a virtual content determination module 415, a virtual content communication module 416, and a database access module 417. The modules 412-417 may include software instructions for execution by at least one processor (e.g., the processing device 460) associated with the XR unit 204. The input determination module 412, the output determination module 413, the sensor communication module 414, the virtual content determination module 415, the virtual content communication module 416, and the database access module 417 may cooperate to perform various operations. For example, the input determination module 412 can determine user interface (UI) input received from the input unit 202. Simultaneously, the sensor communication module 414 can receive data from various sensors to determine the status of the user 100. The virtual content determination module 415 can determine virtual content to display based on the received input and the determined status of the user 100. The virtual content communication module 416 can search for virtual content not determined by the virtual content determination module 415. The search for virtual content may be from the database 380, the database 480, the mobile communication device 206, or the remote processing unit 208. Based on the output of the virtual content determination module 415, the output determination module 413 can cause a change in the virtual content displayed to the user 100 by the projector 454.

[0134] In some embodiments, the input determination module 412 can coordinate operation of the input interface 430 to receive gesture input 431, virtual input 432, audio input 433, and UI input 434. Consistent with the present disclosure, the input determination module 412 can simultaneously receive different types of input data. In one embodiment, the input determination module 412 can apply various rules based on the type of input detected. For example, gesture input may be prioritized over virtual input. In some embodiments, the output determination module 413 can coordinate operation of the output interface 450 to generate output using a light indicator 451, a display 452, a speaker 453, and a projector 454. In one embodiment, the light indicator 451 may include a light indicator that indicates the status of the wearable extended reality device. For example, the light indicator may display a green light when the wearable extended reality device 110 is connected to the input unit 202 and flash when the battery of the wearable extended reality device 110 is low. In another embodiment, display 452 may be used to display operational information. In another embodiment, speaker 453 may include bone conduction headphones used to output audio to user 100. In another embodiment, projector 454 may present virtual content to user 100.

[0135] The operation of the sensor communication module, virtual content determination module, virtual content communication module, and database access module has been described above with reference to Figure 3, and the details thereof will not be repeated here. Modules 412-417 may be implemented in software, hardware, firmware, any mixture thereof, etc.

[0136] 4 is assumed to have functionality similar to that of the network interface 320 described in detail above. The specific design and implementation of the network interface 420 may depend on the communication network in which the XR unit 204 is intended to operate. For example, in some embodiments, the XR unit 204 is configured to be selectively connectable to the input unit 202 via a wire. When connected via a wire, the network interface 420 may enable communication with the input unit 202, and when not connected via a wire, the network interface 420 may enable communication with the mobile communication device 206.

[0137] 4 is assumed to have functionality similar to that of the input interface 330 described in detail above. In this case, the input interface 430 can communicate with an image sensor to obtain gesture input 431 (e.g., a finger of the user 100 pointing at a virtual object), communicate with other XR units 204 to obtain virtual input 432 (e.g., a virtual object shared with the XR unit 204, or a gesture of an avatar detected in the virtual environment), communicate with a microphone to obtain audio input 433 (e.g., a voice command), and communicate with the input unit 202 to obtain UI input 434 (e.g., virtual content determined by the virtual content determination module 315).

[0138] 4 is assumed to have functionality similar to that of the power supply 340 described above, and simply provides electrical energy to power the XR unit 204. In some embodiments, the power supply 440 may be charged by the power supply 340. For example, the power supply 440 may be changed wirelessly when the XR unit 204 is placed on or near the input unit 202.

[0139] The output interface 450 shown in FIG. 4 is assumed to have functionality similar to that of the output interface 350 described in detail above. In this case, the output interface 450 can provide output from a light indicator 451, a display 452, a speaker 453, and a projector 454. The projector 454 can be any device, apparatus, fixture, etc. that can project (or direct) light to display virtual content onto a surface. This surface can be part of the XR unit 204, part of the eye of the user 100, or part of an object in close proximity to the user 100. In one embodiment, the projector 454 can include a lighting unit that focuses light within a limited solid angle by one or more mirrors and lenses to provide a high value of luminous intensity in a defined direction.

[0140] 4 is assumed to have functionality similar to that of output processing device 360, described in detail above. When XR unit 204 is connected to input unit 202, processing device 460 can cooperate with processing device 360. In particular, processing device 460 can implement virtual machine technology or other technology to provide the capability to run, control, operate, manipulate, store, etc., multiple software processes, applications, programs, etc. It should be understood that other types of processor configurations can be implemented to provide the capabilities disclosed herein.

[0141] The sensor interface 470 shown in Figure 4 is assumed to have functionality similar to that of the sensor interface 370 described in detail above. Specifically, the sensor interface 470 can communicate with an audio sensor 471, an image sensor 472, a motion sensor 473, an environmental sensor 474, and other sensors 475. The operation of the audio, image, motion, environmental, and other sensors was described above with reference to Figure 3, and the details will not be repeated here. It should be understood that other types and combinations of sensors may be used to provide the capabilities disclosed herein.

[0142] The components and arrangement shown in FIG. 4 are not intended to limit the disclosed embodiments. As will be understood by one of ordinary skill in the art having the benefit of this disclosure, numerous variations and / or modifications may be made to the illustrated configuration of the XR unit 204. For example, not all components are essential to the operation of the XR unit 204 in all cases. Any component may be located in any suitable part of the system 200, and components may be rearranged in various configurations while still providing functionality of the disclosed embodiments. For example, some XR units may not include all of the elements in the XR unit 204 (e.g., the wearable extended reality device 110 may not have a light indicator 451).

[0143] FIG. 5 is a block diagram of an exemplary configuration of remote processing unit 208. It should be understood that FIG. 5 is merely an exemplary representation of one embodiment, and that some illustrated elements may be omitted and other elements may be added within the scope of the present disclosure. In the embodiment of FIG. 5, remote processing unit 208 may include server 210, directly or indirectly accessing bus 500 (or other communication mechanism) that interconnects subsystems and components for transferring information within server 210. For example, bus 500 may interconnect memory interface 510, network interface 520, power supply 540, processing device 560, and database 580. Remote processing unit 208 may also include one or more data structures, such as data structures 212A, 212B, and 212C.

[0144] The memory interface 510 shown in FIG. 5 is assumed to have functionality similar to that of the memory interface 310 described in detail above. The memory interface 510 may be used to access software products and / or data stored on a non-transitory computer-readable medium or in other memory devices, such as memory devices 311, 411, 511, or data structures 212A, 212B, and 212C. The memory device 511 may include software modules for performing processes consistent with the present disclosure. In particular, the memory device 511 may include a shared memory module 512, a node registration module 513, a load balancing module 514, one or more compute nodes 515, an internal communications module 516, an external communications module 517, and a database access module (not shown). The modules 512-517 may include software instructions for execution by at least one processor (e.g., processing device 560) associated with the remote processing unit 208. The shared memory module 512, the node registration module 513, the load balancing module 514, the computation module 515, and the external communication module 517 may cooperate to perform various operations.

[0145] The shared memory module 512 may enable information sharing between the remote processing unit 208 and other components of the system 200. In some embodiments, the shared memory module 512 may be configured to enable the processing device 560 (and other processing devices in the system 200) to access, retrieve, and store data. For example, using the shared memory module 512, the processing device 560 may at least one of execute software programs stored in the memory device 511, the database 580, or the data structures 212A-C, store information in the memory device 511, the database 580, or the data structures 212A-C, or retrieve information from the memory device 511, the database 580, or the data structures 212A-C.

[0146] The node registration module 513 may be configured to track the availability of one or more computational nodes 515. In some examples, the node registration module 513 may be implemented as a software program, such as a software program executed by one or more computational nodes 515, a hardware solution, or a combined software and hardware solution. In some implementations, the node registration module 513 may communicate with one or more computational nodes 515, for example, using the internal communication module 516. In some examples, one or more computational nodes 515 may inform the node registration module 513 of their status by sending a message, for example, upon startup, upon shutdown, at regular intervals, at selected times, in response to a query received from the node registration module 513, or at any other determined time. In some examples, the node registration module 513 may inquire about the status of one or more computational nodes 515 by sending a message, for example, upon startup, at regular intervals, at selected times, or at any other determined time.

[0147] The load balancing module 514 may be configured to divide the workload among one or more computing nodes 515. In some examples, the load balancing module 514 may be implemented as a software program, such as a software program executed by one or more of the computing nodes 515, a hardware solution, or a combined software and hardware solution. In some implementations, the load balancing module 514 may interact with the node registration module 513 to obtain information regarding the availability of one or more computing nodes 515. In some implementations, the load balancing module 514 may communicate with one or more computing nodes 515 using, for example, the internal communications module 516. In some examples, one or more computing nodes 515 may inform the load balancing module 514 of their status by sending a message, for example, upon startup, upon shutdown, at regular intervals, at selected times, in response to a query received from the load balancing module 514, or at any other determined time. In some examples, the load balancing module 514 may inquire about the status of one or more compute nodes 515 by sending a message, for example, upon startup, at regular intervals, at preselected times, or at any other determined time.

[0148] Internal communications module 516 may be configured to receive and / or transmit information from one or more components of remote processing unit 208. For example, control and / or synchronization signals may be transmitted and / or received through internal communications module 516. In one embodiment, input information for a computer program, output information for a computer program, and / or intermediate information for a computer program may be transmitted and / or received through internal communications module 516. In another embodiment, information received through internal communications module 516 may be stored in memory device 511, database 580, data structures 212A-C, or other memory devices within system 200. For example, information retrieved from data structure 212A may be transmitted using internal communications module 516. In another example, input data may be received using internal communications module 516 and stored in data structure 212B.

[0149] The external communication module 517 may be configured to receive and / or transmit information from one or more components of the system 200. For example, control signals may be transmitted and / or received through the external communication module 517. In one embodiment, information received through the external communication module 517 may be stored in the memory device 511, the database 580, the data structures 212A-C, and / or any memory device within the system 200. In another embodiment, information retrieved from any of the data structures 212A-C may be transmitted to the XR unit 204 using the external communication module 517. In another embodiment, input data may be transmitted and / or received using the external communication module 517. Examples of such input data may include data received from the input unit 202, information captured from the environment of the user 100 using one or more sensors (e.g., audio sensor 471, image sensor 472, motion sensor 473, environmental sensor 474, other sensors 475), etc.

[0150] In some embodiments, aspects of modules 512-517 may be implemented, alone or in various combinations with one another, in hardware executable by one or more processors, in software (including one or more signal processing and / or application specific integrated circuits), in firmware, or any combination thereof. In particular, modules 512-517 may be configured to interact with each other and / or with other modules of system 200 to perform functions consistent with disclosed embodiments. Memory device 511 may include additional modules and instructions or fewer modules and instructions.

[0151] 5 assume functionality similar to that of similar elements described above with reference to FIGS. 4 and 5. The specific design and implementation of the above-described components may vary based on the implementation of system 200. Additionally, remote processing unit 208 may include more or fewer components. For example, remote processing unit 208 may include an input interface configured to receive input directly from one or more input devices.

[0152] Consistent with the present disclosure, a processing device of system 200 (e.g., a processor in mobile communication device 206, a processor in server 210, a processor in a wearable extended reality device such as wearable extended reality device 110, and / or a processor in an input device associated with wearable extended reality device 110, such as keyboard 104) may use machine learning algorithms to implement any of the methods disclosed herein. In some embodiments, the machine learning algorithms (also referred to in this disclosure as machine learning models) may be trained using training examples, for example, as described below. Some non-limiting examples of such machine learning algorithms may include classification algorithms, data regression algorithms, image segmentation algorithms, visual detection algorithms (such as object detectors, face detectors, person detectors, motion detectors, edge detectors, etc.), visual recognition algorithms (such as face recognition, person recognition, object recognition, etc.), speech recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbor algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recurrent neural network algorithms, linear machine learning models, nonlinear machine learning models, ensemble algorithms, etc. For example, the trained machine learning algorithms may include predictive models, classification models, data regression models, clustering models, segmentation models, inference models such as artificial neural networks (such as deep neural networks, convolutional neural networks, recurrent neural networks), random forests, support vector machines, etc. In some examples, the training examples may include example inputs along with desired outputs corresponding to the example inputs. Further, in some examples, training a machine learning algorithm using training examples can produce a trained machine learning algorithm, which can be used to estimate outputs for inputs not included in the training examples.In some examples, engineers, scientists, processes, and machines that train machine learning algorithms may further use validation examples and / or test examples. For example, the validation examples and / or test examples may include example inputs along with desired outputs corresponding to the example inputs, and the trained machine learning algorithm and / or intermediate trained machine learning algorithm may be used to estimate outputs for the example inputs of the validation examples and / or test examples, the estimated outputs may be compared with the corresponding desired outputs, and the trained machine learning algorithm and / or intermediate trained machine learning algorithm may be evaluated based on the results of the comparison. In some examples, the machine learning algorithm may have parameters and hyperparameters, and the hyperparameters may be set manually by a person or automatically by a process external to the machine learning algorithm (such as a hyperparameter search algorithm), and the parameters of the machine learning algorithm may be set by the machine learning algorithm based on training examples. In some implementations, the hyperparameters may be set based on training examples and validation examples, and the parameters may be set based on training examples and selected hyperparameters. For example, given hyperparameters, the parameters may be conditionally independent of the validation examples.

[0153] In some embodiments, a trained machine learning algorithm (also referred to in this disclosure as a machine learning model and a trained machine learning model) may be used to analyze inputs and generate outputs, for example, as described below. In some examples, a trained machine learning algorithm may be used as an inference model that generates an inferred output when provided with an input. For example, the trained machine learning algorithm may include a classification algorithm, where the input may include samples and the inferred output may include a classification of the samples (an inferred label, an inferred tag, etc.). In another example, the trained machine learning algorithm may include a regression model, where the input may include samples and the inferred output may include an inferred value corresponding to the samples. In yet another example, the trained machine learning algorithm may include a clustering model, where the input may include samples and the inferred output may include an assignment of the samples to at least one cluster. In an additional example, the trained machine learning algorithm may include a classification algorithm, where the input may include images and the inferred output may include a classification of an item shown in the image. In yet another example, the trained machine learning algorithm may include a regression model, where the input may include an image, and the inferred output may include an inferred value corresponding to an item shown in the image (such as an estimated property of the item, such as size, volume, age of a person shown in the image, distance from the item shown in the image, etc.). In an additional example, the trained machine learning algorithm may include an image segmentation model, where the input may include an image, and the inferred output may include a segmentation of the image. In yet another example, the trained machine learning algorithm may include an object detector, where the input may include an image, and the inferred output may include one or more detected objects in the image and / or one or more locations of the objects in the image.In some examples, the trained machine learning algorithm may include one or more formulas, and / or one or more functions, and / or one or more rules, and / or one or more procedures, the inputs may be used as inputs to the formulas and / or functions, and / or rules, and / or procedures, and the inferred output may be based on the output of the formulas and / or functions, and / or rules, and / or procedures (e.g., selecting one of the outputs of the formulas and / or functions, and / or rules, and / or procedures, using statistical measures of the output of the formulas and / or functions, and / or rules, and / or procedures, etc.).

[0154] Consistent with the present disclosure, the processing device of system 200 can analyze image data captured by an image sensor (e.g., image sensor 372, image sensor 472, or any other image sensor) to implement any of the methods disclosed herein. In some embodiments, analyzing the image data may include analyzing the image data to obtain preprocessed image data and then analyzing the image data and / or the preprocessed image data to obtain the desired result. Those skilled in the art will recognize that the following are examples and that other types of preprocessing methods may be used to preprocess the image data. In some examples, the image data may be preprocessed by transforming the image data using a transformation function to obtain transformed image data, and the preprocessed image data may include the transformed image data. For example, the transformed image data may include one or more convolutions of the image data. For example, the transformation function may include one or more image filters, such as a low-pass filter, a high-pass filter, a band-pass filter, an all-pass filter, etc. In some examples, the transformation function may include a nonlinear function. In some examples, the image data may be preprocessed by smoothing at least a portion of the image data, for example, using a Gaussian convolution, using a median filter, etc. In some examples, the image data may be preprocessed to obtain different representations of the image data. For example, the preprocessed image data may include a representation of at least a portion of the image data in the frequency domain, a discrete Fourier transform of at least a portion of the image data, a discrete wavelet transform of at least a portion of the image data, a time / frequency representation of at least a portion of the image data, a representation of at least a portion of the image data in a reduced dimension, a lossy representation of at least a portion of the image data, a lossless representation of at least a portion of the image data, a time series of any of the above, any combination of the above, etc. In some examples, the image data may be preprocessed to extract edges, and the preprocessed image data may include information based on and / or related to the extracted edges. In some examples, the image data may be preprocessed to extract image features from the image data.Some non-limiting examples of such image features may include information based on and / or related to edges, corners, blobs, ridges, Scale Invariant Feature Transform (SIFT) features, temporal features, etc. In some examples, analyzing the image data may include computing at least one convolution of at least a portion of the image data and using the computed at least one convolution to compute at least one resultant value and / or to determine, identify, recognize, classify, etc.

[0155] Consistent with other aspects of the present disclosure, the processing device of system 200 can analyze image data to implement any of the methods disclosed herein. In some embodiments, analyzing the image may include analyzing the image data and / or preprocessed image data using one or more rules, functions, procedures, artificial neural networks, object detection algorithms, face detection algorithms, visual event detection algorithms, action detection algorithms, motion detection algorithms, background subtraction algorithms, inference models, etc. Some non-limiting examples of such inference models may include manually pre-programmed inference models, classification models, regression models, results of training algorithms such as machine learning algorithms and / or deep learning algorithms on training examples, etc. The training examples may include example data instances, and in some cases, the data instances may be labeled with corresponding desired labels and / or results. In some embodiments, analyzing the image data (e.g., by the methods, steps, and modules described herein) may include analyzing pixels, voxels, point clouds, range data, etc. included in the image data.

[0156] Convolutions may include convolutions of any dimension. A one-dimensional convolution is a function that transforms an original sequence into a transformed sequence. A one-dimensional convolution may be defined by a sequence of scalars. Each particular value of the transformed sequence may be determined by calculating a linear combination of values ​​of subsequences of the original sequence that correspond to a particular value. The resulting value of the calculated convolution may include any value of the transformed sequence. Similarly, an n-dimensional convolution is a function that transforms an original n-dimensional array into a transformed array. An n-dimensional convolution may be defined by an n-dimensional array of scalars (known as the kernel of the n-dimensional convolution). Each particular value of the transformed array may be determined by calculating a linear combination of values ​​in an n-dimensional region of the original array that correspond to a particular value. The resulting value of the calculated convolution may include any value in the transformed array. In some examples, an image may include one or more components (such as a color component, a depth component, etc.), and each component may include a two-dimensional array of pixel values. In one example, calculating the convolution of an image may include calculating a two-dimensional convolution for one or more components of the image. In another example, computing the convolution of an image may include stacking arrays from various components to create a three-dimensional array and computing the three-dimensional convolution on the resulting three-dimensional array. In some examples, a video may include one or more components (such as a color component, a depth component, etc.), and each component may include a three-dimensional array of pixel values ​​(having two spatial axes and one temporal axis). In one example, computing the convolution of a video may include computing the three-dimensional convolution on one or more components of the video. In another example, computing the convolution of a video may include stacking arrays from various components to create a four-dimensional array and computing the four-dimensional convolution on the resulting four-dimensional array.

[0157] Some disclosed embodiments may include systems, methods, and non-transitory computer-readable media configured to enable content sharing between users of wearable extended reality devices. Content sharing may include the presentation of text, images, programs, or any other information by one entity to another entity. Content may be shared between users of wearable extended reality devices such that an entity wearing a device may view or otherwise access content available through another wearable extended reality device. Content sharing may be enabled in many different ways. In some embodiments, enablement may occur through a link between two extended reality devices. In one example, the link may include a direct and / or indirect communication link between the two extended reality devices. In another example, a system (such as a centralized system) may communicate with each of the two extended reality devices to, for example, provide content for presentation to each of the two extended reality devices and receive location and / or orientation information associated with the extended reality devices and / or any other information related to presenting the content. A link may include a linkage within a system's data structure or database or maintained by the system. In one non-limiting example, detected proximity of two entities may trigger automatic or selectable sharing of content. In other embodiments, one entity may select content for sharing with either a group or an individual, or may tie content to a physical or virtual location, allowing anyone with access to the location to view the content. In some embodiments, content sharing may be enabled by permission, with viewing of shared content being enabled only for entities with appropriate permissions. In yet other embodiments, rules may define with whom content can be shared. In a further exemplary embodiment, sharing may be enabled via a request sent from one device to another.Content sharing may, in some embodiments, be initiated as a result of a signal from one or more sensors. For example, one or more image sensors may detect a gesture indicating an intent to share content. Alternatively, a proximity sensor may trigger the ability to share when two wearable extended reality devices are in close proximity to one another. There are many different ways in which content sharing may be enabled consistent with this disclosure, and this disclosure is not limited to any particular one.

[0158] Some disclosed embodiments may include a computer-readable medium including instructions that, when executed by at least one processor, cause the at least one processor to establish a link between a first wearable extended reality device and a second wearable extended reality device. In some embodiments, establishing a link may refer to any means of communicating between the first wearable extended reality device and the second wearable extended reality device. In some embodiments, there may be a formal protocol that governs how the first wearable extended reality device and the second wearable extended reality device send and receive data. Such protocols may include, but are not limited to, Transmission Control Protocol / Internet Protocol (TCP / IP), Bluetooth, infrared, near field communications, ultra-wideband, WiFi, ZigBee, short-range communication protocols, and / or long-range communication protocols. Such communication means between the first wearable extended reality device and the second wearable extended reality device may include, but are not limited to, a local area network (LAN), a wireless local area network (WLAN), a virtual private network (VPN), an indirect communication link, and / or a directional communication link. The processor may cause the first wearable extended reality device to initiate communication with the second wearable extended reality device via one or more of the above-identified protocols. In some examples, establishing a link may refer to linkage in data structures and / or databases, for example, in a system that coordinates the extended reality environment, a system that communicates with both the first and second wearable extended reality devices, and / or a system that provides content for presentation to both the first and second wearable extended reality devices.

[0159] Some disclosed embodiments may include establishing a link (e.g., a communication link, a link in a data structure, and / or a database) and enabling information exchange when a first wearable extended reality device is detected in proximity to a second wearable extended reality device. Such instructions may be configured based on a threshold distance, which may vary between users of the extended reality devices. For example, a first wearable extended reality device user may prefer to initiate a link with a second wearable extended reality device at a shorter threshold distance, e.g., one or two meters, when a particular type of user, such as family, is around. In contrast, the same first wearable extended reality device user may prefer to initiate a link with a second wearable extended reality device at a longer threshold distance to facilitate communication with colleagues or team members. Users are free to update the threshold distance setting according to their preferences.

[0160] In some examples, the communication link between the wearable extended reality devices may be via an intermediate device. The intermediate device may include any networking device disposed between a remote access server (RAS) and an RAS client. The intermediate device may assist in providing connectivity between two separate wearable extended reality devices. The intermediate device may provide an additional layer of security before a communication link between the two wearable extended reality devices is established. For example, if a security protocol is not met, the intermediate device may prevent the establishment of a connection between the two extended reality devices.

[0161] Some disclosed embodiments may include presenting first virtual content through a first wearable extended reality device. The virtual content may take a variety of different forms and may include documents, photos, videos, virtual characters, and any other shareable media that may be transmitted wirelessly. In some examples, the virtual content may be presented as part of an extended reality environment. In other examples, the virtual content may be displayed on a physical screen, such as a television, tablet, laptop, or smartphone, or may be displayed on a virtual screen, such as through one or more extended reality devices.

[0162] Some disclosed embodiments may include obtaining a first command to display first virtual content via a second wearable extended reality device. The first command may refer to a signal received from the at least one processor to display the virtual content via the second wearable extended reality device. In another example, the first command may refer to a command obtained from a user and / or in response to a user action. The non-transitory computer-readable medium may be configured for sharing content and may include instructions for the at least one processor to send a signal to the first wearable extended reality device to display the virtual content.

[0163] In some disclosed embodiments, obtaining the first command to present virtual content may include a sharing intent in the data captured by the first wearable extended reality device. The sharing intent may refer to a user's desire to present virtual content to another user or exchange virtual content with another user. In one example, a machine learning model may be trained using training examples to identify the sharing intent by analyzing the captured data. Examples of such training examples may include sample captured data along with a label indicating whether the sample captured data corresponds to the sharing intent and / or parameters of the sharing intent (e.g., an intention to share specific content, an intention to share content presented in a specific area, an intention to share content with a specific entity, and / or an intention to share content in a specific manner). The trained machine learning model may be used to analyze the data captured by the first wearable extended reality device and identify the sharing intent and / or parameters of the sharing intent. For example, the user's sharing intent may be identified in the captured image data. Some non-limiting examples of such captured image data may include image data (such as one or more images and / or one or more videos) captured using an image sensor included in the first wearable extended reality device, an image sensor included in the second wearable extended reality device, an image sensor included in a device in the environment of the first and second wearable extended reality devices (the device may be different from the first and second wearable extended reality devices, such as another wearable extended reality device, a fixed camera mounted in the room, and / or any other device that may capture image or video data). The user's shared intent may be identified by analyzing the image data to detect movement or changes in position or orientation of objects in the captured image data.For example, a user's sharing intent may be identified by detecting the user's hand movements (e.g., the user is waving) by using a visual gesture recognition algorithm to analyze image data. In another example, a user's sharing intent may be identified by detecting that the user has moved hardware in one or more captured images, for example, using a visual object tracking algorithm. Additionally, such a sharing intent may be captured when a user moves their finger, nods, touches an extended reality device, or otherwise gestures to another user with whom they wish to share content. In some examples, a machine learning model may be trained using training examples to identify sharing intents by analyzing images and / or videos. Examples of such training examples may include sample images and / or sample videos along with labels indicating whether the sample images and / or sample videos correspond to sharing intents and / or parameters of the sharing intent (e.g., an intention to share specific content, an intention to share content presented in a specific area, an intention to share content with a specific entity, an intention to share content in a specific manner, and / or an intention to share content at a specific time). The trained machine learning model may be used to analyze the captured image data and identify sharing intents and / or parameters of the sharing intent. In some examples, a convolution of at least a portion of the image data may be calculated to obtain a result value of the calculated convolution. In one example, a shared intention of a first virtual content may be identified in response to a first result value of the calculated convolution, and a shared intention of the first virtual content may not be identified in response to a second result value of the calculated convolution. In another example, a shared intention of one virtual content may be identified in response to a first result value of the calculated convolution, and a shared intention of another virtual content may be identified in response to a second result value of the calculated convolution.

[0164] As an example, Figure 6 illustrates a sharing intent of a first extended reality device user 610 via a sharing action 612. For example, as shown in Figure 6, a hand gesture by the first user 610 reflects an intent to share virtual content 614 with a second extended reality device user 616.

[0165] Hardware that may be used to capture a user's shared intent may include a pointer, keyboard, mouse, joystick, or any other object designed to facilitate sharing data. Such a shared intent may be captured when a user moves hardware, such as a mouse or joystick, toward another user. In some embodiments, a user's shared intent may also be determined when the user waves a wand, presses a button on a keyboard, or otherwise gestures toward another user. In some embodiments, identification of a shared intent based on data captured from the hardware may be configured based on user preferences and may not be limited to moving hardware toward a second user.

[0166] User preferences refer to how a particular wearable extended reality device user chooses to configure their device to share virtual content. For example, one user may prefer to share virtual content via hand movements, while another user may prefer to share content via a pointer or joystick. Alternatively, users may be permitted to define gesture preferences for sharing content.

[0167] In some embodiments, the sharing intent may also be identified in the captured audio data. For example, the captured audio data may include audio data captured using an audio sensor included in the first wearable extended reality device, using an audio sensor included in the second wearable extended reality device, and / or using an audio sensor included in another device. In one example, a user may verbally initiate sharing through a voice command. The command may include, for example, but is not limited to, "share content," "send content," "view content," "present content," or any other word combination that may be interpreted as a request to share content with another user. In addition, the verbal command may specifically name the second user and may take the form of, for example, "share content with user X." The verbal command may also be configured based on user preferences. In one example, the captured audio data may be analyzed using a voice recognition algorithm to identify a voice command that corresponds to the sharing intent and / or parameters of the sharing intent.

[0168] As mentioned above, the preferences of each unique wearable extended reality device user may be stored in a non-transitory computer-readable medium that allows for control of content sharing.

[0169] In some embodiments, a shared intention may be identified in captured positioning data. Such positioning data may be captured based on the position and / or orientation of a first wearable extended reality device user, based on the position and / or orientation of the first wearable extended reality device, and / or based on the position and / or orientation of a device associated with the first wearable extended reality device (e.g., another device used by the user of the first wearable extended reality device, such as a keyboard, gloves, and / or pointing device). For example, a shared intention may be identified when a user positions their body and the wearable extended reality device close to the body of another user. As another example, a shared intention may be identified when a user stands from a sitting position, leans toward another user, or otherwise positions their body or the extended reality device to indicate a shared intention. Shared intentions based on positioning data may also be configured based on user preferences. In one example, the position and / or orientation may be determined by analyzing data captured using sensors included in the first wearable extended reality device and / or sensors included in a different device, such as positioning sensors (such as accelerometers, gyroscopes, and / or GPS sensors) and / or image sensors (e.g., by analyzing captured image data using image sensors, using ego-motion algorithms, and / or using visual object tracking algorithms).

[0170] In some embodiments, obtaining the first command may include identifying a shared intent associated with an action performed in an extended reality environment (such as a virtual reality environment, an augmented reality environment, and / or a mixed reality environment). As described in the following paragraphs, the action performed in the extended reality environment may include sharing content via a sharing bubble or a sharing ring, or by automatically sharing content with another wearable extended reality device user upon establishing a link between two or more wearable extended reality devices.

[0171] In some embodiments, identifying the shared intent may include determining that a first user of a first wearable extended reality device has changed the orientation of a virtual screen presenting first virtual content toward a second user of a second wearable extended reality device. In one example, determining that a first user of a first wearable extended reality device has changed the orientation of the virtual screen toward the second user may include determining that the virtual screen is oriented toward the second user and determining that the virtual screen was oriented toward the second user as a result of an action of the first user (and not, for example, as a result of a movement of the second user). In one example, determining whether the virtual screen was oriented toward the second user as a result of an action of the first user may be based on an analysis of a movement of the second user and / or a movement of the virtual screen and / or an input leading to the movement of the virtual screen. For example, at least one of the system for coordinating the extended reality environment, the system for communicating with both the first and second wearable extended reality devices, the system for providing content for presentation to both the first and second wearable extended reality devices, or the first wearable extended reality device may determine that the orientation of the virtual screen has changed toward the second user. Based on preferences of the first wearable extended reality device user, this may indicate a shared intent with the second user. In some examples, the first angle may be an angle between a surface corresponding to the virtual screen and a direction associated with the user of the first wearable extended reality device, and the second angle may be an angle between a surface corresponding to the virtual screen and a direction associated with the user of the second wearable extended reality device, and the determination that the orientation of the virtual screen presenting the first virtual content has changed toward the second user of the second wearable extended reality device may be based on the first angle and the second angle.Some non-limiting examples of such directions associated with a user may include a direction of the user's head, a direction of the user's body, a direction of the user's gaze and / or eyes, a direction of the user's gaze fixation, a direction of the user's line of sight, a direction of the user's visual axis, a direction of the user's pupillary axis, a direction of the user's optical axis, and / or a direction associated with the orientation of the user's wearable extended reality device. In one example, a function of a first angle and a second angle may be calculated and compared to a threshold to determine whether the orientation of a virtual screen presenting a first virtual content has changed toward a second user. In another example, a machine learning model may be trained using training examples to determine whether the orientation of the virtual screen changes toward a user based on the angle and / or distance. Examples of such training examples may include a sample angle of a sample virtual screen relative to a sample user and / or a sample distance of a sample virtual screen from a sample user, along with a label indicating whether the sample virtual screen is oriented toward each of the sample users. The trained machine learning model can be used to analyze the first angle, the second angle, and possibly the user's distance from the virtual screen to determine that the virtual screen is oriented toward the second user.

[0172] Additionally, a user of a second wearable extended reality device may be located within proximity of the virtual screen, whether at a physical threshold distance or virtual proximity. In one example, a sharing intent may be determined when a user of a first wearable extended reality device leans or nods toward a user of a second wearable extended reality device, resulting in an orientation change being captured. The user of the second wearable extended reality device may also be a representation of the user, such as an avatar. Thus, even if the second user is not physically located next to the first user, the first user may share content based on an orientation change.

[0173] In some embodiments, identifying the sharing intent may include determining that a first user of a first wearable extended reality device has moved first virtual content toward a virtual shared space surrounding a second user of a second wearable extended reality device. In one example, the virtual shared space surrounding the second user may be referred to as a shared bubble. For example, the at least one processor may be configured to detect an action that may take the form of a gesture, a drag of the first virtual content, or other physical movement performed by the first user. The at least one processor may be configured to determine a movement of the first virtual content toward the virtual shared space based on the detected gesture, drag, or other physical movement. This sharing action may also be based on people's natural movement of leaning toward other people with whom they wish to share content. Such movement may be detected by one or more sensors within the at least one extended reality device or via a sensor associated with another device.

[0174] The virtual shared space in the above embodiments, known as a shared bubble, can be automatically determined in a myriad of different ways. For example, the virtual shared space can be determined based on the position and / or movement of the second user and / or the second wearable extended reality device. For example, the virtual shared space can include a space within a selected distance of the second user and / or the second wearable extended reality device. In this embodiment, when a first user is within a certain distance of the second user, virtual content can be automatically shared with the second user. In another example, the shared bubble can be elongated in the direction of movement. In yet another example, the shared bubble can be larger horizontally than vertically.

[0175] A shared bubble may be the space around a user. This space may be defined in a variety of different ways. In some embodiments, the size and / or orientation of a bubble surrounding one or more users may be determined by the number of users, the user's location, the location of other people not belonging to the bubble, the user's orientation, the orientation of other people not belonging to the bubble, the user's identity, the identity of users not belonging to the bubble, the location of non-animated objects in the user's vicinity, the type of non-animated objects in the user's vicinity, the status of non-animated objects in the user's vicinity, the user's movement pattern, the movement pattern of other people not belonging to the bubble, the user's preferences, the user's past behavior, the user's gestures, and / or other factors related to the proximity of users to each other. In one example, a phase function of one or more of these parameters may define the space of the shared bubble.

[0176] When two or more users' virtual shared bubbles collide, the two or more users may be automatically provided with an opportunity to create a new shared bubble in which virtual content items may be shared between the extended realities of the two or more users. At least one processor may be configured to determine that there is an overlap between the virtual shared bubbles, and in response, the at least one processor may suggest to users of the overlapping bubbles that virtual content may be shared between these users. A user's wearable extended reality device may also be configured to automatically share content when it overlaps with another user's bubble.

[0177] 7A shows a pair of users 714 and 716. As shown in FIG. 7A, a first wearable extended reality device user 714 can have an associated virtual shared bubble 710, and a second wearable extended reality device user 716 can have an associated virtual shared bubble 718. The virtual shared bubble 710 of the first wearable extended reality device user 714 may come into contact with and overlap with the virtual shared bubble 718 of the second wearable extended reality device user 716, for example, when the first wearable extended reality device user 714 moves toward the second user 716. FIG. 7B shows a condition in which the virtual shared bubble 710 overlaps with the virtual shared bubble 718 via an overlap region 720. The virtual content 712 may be automatically shared with a second wearable extended reality device user 716 when the sharing bubbles 710 and 718 overlap, for example, at an overlap region 720, and / or when the volume of the overlap region 720 exceeds a selected volume threshold. In one example, the volume threshold may be selected by a user, by configuration, as a function of the volume of the virtual sharing bubble 710 and / or the volume of the virtual sharing bubble 718, the volume of the users 714, 716, and / or the volume of the virtual content 712.

[0178] 8A and 8B show a combined virtual shared bubble 810 after the sharing bubbles from a first extended reality device user 812 and a second extended reality device user 816 overlap, as shown in FIG. 8A. Virtual content 814 may be automatically shared when these virtual shared bubbles overlap, as shown in FIG. 8B, but virtual content 814 may also be automatically shared when the first user 812 leans toward the second user 816 (e.g., when the first user 812 leans toward the second user 816 when their virtual shared bubbles overlap). The combined shared bubble 810 may decrease in size as the users move closer to each other.

[0179] The virtual shared space may also be determined based on the use mode of the second wearable extended reality device, the user, the second wearable extended reality device, and / or objects surrounding the second wearable extended reality device. For example, a user may configure the extended reality device to automatically share content when the user enters a room, passes the second user's desk, or walks within proximity of any other object in the room. Additionally, the use mode may be configured to automatically share content with some users and not share content with other users.

[0180] A use mode may refer to different settings in which users configure their extended reality system (e.g., their wearable extended reality device) based on different tasks the users wish to accomplish. For example, one use mode may apply only to activities performed at work. In this example, certain types of content, including documents and emails, may be shared at a particular distance between two wearable extended reality device users. In another example, a use mode may apply only to activities performed at home. A user may, for example, wish to share videos but not documents or emails.

[0181] In some embodiments, identifying the sharing intent may also include determining that a first user of a first wearable extended reality device has moved first virtual content toward a defined shared space. For example, in addition to or instead of automatically sharing content when two extended reality device users move toward each other, users may create a shared space based on predefined parameters, such as length, width, height, and / or location within a room. Instead of sharing content when two users' virtual sharing bubbles overlap, the sharing intent may be determined when users move content toward and / or within such a predefined space. In contrast to the sharing bubbles described above, which encompass each user, this sharing intent may be referred to as a sharing ring, since its characteristics are predefined by the extended reality device users. The sharing ring may be configured based on user preferences. In particular, the location and / or size of the sharing ring may be configured by each user of the wearable extended reality device.

[0182] Some non-limiting examples of such virtual shared spaces may include a ring, a rectangle, a cube, a sphere, a convex shape, a non-convex shape, a smooth shape, an irregular shape, or any other well-defined shape. In some examples, the virtual shared space may be anchored to a physical location, anchored to a physical object, created by a user, or created based on any other predefined characteristics. The virtual shared space (such as a shared ring) may have any two-dimensional or three-dimensional shape and is not limited to a ring. In one example, the virtual shared space (such as a shared ring) may be configured to be anchored to an object and move with the object. For example, the virtual shared space may be anchored to a keyboard, input device, output device, chair, or desk associated with a particular user, and moving virtual content into the virtual shared space may share the virtual content with the particular user. In the desk example, the virtual shared space may correspond to at least a portion of the area of ​​the desk's top surface. In the chair example, the virtual shared space may correspond to a three-dimensional shape that encompasses at least a portion of the chair. In the keyboard example, the keyboard may be placed on a desk, and the virtual shared space may correspond to at least a portion of the area of ​​the desk top, e.g., a portion of the area of ​​the desk top near the keyboard. Sharing content may also be based on predefined privacy settings that may be configured by the extended reality device user. In some examples, content may be shared between two users, shared with the public, shared among multiple users, or may include any other combination of public and private sharing. The type and level of sharing and / or the identities of users who may access the virtual content may be determined based on the virtual shared space, the content, the user, or any other factors based on where the user is located. In the shared space, movement of the virtual content may be the result of an action of the user of the first wearable extended reality device. Such an action may take the form of, for example, a gesture, dragging the first virtual content, or any other physical movement or movement involving the first virtual content.

[0183] 9 shows a shared ring 914, which may be a predefined virtual shared space created by a first wearable extended reality device user 910, a virtual shared space created by another wearable extended reality device user (such as one of users 918 or a user of a different wearable extended reality device), a virtual shared space created by a user not using a wearable extended reality device, and / or an automatically generated virtual shared space. The first user 910 can place virtual content 912 in the predefined area 914 such that the virtual content 912, which was previously visible only to the first wearable extended reality device user 910, becomes shared content 916 that may be visible to other extended reality device users 918.

[0184] In some examples, virtual content items may be shared based on people's natural movements and / or based on a user moving a virtual display screen and positioning the screen so that it is visible to other users or participants. For example, a hand movement or gesture may be recognized, and in response, a virtual content item (e.g., a virtual display screen) may be shared with the associated user. In some embodiments, all participants can view the virtual display screen docked in the same location. In some embodiments, a virtual content item may be viewed simultaneously by multiple participants in a public area and in a private mode as a specific user's additional / duplicate virtual display screen docked in a location of the user's choosing. For example, a user may prefer to view a virtual content item from a closer distance, especially if the virtual content item includes text, or may prefer to add a private overlay or any other configuration to the virtual content item. In one example, a virtual content item with a lower level of privacy may be shared without the need for user approval. In one example, a public content item may be viewed in a shared mode and a private mode, and shared content may be viewed in the private mode.

[0185] In some embodiments, for example, sharing of virtual content may be triggered when the sharing bubbles of a first user and a second user overlap, or when users share content within a predefined area, i.e., a sharing ring, or by any other means described herein. However, before virtual content between a first wearable extended reality device and a second wearable extended reality device can be shared, the non-transitory computer-readable medium may establish a link between the two devices. In some embodiments, virtual content may be automatically shared through the virtual shared space when a link between the first wearable extended reality device and the second wearable extended reality device is established. A processor may automatically generate a share of virtual content when the first and second wearable extended reality devices are linked to communicate with each other. The shared virtual content may be selected based on preferences of the users of the first and / or second extended reality devices. Whether a share is created immediately after establishing a link may be based on user preferences, and the extended reality devices may be configured for various uses.

[0186] For example, a user may configure a wearable extended reality device or system to automatically share content at work. In this example, the device may be configured to share content over longer distances, such as when coworkers are within line of sight, and the content may be limited to documents and emails. In another example, a user may configure a wearable extended reality device or system to automatically share content at home. In this example, the device may be configured to automatically share content only over shorter distances, such as when family members are immediately next to the user. Such virtual content may be limited to videos and music and may not include documents or emails.

[0187] In some embodiments, a visualization of the virtual shared space may be presented by each of the first wearable extended reality device and the second wearable extended reality device. For example, on a typical computer screen, documents may be shared by dragging and dropping files between various folders, such as Dropbox folders. In contrast, in some embodiments of the present disclosure, sharing virtual objects need not be limited to folders. Rather, virtual objects may be shared via virtual boxes, virtual windows, or otherwise marked areas. A virtual box may refer to a space on a virtual display presented by the first or second wearable extended reality device in which multiple users may share virtual content. This act of sharing may also occur in a virtual space in three-dimensional space, and the act of sharing virtual objects between such folders, virtual boxes, or windows may be visible to users of both the first and second wearable extended reality devices. In some examples, the visualization of the virtual shared space may include a visual indication of the boundary of the virtual shared space, a visual indication of the volume of the virtual shared space, and / or a visual indication of the center of the virtual shared space. Some non-limiting examples of such visual indicators may include virtual coloring of a surface or volume, virtual highlighting of a surface or volume, and / or one or more virtual visual symbols at a selected location. When a user virtually moves virtual content toward and / or into the visualized virtual shared space, sharing of the moved virtual content may be triggered. For example, a user may move virtual content using gestures, using a computer pointing device, using text input, and / or using voice commands.

[0188] In some embodiments, the location of the virtual shared space may be determined based on the location of physical objects in the environment of the first wearable extended reality device. For example, a particular physical object, such as a whiteboard or other display surface, may be linked to the virtual shared space so that anyone with permission and in the vicinity of the object can access the shared content. In some embodiments, the extended reality device user may select objects to define the virtual shared space. For example, artwork on a wall or a physical object on a table may be selected as the basis for the virtual shared space, and upon detection of the selected object, a virtual display screen may appear in an area adjacent to the selected object.

[0189] As an example, Figure 10A shows a sharing bubble in which a second wearable extended reality device 1012 receives first virtual content 1014 from a second wearable extended reality device 1010 via a sharing area 1016. Such virtual content can also be shared via a sharing ring as shown in Figure 9.

[0190] In another example, Figure 10B shows presenting received virtual content, where a second wearable extended reality device 1012 presents first virtual content 1014 received from a first wearable extended reality device 1010. The virtual content may be presented via the wearable extended reality device on a screen, such as a tablet or television, the device itself, or any other physical object, as represented by the square in Figure 10B.

[0191] Some embodiments may provide an indicator via the first wearable extended reality device when the first virtual content is presented via the second wearable extended reality device. The indicator may refer to any type of mark, annotation, or signal that notifies the user that their content is being displayed on another extended reality device. Additionally, the indicator may indicate whether the content is being shared privately, publicly, with a specific individual (and / or the identity of a specific individual), with a specific group (and / or the identity of a specific group), and other public or private sharing options. As another example, the indicator may indicate whether the user of the second wearable extended reality device has included annotations or other comments in the first virtual content or edited or transformed the first virtual content. In one example, the indicator provided via the first wearable extended reality device may be an audible indicator. In another example, the indicator provided via the first wearable extended reality device may be a visual indicator. For example, the visual indicator may be present on or along with the first virtual content. In another example, the visual indicator may include a visual modification to the presentation of the first virtual content.

[0192] There are many ways in which virtual content can be shared between wearable extended reality device users. Content can be shared via a sharing bubble or a sharing ring. A first wearable extended reality device user can share content with a second wearable extended reality device user via a sharing bubble or a sharing ring. In the above embodiment, the second wearable extended reality device user can display content received from the first user. This content can be displayed in a myriad of ways, whether through the second user's extended reality device, through a screen such as a tablet or television, through a flat surface such as a wall, window, or chalkboard, and / or through any other physical object that the user may designate to present virtual content. For example, the second wearable extended reality device may receive virtual content from the first user and then display the content through the second device and / or through a television or whiteboard.

[0193] In some embodiments, the first virtual content may be associated with at least one private virtual object and at least one public virtual object, and causing the virtual content to be transmitted for display by the second wearable extended reality device may include transmitting the at least one public virtual object and avoiding the transmission of the at least one private virtual object. As described herein, a virtual object may refer to a virtual display screen, a virtual content item, a virtual two-dimensional object, a virtual three-dimensional object, a document, a media item, a photo, a video, a virtual character, user-generated content, and / or a component of an enumerated object (e.g., a volume or brightness control bar). Furthermore, a public virtual object may include a virtual object that may be shared for viewing by other wearable extended reality device users and may not be subject to any restrictions on sharing with other users. In contrast, a private virtual object may include virtual content that the wearable extended reality device user does not want to and / or is not permitted to share with any other users or any other users not part of a predefined group. In some examples, a user may configure which virtual objects are public and which are private, or different privacy levels may be configured for different virtual objects. In other examples, whether a virtual object is private or public may be determined automatically, for example, based on an analysis of the virtual object's visual and / or non-visual characteristics.

[0194] In the above embodiments, a user can specify that some or all portions of the virtual content cannot be shared with another user. For example, if the virtual content is visualized as a screen, the screen can have both public and private windows. The public and private windows can each include virtual objects. Thus, in some disclosed embodiments, the virtual content can be displayed with only the public virtual objects visible to a second wearable extended reality device user, while the private virtual content remains hidden. Virtual content intended to be shared between wearable extended reality device users can consist of a variety of virtual objects, some private and some public.

[0195] As an example, the virtual content may include a table or graph showing financial information associated with a user, and the user may not want to share the financial information with other users. The user may designate the financial information in the virtual content as a private virtual object, thus preventing other users from viewing the virtual content. In another example, the virtual content may include a document with confidential exhibits. The user may designate the exhibits as private virtual objects, thus preventing other users from viewing the virtual object.

[0196] In some embodiments, virtual content items may be shared among users of the same level of privacy without requiring user approval. The level of privacy may refer to which virtual content items can be or are shared with which specific wearable extended reality device users. The level of privacy may be established by dividing other users' virtual content into one or more groups, with each group being allowed to visualize some or all of the virtual content. The number of levels of privacy may be configured based on user preferences. For example, some of the virtual content may be designated as content that can be shared with all other users. This may correspond to a low level of privacy. In another example, a portion of the virtual content may be designated as content that can be shared with users that may be related to the user sharing the virtual content. This relationship may be based on family or professional relationships. For example, a user may designate some or all of the virtual content as content that can be shared only with immediate family members (e.g., spouse, children, or parents) or users within a specific professional group (e.g., office team or department). This may correspond to a medium level of privacy. As another example, a user may designate some or all of the virtual content as content that may only be viewed by that user and not by other users. This may represent a high level of privacy. Although only three privacy levels (e.g., low, medium, and high) are described herein, the present disclosure is not so limited, and it is contemplated that any number of privacy levels may be established consistent with embodiments of the present disclosure.

[0197] In some embodiments, a virtual content item may require user approval before being shared at a privacy level lower than the current privacy level of the visual content item. In some examples, the determination of whether a virtual display screen needs to be shared may be based on factors related to the virtual size of the virtual display screen, the orientation of the virtual display screen, the location of the virtual display screen, and the virtual screen configuration.

[0198] In some embodiments, various sharing levels may be applied. Some non-limiting examples of such sharing levels may include private (available only to the user), shared (available to people the user chooses to share with), public (available to anyone within a particular physical space), or any other combination of public, private, or shared. In one example, a particular physical space may have an owner, and only the owner of the particular physical space may share public content within that particular physical space. For example, a company may share public content within the company's office, or a user may share public content within the user's home.

[0199] In some examples, private content items, including but not limited to virtual display screens and widgets, may be shared with other users in a manner that does not enable identification of some types of items. For example, items such as text, images, fine details, or any other media that a user may not want to share may not be identified or visible to other users. As an example, some or all of the virtual content may appear blurred to some users but clearly visible to other users based on how a first wearable extended reality device user configures its device or privacy level. Other methods of obscuring the virtual content may also be used. In one example, when a first user approaches a second user, the first user can see the second user's private content, as if viewed through milky glass, but cannot see the details of the content. Thus, the second user's content may be protected from the first user, who may not have permission to view the private documents.

[0200] In some embodiments, a user may be communicated that particular content is about to be shared with other users, and the user may be provided with time to cancel the sharing before the particular content is shared. Additionally, a user may obtain information related to the user's objects being shared with other users, information related to other users' objects shared with the user, information related to these other users, and / or information related to various users' levels of sharing.

[0201] In some embodiments, a user can use gestures to select a virtual content item, such as a virtual display screen, document, or other media, to be shared and send it to a physical screen. Such physical screens may include, but are not limited to, a television, tablet, laptop, smartphone, smart board, board, a physical screen designated for content sharing, or any other physical screen. Gestures used by a user to share virtual content may include, for example, dragging, pinching, swiping, or any other hand movement. As a result, the virtual content item may be shared, presented on the physical screen, or manipulated in any other manner as desired by the user. In some examples, the physical screen may provide and / or guide a virtual shared space, e.g., with the screen boundaries as the shared area, and the virtual content item may be presented as a virtual layer and viewed by an extended reality device.

[0202] 11A and 11B illustrate sharing public virtual content 1110. This content may be shared to a physical screen 1114, here a television, as shown in FIG. 11B, for example, by using a gesture 1112 as shown in FIG. 11A. Such a gesture may be a hand wave, a pointing finger, or any body movement. In this embodiment, the virtual content shared to the physical screen may be public virtual content, allowing everyone to view the content, including those not wearing extended reality equipment.

[0203] In some examples, a user can move their virtual content item, such as a virtual display screen or a document, to one of their personal physical screens. Such personal screens may include a personal physical display screen, a smartphone screen, or a tablet screen. The user can move their content by using gestures. In response, the virtual content item may be moved to (i.e., presented using) the physical display screen, or, for example, without additional approval, the virtual content item may be virtually presented on the physical display screen while maintaining the same privacy level.

[0204] However, when sharing some content, additional authorization may be required. For example, a user may want to share private virtual content with another user that has a higher privacy level, i.e., is confidential. Such virtual content may be password protected, accessible only to specific users, or otherwise made private. In this scenario, the sharing user may be prompted to enter a password, contact a system administrator, or otherwise request permission to share the virtual content.

[0205] 12A and 12B illustrate sharing private virtual content 1210. Here, an extended reality device user can share their virtual content 1210 with their physical screen 1212 by using gesture 1214, as shown in FIG. 12B. This content remains at the same privacy level; i.e., no additional approval is required to share the private virtual content. In some examples, a user can initiate sharing of virtual content with a physical screen. Identification of one or more users exposed to the physical screen may be performed (e.g., based on location information associated with the users and / or based on analysis of an image of the room including the physical screen). Based on the identified one or more users, it may be determined whether the virtual content can be shared with the physical screen without additional approval, whether the virtual content can be shared with the physical screen with additional approval, and / or whether the virtual content cannot be shared with the physical screen. In one example, after virtual content is shared with a physical screen, (potential or actual) exposers of the physical screen to the additional users may be identified (e.g., based on location information associated with the additional users and / or based on an analysis of an image of the room including the physical screen), and in response to the identified exposers of the physical screen to the additional users, sharing of the virtual content with the physical screen may be stopped or a prompt may be provided to the users sharing the virtual content. Such prompts may include indicators of the exposers and / or additional users and / or may be configured to allow the users sharing the virtual content to continue or stop sharing the virtual content.

[0206] In some embodiments, the above-mentioned privacy settings may be configured as part of a user's default settings. In some embodiments, causing the first virtual content to be displayed by the second wearable extended reality device may include modifying the first virtual content according to default settings associated with the second wearable extended reality device. The default settings may be configured according to preferences of a user of the second wearable extended reality device and / or hardware available on the second wearable extended reality device and / or environmental conditions associated with the second wearable extended reality device. For example, the second wearable extended reality device may operate at a lower or higher resolution than the first wearable extended reality device, and the rendering resolution of the first virtual content may be modified accordingly. In another example, the second wearable extended reality device may operate in different ambient light conditions than the first wearable extended reality device, and the intensity and / or opacity of the rendering of the first virtual content may be modified to compensate for the different ambient light conditions. In yet another example, the second wearable extended reality device may enable a different viewing angle than the first wearable extended reality device, and the size of the first virtual content may be modified to adjust for the different viewing angle. In some examples, the default settings may refer to how the wearable extended reality device is configured for use or how the wearable extended reality device is used. In some embodiments, the default settings may be configured to hide personal information such as passwords or credit card numbers, or the default settings may include a user's preferred display parameters, i.e., where content is displayed or how large the content appears.The default settings may be customizable by a user of the wearable extended reality device and may include, for example, privacy settings, proximity settings, gesture settings, or any other settings that may determine how the wearable extended reality device presents and / or shares virtual content.

[0207] In the above embodiments, the privacy setting may determine the initial privacy level associated with any virtual content that may be shared with another user. For example, the default privacy setting may allow the virtual content to be shared with all users. As another example, the default privacy setting may prohibit the virtual content from being shared with any other users. As another example, the default privacy setting may allow the virtual content to be shared with a predefined group of users (e.g., the user's close relatives or colleagues on a particular team or department associated with the user).

[0208] In the above embodiments, the sharing settings can determine how far a user must be from another user before a sharing bubble is created, how large a predefined sharing ring is, and where this ring is presented or the default physical object through which virtual content is presented. In the above embodiments, the gesture settings can determine which default user movements indicate a sharing intent. For example, a wearable extended reality device user can configure gesture settings to automatically share content when the user waves their hand, but not when the user nods. Such privacy sharing and gesture settings can be adjusted at any time depending on the user's preferences and desired use of the wearable extended reality device.

[0209] Some disclosed embodiments may include receiving second virtual content from a second wearable extended reality device for display via the first wearable extended reality device. Receiving may include any form of transmission of the content. For example, the content may be received through network-based content communication or through direct communication of the content, such as through a Bluetooth, Wi-Fi, NFC, or other direct communication protocol connection between the associated extended reality devices or a modem. In some embodiments, the processor can receive virtual content from the second wearable extended reality device and instruct the first wearable extended reality device to display the virtual content. In some other examples, the processor can receive virtual content from the second wearable extended reality device and, for example, provide the virtual content to the first wearable extended reality device for display of the virtual content. As described elsewhere herein, there are many ways in which the first wearable extended reality device may receive virtual content from the second wearable extended reality device. For example, a second wearable extended reality device can share virtual content with a first wearable extended reality device via a sharing bubble or a sharing ring. As another example, a second wearable extended reality device can share virtual content with a first wearable extended reality device as soon as a link between the two devices is established.

[0210] Some disclosed embodiments may include displaying multiple versions of virtual content. In some embodiments, a first virtual content may include a two-dimensional virtual object, and a second virtual content may include alternative versions of the two-dimensional object. The alternative versions of the two-dimensional object may include, for example, a scaled-down version of the two-dimensional object, an enlarged version of the two-dimensional object, a two-dimensional object having a different position or orientation, a two-dimensional object having a different color, texture, or shading, or a two-dimensional object with some or all of its characteristics or functionality altered in some way. The alternative versions of the two-dimensional object may be generated based on the preferences of the viewing user.

[0211] For example, a two-dimensional object may be a virtual document, and an alternate version of the two-dimensional virtual object may be annotated or highlighted versions of the virtual document. As another example, a two-dimensional object may be a presentation, and an alternate version of the two-dimensional object may have modified colors, graphics, and / or shapes in the presentation. As another example, multiple versions of virtual content may consist of both public and private virtual objects. For example, a first user of a first wearable extended reality device may send a document to a second user, who may then share the document with a larger group. However, some of the comments and annotations on the virtual object may remain hidden based on the privacy and / or document sharing settings of the first or second user.

[0212] In some embodiments, the first virtual content can include a three-dimensional virtual object, and the second virtual content can include a two-dimensional virtual object associated with the three-dimensional object. When one wearable extended reality device user shares content with another user, such content can include multiple virtual objects. As described in the above embodiments, these virtual objects can include both two-dimensional and three-dimensional objects and can be subject to various privacy settings. The three-dimensional virtual object can be a design model or project mockup, and the two-dimensional virtual object associated with the three-dimensional object can be a list of comments on the model. As with the two-dimensional model, subsequent versions of the three-dimensional virtual object can be configured to have various privacy settings.

[0213] For example, a project mockup, such as a bridge, building, or other scale or architectural model, can be shared with another user as a three-dimensional virtual object. The project mockup can be associated with a list of comments regarding the size, length, width, and any of a myriad of other parameters associated with the project mockup, i.e., a two-dimensional virtual object. In addition to the two-dimensional objects associated with the three-dimensional object, a user can also include the three-dimensional virtual object in a modified version. For example, in the project mockup example, a wearable extended reality device user can share a first version of the mockup and also share a modified version of the model that may have more components, be a different size, or have a different texture than the first version.

[0214] In response to receiving virtual content from the second wearable extended reality device, some disclosed embodiments may include presenting the second virtual content received from the second wearable extended reality device via the first extended reality device. There are many ways in which virtual content can be shared between wearable extended reality device users. The content can be shared via a sharing bubble or a sharing ring. The second wearable extended reality device user can share the content with the first wearable extended reality device user via the sharing bubble or the sharing ring. In the above embodiments, the first wearable extended reality device user can present the content received from the second user. This content can be presented in countless ways, for example, through the second user's extended reality device, via a screen such as a tablet or television, or via a physical object such as a whiteboard or chalkboard. In one embodiment, the first user, after receiving the virtual content from the second user, can present the content via the first device or via a television or whiteboard. The first wearable extended reality device user can also present the content received from the second user to other users in the vicinity of the first user.

[0215] 13A and 13B show a sharing bubble in which a first wearable extended reality device 1310 receives second virtual content 1314 from a second wearable extended reality device 1312 via a sharing area 1316. Such virtual content may also be shared via a sharing ring. FIG. 13B shows an example of presenting received content 1314, where the first wearable extended reality device 1310 can present the second virtual content 1314 received from the second wearable extended reality device 1312. The virtual content may be presented via a screen, such as a wearable extended reality device, a tablet, or a television, or any other physical object.

[0216] 14 shows a flowchart illustrating an example method 1410 of coordinating between a first wearable extended reality device and a second wearable extended reality device to display various virtual content. Method 1410 may include step 1412, in which at least one processor may establish a link between the first wearable extended reality device and the second wearable extended reality device, e.g., for communication between the two devices or for control of the two devices. Method 1410 may also include step 1414, in which first virtual content may be presented to the first wearable extended reality device before, after, or simultaneously with establishing the link with the second wearable extended reality device. Method 1410 may also include step 1416, in which the first virtual content may be presented to the first wearable extended reality device before, after, or simultaneously with establishing the link with the second wearable extended reality device. Method 1410 may also include step 1418, in which the first virtual content in some embodiments may be transmitted to a second wearable extended reality device for display. Method 1410 may also include step 1420, in which second virtual content may be received from the second wearable extended reality device. Method 1410 may include step 1422, in which the first wearable extended reality device may display the second virtual content.

[0217] Some disclosed embodiments may include operations for providing situational awareness to a user of a wearable extended reality device. Situational awareness may include one or more of perceiving elements in an environment, understanding a current situation, predicting future conditions, or any ability to identify, process, or understand elements of information about what is occurring in an environment. For example, situational awareness may include understanding one or more of a location, area, point, geography, region, scene, setting, site, surroundings, topography, section, angle, slope, dimension, amount, width, capacity, content, diameter, height, intensity, length, size, ratio, range, volume, breadth, amplitude, age, date, presence, time, moment, occasion, or any other condition associated with any portion of the environment. In some examples, situational awareness may include understanding the status of another person in the user's environment. For example, situational awareness may include understanding whether another person is engaging with virtual content (such as a virtual display, a virtual avatar, etc.).

[0218] Some embodiments may display virtual content through the first wearable extended reality device. The term virtual content may include any type of data representation that can be displayed to a user, such as through an extended reality device or other presentation device. Virtual content may include virtual objects, non-animated virtual content, animated virtual content configured to change over time or in response to a trigger, virtual two-dimensional content, virtual three-dimensional content, virtual overlays on portions of a physical environment or on physical objects, virtual additions to a physical environment or object, virtual promotional content, virtual representations of physical objects, virtual representations of a physical environment, virtual documents, virtual characters or personas, virtual computer screens, virtual widgets, or any other format for virtually displaying information. Consistent with this disclosure, virtual content may include any visual presentation rendered by a computer or processing device. In one embodiment, virtual content may include virtual objects, which are visual presentations rendered by a computer within a limited area and configured to represent a particular type of object (such as a non-animated virtual object, an animated virtual object, a virtual fixture, a virtual decorative object, a virtual widget, or other virtual representation). The rendered visual presentation can be changed to reflect changes to a status object or a change in the viewing angle of an object, for example, to mimic a change in the appearance of a physical object.

[0219] Some embodiments may include detecting a second wearable extended reality device in proximity to a first wearable extended reality device. Proximity may refer to proximity, adjacency, simultaneity, contiguity, closeness, or any other state or condition of being nearby. Proximity may be measured by any of capacitive, inductive, magnetic, or optical effects. Proximity may also be detected using, for example, one or more of radar, sonar, ultrasonic, fiber optic, or Hall effect technology, reflection of ionizing radiation, or any other technology capable of detecting the presence of a nearby object. Proximity may include distance, direction, or a combination of distance and direction. In other examples, the locations of the first and second wearable extended reality devices may be based on positioning data corresponding to the two. In one example, the location may be determined using indoor or outdoor positioning sensors included in each of the two wearable extended reality devices, from locating the two wearable extended reality devices in an image (e.g., an image captured using image sensors in the environment of the two wearable extended reality devices and analyzed using an object detection algorithm to locate the wearable extended reality devices), etc. In another example, an image captured using an image sensor included in a first wearable extended reality device may be analyzed using an object detection algorithm to detect the second wearable extended reality device, and the distance between the two wearable extended reality devices may be determined based on, for example, the size of the second wearable extended reality device in the captured image, thereby determining whether the second wearable extended reality device is in proximity to the first wearable extended reality device.In another example, an image captured using an image sensor included in a second wearable extended reality device may be analyzed using an object detection algorithm to detect the first wearable extended reality device, and the second wearable extended reality device may be considered to be in proximity to the first wearable extended reality device when the first wearable extended reality device is detected in the captured image (and therefore within the line of sight of the second wearable extended reality device).

[0220] In some embodiments, detecting the second wearable extended reality device in proximity to the first wearable extended reality device may be continuous. In some embodiments, detecting the second wearable extended reality device in proximity to the first wearable extended reality device may occur at regular or irregular time intervals. In some embodiments, detecting the second wearable extended reality device in proximity to the first wearable extended reality device may be triggered by an input. The input may be received from a user, a system, or any other source that may trigger proximity detection.

[0221] In particular embodiments, detecting a second wearable extended reality device in proximity to a first wearable extended reality device may include detecting that the second wearable extended reality device is within a particular distance, width, height, radius, or other measure of separation from the first wearable extended reality device. For example, detecting a second wearable extended reality device in proximity to a first wearable extended reality device may include detecting that the second wearable extended reality device is within 1 foot, 2 feet, 5 feet, or any distance from the first extended reality device. The measure of separation used to detect proximity may be input by a user, the system, or determined from any other source. For example, a user may input a value of 5 feet to trigger proximity detection. In another embodiment, the system may include a default setting for triggering proximity detection when a distance of 5 feet is detected between the first wearable extended reality device and the second wearable extended reality device. In one example, the measure of separation used to detect proximity may be different for different directions (e.g., one measure of separation for horizontal distance and a smaller measure of separation for vertical distance, or in another example, one measure of separation for upward distance and a smaller measure of separation for downward distance). In some examples, a second wearable extended reality device may be considered to be in proximity to a first wearable extended reality device when the two wearable extended reality devices are in the same defined space (same room, same apartment, same office, same building, etc.). In some examples, a second wearable extended reality device may be considered to be in proximity to a first wearable extended reality device when the first wearable extended reality device is within the line of sight of the second wearable extended reality device.

[0222] In some embodiments, proximity may be detected based on an arbitrary reference point on each of the first wearable extended reality device and the second wearable extended reality device. For example, proximity may be detected based on a distance from an arbitrary physical location of the wearable extended reality device. Alternatively, proximity may be detected based on a distance from a virtual location of a virtual object presented through the wearable extended reality device. In another example, proximity may be detected based on a distance from a reference point within a specific range of the wearable extended reality device.

[0223] Some embodiments may include establishing a link between a first wearable extended reality device and a second wearable extended reality device. The link may include one or more of physical or non-physical attachment, or any other connection mode. In some embodiments, the link may be established using physical attachment, including, for example, a wire, a cable, or any other type of attachment means that requires a tangible connection between the first wearable extended reality device and the second wearable extended reality device. In some embodiments, the link may be established using non-physical means and may include wireless communication, such as Bluetooth, ZigBee, Wi-Fi, or any other technology that implements the transfer of information between two or more points without using electrical conductors as a medium for performing the transfer.

[0224] Furthermore, the link may be a direct or indirect link between the first wearable extended reality device and the second wearable extended reality device. A direct link between the first wearable extended reality device and the second wearable extended reality device may be an unobstructed connection between the first wearable extended reality device and the second wearable extended reality device. Alternatively, an indirect link between the first wearable extended reality device and the second wearable extended reality device may be an obstructed connection between the first wearable extended reality device and the second wearable extended reality device. This type of link may include a connection connecting a system connected to one or both of the first wearable extended reality device and the second wearable extended reality device. Such a system may include, for example, a laptop, a smartphone, or any other type of non-wearable device that can be connected to a wearable extended reality device. In one example, the link between the first wearable extended reality device and the second wearable extended reality device may include a direct and / or indirect communication link between the two extended reality devices. In another example, a system (such as a centralized system) can communicate with each of the first and second wearable extended reality devices to, for example, provide content for presentation to each of the two wearable extended reality devices, receive location and / or orientation information associated with the wearable extended reality devices, etc. The link between the first wearable extended reality device and the second wearable extended reality device may include a linkage within a data structure or database of the system or maintained by the system.In some examples, establishing a link between a first wearable extended reality device and a second wearable extended reality device may refer to a linkage in a data structure and / or database, for example, in a system that coordinates an extended reality environment, a system that communicates with both the first and second wearable extended reality devices, a system that provides content for presentation to both the first and second wearable extended reality devices, etc.

[0225] Some embodiments may include transmitting data representing at least a portion of the virtual content in an obscured form to a second wearable extended reality device, the obscured form providing an indication of a location of the at least a portion of the virtual content in three-dimensional space without revealing the substance of the virtual content in the obscured form. In one example, the data may be transmitted from a first wearable extended reality device to a second wearable extended reality device, e.g., through a direct or indirect communication link. In this example, the first wearable extended reality device may process at least a portion of the virtual content to generate the obscured form, e.g., as described below. In another example, the data may be transmitted to the second wearable extended reality device from a computerized system (e.g., a system that adjusts the extended reality environment, a system that communicates with both the first and second wearable extended reality devices, a system that provides content for presentation to both the first and second wearable extended reality devices, etc.). In this example, the computerized system can process at least a portion of the virtual content to generate an obfuscated form, e.g., as described below. The obfuscated form can include presenting any portion of the virtual content in a manner that obfuscates that portion of the virtual content. In particular embodiments, the obfuscated form, which provides an indication of the location of at least a portion of the virtual content in three-dimensional space without revealing the identity of the virtual content, can include presenting at least a portion of the virtual content by masking, redacting, omitting, pixilation, blurring, or any other means that at least partially conceals any portion of the virtual content. In some examples, the obfuscated form can provide a visual indication of the location of at least a portion of the virtual content in three-dimensional space and / or a visual indication of the type of at least a portion of the virtual content without revealing at least one detail of at least a portion of the virtual content.For example, the obscured form may provide a visual indication that at least a portion of the virtual content includes textual content without revealing the exact words and / or characters of the textual content. In another example, the obscured form may provide a visual indication that at least a portion of the virtual content includes graphical content without revealing the exact images and / or graphics of the graphical content.

[0226] The functionality of the at least one processor as defined by the instructions is illustrated in Figures 15 and 16. For example, Figure 15 is a diagram of an exemplary environment including multiple users using multiple wearable extended reality devices, consistent with some embodiments of the present disclosure. As shown in Figure 15, a first user 1514 of a first wearable extended reality device 1515 may be located at a first location 1510. A second user 1516 of a second wearable extended reality device 1517 may be located at a second location 1512 that is not proximate (e.g., at a distance greater than a selected threshold, not in the same defined space, not in the same room, not within line of sight, etc.) to the first location 1510 of the first user 1514. The first user 1514 may be presented with first content 1522 in a first virtual display 1518, and the second user 1516 may be presented with second content 1524 in a second virtual display 1520. Because the first wearable extended reality device 1515 is not in proximity to the second wearable extended reality device 1517 , none of the first virtual content 1522 can be transmitted to the second user 1516 .

[0227] 16 is a diagram of an example virtual display including a portion of the virtual display provided in obscured form, consistent with some embodiments of the present disclosure. As shown in FIG. 16 , a first user 1614 of a first wearable extended reality device 1615 may be located at a first location 1610. A second user 1616 of a second wearable extended reality device 1617 may be located at a second location 1612 that is proximate to the first user's 1614 first location 1610 (e.g., at a distance less than a selected threshold, in the same defined space, in the same room, within the line of sight of the wearable extended reality device 1615, etc.). In this example, the first user 1614 may be presented with first content 1622 in a first virtual display 1618, and the second user 1616 may be presented with second content 1624 in a second virtual display 1620. Here, due to the proximity of the second wearable extended reality device 1617 to the first wearable extended reality device 1615, the portion 1626 of the first virtual content 1622 may be transmitted to the second wearable extended reality device 1617. Additionally, the portion 1626 of the first virtual content 1622 may be presented to the second wearable extended reality device 1617 in an obscured form by being blurred, such that the blurred image provides an indication of the location of the portion 1626 of the first virtual content 1622 in three-dimensional space without revealing the substance of the virtual content 1622.

[0228] In some embodiments, the virtual content displayed through the first wearable extended reality device may include different types of content, and the operations further include presenting, via the second wearable extended reality device, a type associated with the obfuscated information without revealing the substance of the content. The different types of content may include content that varies in format, presentation, shape, size, color, or any other aspect. In some embodiments, the different types of content may include, for example, word processing content, game content, spreadsheet content, database content, graphics content, or web browser content. As an example of a presentation consistent with such embodiments, the virtual content displayed through the first wearable extended reality device may include web browser-type content. Thus, when data representing at least a portion of the virtual content is transmitted to the second wearable extended reality device in obfuscated form, a web browser type associated with the obfuscated form is presented via the second display.

[0229] In some embodiments, the virtual content displayed through the first wearable extended reality device may include multiple virtual screens, and the virtual content presented in an obscured form through the second wearable extended reality device may further provide an indication of the size and orientation of the multiple virtual screens without revealing the actual content associated with the multiple virtual screens. A virtual screen (also referred to herein as a virtual display) may include any bounded shape or presentation area containing virtual content, such as a rectangular window containing multiple virtual objects. The multiple virtual screens may include two or more bounded shapes or presentation areas, each containing virtual content. The multiple virtual screens may have different sizes, orientations, display settings, and other visual characteristics. One or more of these visual characteristics may be obfuscated when the multiple virtual screens are presented to the second wearable extended reality device. For example, the virtual content displayed through the first wearable extended reality device may include one virtual screen that is square-shaped and another virtual screen that is rectangular-shaped. In such an embodiment, when the obscured form is presented through a second wearable extended reality device, the square and rectangular shapes of the corresponding virtual screen may be displayed to a user of the second wearable extended reality device, but the content of the square and rectangular shapes may be obscured. Figure 17 is an example diagram of ...

Claims

1. 1. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations for presenting virtual content to a plurality of viewers, the operations comprising: receiving sensor data indicative of a plurality of wearable extended reality devices located within a room; receiving a command to share a virtual object with the plurality of wearable extended reality devices; analyzing the sensor data to determine a first location within the room of a first wearable extended reality device, a second location within the room of a second wearable extended reality device, and a third location within the room of a third wearable extended reality device; determining a position for displaying the virtual object within the room based on the determined first location, the determined second location, and the determined third location; generating a first representation of the virtual object at the determined location through the first wearable extended reality device, the first representation being rendered from a first perspective; and generating a second representation of the virtual object at the determined location through the second wearable extended reality device, the second representation being rendered from a second perspective different from the first perspective; and generating a third representation of the virtual object at the determined location through the third wearable extended reality device, the third representation being rendered from a third perspective different from the first perspective and the second perspective; and A non-transitory computer readable medium comprising:

2. 2. The non-transitory computer-readable medium of claim 1 , wherein the operations further include analyzing the sensor data to determine that at least one of the first wearable extended reality device, the second wearable extended reality device, or the third wearable extended reality device is virtually located within the room, and the determined location of the at least one wearable extended reality device virtually located within the room reflects a location of an avatar of at least one user.

3. 2. The non-transitory computer-readable medium of claim 1 , wherein the determined position for displaying the virtual object includes a determined orientation of the virtual object within the room, and the operations further include analyzing additional sensor data to determine that at least two of the first wearable extended reality device, the second wearable extended reality device, or the third wearable extended reality device have changed orientation, and adjusting the determined orientation of the virtual object is based on the change in orientation.

4. 2. The non-transitory computer-readable medium of claim 1, wherein the determined position for displaying the virtual object includes a determined location of the virtual object, and the operations further include analyzing additional sensor data to determine that at least two of the first wearable extended reality device, the second wearable extended reality device, or the third wearable extended reality device have changed location, and adjusting the determined location of the virtual object is based on the change in location.

5. 2. The non-transitory computer-readable medium of claim 1, wherein the operations further include determining an identity of at least one of a user of the first wearable extended reality device, a user of the second wearable extended reality device, or a user of the third wearable extended reality device, and determining the location for displaying the virtual object is further based on the identity of the at least one user.

6. 2. The non-transitory computer-readable medium of claim 1, wherein the operations further include determining physical characteristics of at least one user of the first wearable extended reality device, the second wearable extended reality device, or the third wearable extended reality device, and determining the location for displaying the virtual object is further based on the physical characteristics of the at least one user.

7. 2. The non-transitory computer-readable medium of claim 1, wherein the operations further include determining a layout of the room including one or more physical locations of furniture within the room, and wherein the location for displaying the virtual object is further determined based on the layout of the room.

8. 2. The non-transitory computer-readable medium of claim 1, wherein the operations further include determining lighting conditions in the room, and wherein the location for displaying the virtual object is further determined based on the lighting conditions.

9. 2. The non-transitory computer-readable medium of claim 1, wherein the operations further include determining a type of the virtual object, and wherein the location for displaying the virtual object is further determined based on the type of the virtual object.

10. 2. The non-transitory computer-readable medium of claim 1 , wherein the operations further include analyzing the sensor data to identify physical objects in the room, and wherein the location for displaying the virtual object is determined such that none of the first representation, the second representation, and the third representation are occluded by the physical objects.

11. 2. The non-transitory computer-readable medium of claim 1 , wherein the operations further include analyzing the sensor data to identify physical objects in the room, and wherein the position for displaying the virtual object is determined such that none of the first representation, the second representation, and the third representation occludes the physical object.

12. 2. The non-transitory computer-readable medium of claim 1 , wherein the first representation of the virtual object, the second representation of the virtual object, and the third representation of the virtual object are associated with a single version of the virtual object, and the operations further include detecting changes to the virtual object introduced by a user of the first wearable extended reality device, and updating the second representation and the third representation to reflect the changes to the virtual object introduced by the user of the first wearable extended reality device.

13. 2. The non-transitory computer-readable medium of claim 1, wherein the first representation of the virtual object, the second representation of the virtual object, and the third representation of the virtual object are associated with different versions of the virtual object, and the operations further include obtaining a profile associated with a user of the first wearable extended reality device, a profile associated with a user of the second wearable extended reality device, and a profile associated with a user of the third wearable extended reality device, and determining a personalized version of the virtual object based on the profile associated with each user.

14. 2. The non-transitory computer-readable medium of claim 1, wherein while the virtual object is displayed at the determined location through the first wearable extended reality device and the second wearable extended reality device, the operations further include receiving additional sensor data indicating a change to a status of the third wearable extended reality device, and repositioning the virtual object based on the determined first location of the first wearable extended reality device and the determined second location of the second wearable extended reality device.

15. 2. The non-transitory computer-readable medium of claim 1 , wherein while the virtual object is displayed at the determined location, the operations further include receiving additional sensor data indicating that a fourth wearable extended reality device is within the room, determining a fourth location of the fourth wearable extended reality device, and repositioning the virtual object based on the determined first location, the determined second location, the determined third location, and the determined fourth location.

16. 2. The non-transitory computer-readable medium of claim 1, wherein the operations are performed by the first wearable extended reality device, and wherein generating the first display includes generating a display signal, generating the second display includes transmitting data reflecting the virtual object to the second wearable extended reality device, and generating the third display includes transmitting data reflecting the virtual object to the third wearable extended reality device.

17. 2. The non-transitory computer-readable medium of claim 1, wherein the virtual object is a virtual display configured to present text entered using an input device, and determining the location for displaying the virtual display is further based on at least one of a location of the input device or an orientation of the input device.

18. 2. The non-transitory computer-readable medium of claim 1, wherein the virtual object is a virtual display configured to present text entered using a physical keyboard, the physical keyboard being positioned on a physical surface, and wherein determining the location for displaying the virtual display is further based on characteristics of the physical surface.

19. 1. A method for presenting virtual content to a plurality of viewers, the method comprising: receiving sensor data indicative of a plurality of wearable extended reality devices located within a room; receiving a command to share a virtual object with the plurality of wearable extended reality devices; analyzing the sensor data to determine a first location within the room of a first wearable extended reality device, a second location within the room of a second wearable extended reality device, and a third location within the room of a third wearable extended reality device; determining a position for displaying the virtual object within the room based on the determined first location, the determined second location, and the determined third location; generating a first representation of the virtual object at the determined location through the first wearable extended reality device, the first representation being rendered from a first perspective; and generating a second representation of the virtual object at the determined location through the second wearable extended reality device, the second representation being rendered from a second perspective different from the first perspective; and generating a third representation of the virtual object at the determined location through the third wearable extended reality device, the third representation being rendered from a third perspective different from the first perspective and the second perspective; and A method comprising:

20. 1. A system for presenting virtual content to a plurality of viewers, the system comprising: At least one processor, the at least one processor comprising: receiving sensor data indicative of a plurality of wearable extended reality devices located within a room; receiving a command to share a virtual object with the plurality of wearable extended reality devices; analyzing the sensor data to determine a first location within the room of a first wearable extended reality device, a second location within the room of a second wearable extended reality device, and a third location within the room of a third wearable extended reality device; determining a position for displaying the virtual object within the room based on the determined first location, the determined second location, and the determined third location; generating a first representation of the virtual object at the determined location through the first wearable extended reality device, the first representation being rendered from a first perspective; and generating a second representation of the virtual object at the determined location through the second wearable extended reality device, the second representation being rendered from a second perspective different from the first perspective; and generating a third representation of the virtual object at the determined location through the third wearable extended reality device, the third representation being rendered from a third perspective different from the first perspective and the second perspective; and The system is programmed to:

Images