Content Sharing in Extended Reality

Abstract

A system, method, and non-transitory computer-readable medium configured to enable content sharing between users of wearable extended reality devices are provided. In one implementation, the computer-readable medium may be configured to include instructions for causing at least one processor to establish a link between a first wearable extended reality device and a second wearable extended reality device. The first wearable extended reality device may display a first virtual content. The second wearable extended reality device may obtain a command to display the first virtual content via the second wearable extended reality device, and in response, the content may be transmitted and displayed via the second extended reality device. Additionally, the first wearable extended reality device may receive a second virtual content from the second wearable extended reality device and display the second virtual content via the first wearable extended reality device.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 147,051, filed on February 8, 2021; U.S. Provisional Patent Application No. 63 / 157,768, filed on March 7, 2021; U.S. Provisional Patent Application No. 63 / 173,095, filed on April 9, 2021; U.S. Provisional Patent Application No. 63 / 213,019, filed on June 21, 2021; U.S. Provisional Patent Application No. 63 / 215,500, filed on June 27, 2021; U.S. Provisional Patent Application No. 63 / 216,335, filed on June 29, 2021; U.S. Provisional Patent Application No. 63 / 226,977, filed on July 29, 2021; U.S. Provisional Patent Application No. 63 / 300,005, filed on January 16, 2022; U.S. Provisional Patent Application No. 63 / 307,207, filed on February 7, 2022; U.S. Provisional Patent Application No. 63 / 307,203, filed on February 7, 2022; and U.S. Provisional Patent Application No. 63 / 307,217, filed on February 7, 2022, all of which are hereby incorporated by reference in their entirety.

[0002] Technical Field The present disclosure generally relates to the field of extended reality. More specifically, the present disclosure relates to systems, methods, and devices for providing productivity applications using an extended reality environment.

Background Art

[0003] For years, PC users have faced a productivity dilemma of either limiting mobility (when choosing a desktop computer) or limiting screen size (when choosing a laptop computer). One partial solution to this dilemma is to use a docking station. A docking station is an interface device for connecting a laptop computer to other devices. By plugging the laptop computer into the docking station, laptop users can enjoy the improved visibility provided by a larger monitor. However, since the large monitor is fixed, although improved, the user's mobility is still limited. For example, even a laptop user with a docking station does not have the freedom to use two 32-inch screens where the user desires.

[0004] Some of the disclosed embodiments are directed to providing a new approach for solving the productivity dilemma, which is to use extended reality (XR) to provide a virtual desktop-like screen, thereby providing a mobile environment that enables users to experience the comfort of a stationary workspace wherever the user desires. SUMMARY OF THE INVENTION PROBLEM TO BE SOLVED BY THE INVENTION

[0005] Embodiments consistent with the present disclosure provide systems, methods, and devices for providing and supporting productivity applications using an extended reality environment. MEANS FOR SOLVING THE PROBLEM

[0006] Some of the disclosed embodiments may include a system, a method, and a non-transitory computer-readable medium for enabling content sharing among users of wearable extended reality devices. Some of these embodiments include establishing a link between a first wearable extended reality device and a second wearable extended reality device, presenting first virtual content through the first wearable extended reality device, obtaining a first command for displaying the first virtual content via the second wearable extended reality device, in response to the first command, causing the first virtual content to be transmitted for display to the second wearable extended reality device, receiving second virtual content from the second wearable extended reality device for display via the first wearable extended reality device, and presenting the second virtual content received from the second wearable extended reality device via the first wearable extended reality device.

[0007] Some of the disclosed embodiments may include a system, a method, and a non-transitory computer-readable medium for providing situation awareness to a user of a wearable extended reality device. Some of these embodiments include causing virtual content to be displayed through a first wearable extended reality device, detecting a second wearable extended reality device in proximity to the first wearable extended reality device, establishing a link between the first wearable extended reality device and the second wearable extended reality device, and transmitting data representing at least a portion of the virtual content in an obfuscated form to the second wearable extended reality device, the obfuscated form providing an indication of the location of at least a portion of the virtual content in three-dimensional space without revealing the identity of the virtual content in the obfuscated form.

[0008] Some of the disclosed embodiments may include a system, method, and non-transitory computer-readable medium for associating a virtual whiteboard with a physical space. Some of these embodiments include receiving an indication of the location of a first wearable extended reality device via a wireless network, performing a lookup in a repository of virtual whiteboards and at the location of the virtual whiteboard to determine that the location of the first wearable extended reality device corresponds to the location of a particular virtual whiteboard, transmitting data corresponding to the content of the particular virtual whiteboard to the first wearable extended reality device, thereby enabling a first user of the first wearable extended reality device to virtually view the content of the particular virtual whiteboard and add virtual content to the particular virtual whiteboard, receiving virtual content added by the first user during a first period, and during a second period after the first wearable extended reality device is no longer at the location of the particular virtual whiteboard, receiving an indication via the wireless network that a second wearable extended reality device is at the location of the particular virtual whiteboard, and transmitting data corresponding to the content of the particular virtual whiteboard and the added content to the second wearable extended reality device, thereby enabling a second user of the second wearable extended reality device to view the content and the added content while the first user is not at the location of the particular virtual whiteboard.

[0009] Some of the disclosed embodiments may include a system, a method, and a non-transitory computer-readable medium for transferring virtual content to a physical display device. Some of these embodiments are to present an extended reality environment within a room via a wearable extended reality device, wherein the wearable extended reality device is configured to be paired with a plurality of display devices located within the room, and each display device is associated with a unique network identifier, presenting, receiving an input associated with the wearable extended reality device to cause a specific virtual object within the extended reality environment to be presented on a target display device, receiving image data from an image sensor associated with the wearable extended reality device, wherein the image data shows the target display device, analyzing the image data to identify the target display device, determining the network identifier of the target display device at the time when the target display device is identified, establishing a communication link with the target display device using the determined network identifier of the target display device, and transmitting data representing the specific virtual object to the target display device, wherein the transmitted data enables the target display device to present the specific virtual object.

[0010] Some of the disclosed embodiments may include a system, method, and non-transitory computer-readable medium for simulating user interactions with shared content. Some of these embodiments include establishing a communication channel for sharing content and user interactions between a first wearable extended reality device and at least one second wearable extended reality device, and transmitting, to the at least one second wearable extended reality device, first data representing an object associated with the first wearable extended reality device, the first data enabling a virtual representation of the object to be displayed through the at least one second wearable extended reality device, receiving image data from an image sensor associated with the first wearable extended reality device, detecting, within the image data, at least one user interaction associated with the object, the at least one user interaction including a human hand pointing to a particular portion of the object, and transmitting, to the at least one second wearable extended reality device, second data indicating an area of a particular portion of the object based on the detection of the at least one user interaction within the image data.

[0011] Some of the disclosed embodiments may include a system, method, and non - transitory computer - readable medium for managing content placement in an extended reality environment. Some of these embodiments include receiving, from an entity, a request to place virtual content at a particular geographical location within at least one shared extended reality environment that includes a plurality of virtual objects, where the plurality of virtual objects are viewable by a plurality of wearable extended reality devices registered in the at least one shared extended reality environment, obtaining information associated with the request, accessing a plurality of content placement rules that define geographical restrictions for extended reality environment content placement, determining, based on the obtained information, that the request from the entity to place virtual content at the particular geographical location corresponds to a particular content placement rule, implementing the particular content placement rule to prevent the display of virtual content at the particular geographical location within the at least one shared extended reality environment by at least some of the plurality of wearable extended reality devices when the conditions of the particular content placement rule are not met, and implementing the particular content placement rule to enable the display of virtual content at the particular geographical location within the at least one shared extended reality environment by at least some of the plurality of wearable extended reality devices when the conditions of the particular content placement rule are met.

[0012] Some of the disclosed embodiments may include a system, a method, and a non-transitory computer-readable medium for presenting virtual content to a plurality of viewers. Some of these embodiments include receiving sensor data indicating a plurality of wearable extended reality devices located within a room, receiving commands for sharing virtual objects with the plurality of wearable extended reality devices, analyzing the sensor data to determine a first location within the room of a first wearable extended reality device, a second location within the room of a second wearable extended reality device, and a third location within the room of a third wearable extended reality device, determining a position for displaying a virtual object within the room based on the determined first location, the determined second location, and the determined third location, causing a first display of the virtual object at the position determined through the first wearable extended reality device, the first display being rendered from a first perspective, causing a second display of the virtual object at the position determined through the second wearable extended reality device, the second display being rendered from a second perspective different from the first perspective, and causing a third display of the virtual object at the position determined through the third wearable extended reality device, the third display being rendered from a third perspective different from the first perspective and the second perspective.

[0013] Some of the disclosed embodiments may include a system, method, and non-transitory computer-readable medium for performing virtual coloring marking on an object. Some of these embodiments include receiving an indication of an object, receiving an image of a person's hand holding a physical marking tool from an image sensor, detecting a color associated with the marking tool within the image, receiving image data from the image sensor indicating the movement and location of the tip of the marking tool, determining from the image data when the location of the tip corresponds to a location on the object, and generating a virtual marking on the object at the corresponding location with the detected color.

[0014] Some of the disclosed embodiments may include a system, method, and non-transitory computer-readable medium for enabling wearable extended reality devices to share virtual content. Some of these embodiments include generating a visual code that reflects a first physical location of a mobile device, the visual code being configured to be read by a plurality of wearable extended reality devices, presenting the visual code on a display of the mobile device for detection by the plurality of wearable extended reality devices, thereby enabling the plurality of wearable extended reality devices to share content in a common coordinate system when the visual code is detected, detecting a movement of the mobile device to a second physical location different from the first physical location, and changing the presentation of the visual code such that the visual code is unavailable for use in content sharing when the movement of the mobile device is detected.

[0015] In accordance with other disclosed embodiments, a non-transitory computer-readable storage medium can store program instructions that are executed by at least one processing device to perform any of the methods described herein.

[0016] The foregoing summary and the following detailed description are exemplary and explanatory only and are not restrictive of the claims.

[0017] The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate various disclosed embodiments. **Brief Description of the Drawings**

[0018]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37-1

Figure 37-2

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42A

Figure 42B

Figure 43

Figure 44

Figure 45

Figure 46

Figure 47

Figure 48

Figure 49

Figure 50

Figure 51

Figure 52

Figure 53

Figure 54

Figure 55

Figure 56

Figure 57

Figure 58

Figure 59

Figure 60

Figure 61

[0019] The following detailed description refers to the accompanying drawings. Whenever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. Although some exemplary embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or modifications may be made to the components shown in the drawings, and the exemplary methods described herein may be modified by substituting, rearranging, removing, or adding steps to the disclosed methods. Accordingly, the following detailed description is not limited to specific embodiments and examples, but includes the general principles described and illustrated herein, in addition to the general principles encompassed by the appended claims.

[0020] The present disclosure is directed to systems and methods for providing an extended reality environment to a user. The term "extended reality environment", which may also be referred to as "extended reality", "extended reality space", or "extended environment", refers to any type of combined real and virtual environment that is at least partially generated by computer technology, and the interaction between humans and machines. The extended reality environment can be a fully simulated virtual environment or a combined real and virtual environment that a user can perceive from various perspectives. In some examples, the user can interact with elements of the extended reality environment. One non-limiting example of an extended reality environment can be a virtual reality environment, also known as "virtual reality" or "virtual environment". An immersive virtual reality environment can be a simulated non-physical environment that provides the user with the perception of being present within the virtual environment. Another non-limiting example of an extended reality environment can be an augmented reality environment, also known as "augmented reality" or "augmented environment". An augmented reality environment can include a live direct or indirect view of a physical real-world environment enhanced with virtual computer-generated perceptual information, such as virtual objects with which the user can interact. Another non-limiting example of an extended reality environment is a mixed reality environment, also known as "mixed reality" or "mixed environment". A mixed reality environment can be a hybrid of a physical real-world and a virtual environment in which physical objects and virtual objects coexist and can interact in real time. In some examples, both augmented reality environments and mixed reality environments can include a combination of the real world and the virtual world, real-time interaction, and accurate 3D registration of virtual and real objects. In some examples, both augmented reality environments and mixed reality environments can include structured overlaid sensory information that can be added to the physical environment. In other examples, both augmented reality environments and mixed reality environments can include disruptive virtual content that can mask at least a portion of the physical environment.

[0021] In some embodiments, the system and method can provide an extended reality environment using extended reality devices. The term "extended reality device" can include any type of device or system that enables a user to perceive and / or interact with an extended reality environment. The extended reality device may enable a user to perceive and / or interact with the extended reality environment through one or more sensory modalities. Some non-limiting examples of such sensory modalities can include vision, hearing, touch, proprioception, and smell. An example of an extended reality device is a virtual reality device that enables a user to perceive and / or interact with a virtual reality environment. Another example of an extended reality device is an augmented reality device that enables a user to perceive and / or interact with an augmented reality environment. Yet another example of an extended reality device is a mixed reality device that enables a user to perceive and / or interact with a mixed reality environment.

[0022] Consistent with one aspect of the present disclosure, an extended reality device may be a wearable device, such as a head-mounted device, for example, smart glasses, smart contact lenses, a headset, or any other device worn by a human for the purpose of presenting extended reality to the human. Other extended reality devices may include holographic projectors or any other device or system capable of providing augmented reality (AR), virtual reality (VR), mixed reality (MR), or any immersive experience. Typical components of a wearable extended reality device may include a stereoscopic head-mounted display, a stereoscopic head-mounted sound system, head motion tracking sensors (such as gyroscopes, accelerometers, magnetometers, image sensors, structured light sensors, etc.), a head-mounted projector, eye tracking sensors, and at least one of the additional components described below. Consistent with another aspect of the present disclosure, the extended reality device may be a non-wearable extended reality device. Specifically, the non-wearable extended reality device may include a multi-projection environment device. In some embodiments, the extended reality device may be configured to change the viewing perspective of the extended reality environment in response to the movement of the user, and in particular, in response to the movement of the user's head. In one example, a wearable extended reality device can change the field of view of the extended reality environment in response to a change in the user's head pose, such as by changing the orientation of the space without changing the user's spatial position within the extended reality environment. In another example, a non-wearable extended reality device can change the spatial position of the user within the extended reality environment in response to a change in the position of the user in the real world, such as by changing the spatial position of the user within the extended reality environment without changing the direction of the field of view relative to the spatial position.

[0023] According to some embodiments, the extended reality device may include a digital communication device configured to perform at least one of receiving virtual content data configured to enable presentation of virtual content, transmitting virtual content for sharing with at least one external device, receiving context data from at least one external device, transmitting context data to at least one external device, transmitting usage data indicating usage of the extended reality device, and transmitting data based on information captured using at least one sensor included in the extended reality device. In additional embodiments, the extended reality device may include memory for storing at least one of virtual data configured to enable presentation of virtual content, context data, usage data indicating usage of the extended reality device, sensor data based on information captured using at least one sensor included in the extended reality device, software instructions configured to cause a processing device to present virtual content, software instructions configured to cause a processing device to collect and analyze context data, software instructions configured to cause a processing device to collect and analyze usage data, and software instructions configured to cause a processing device to collect and analyze sensor data. In additional embodiments, the extended reality device may include a processing device configured to perform at least one of rendering virtual content, collecting and analyzing context data, collecting and analyzing usage data, and collecting and analyzing sensor data. In additional embodiments, the extended reality device may include one or more sensors.One or more sensors may include one or more image sensors (e.g., configured to capture images and / or videos of a user of the device and / or the user's environment), one or more motion sensors (such as accelerometers, gyroscopes, magnetometers, etc.), one or more positioning sensors (GPS, outdoor positioning sensors, indoor positioning sensors, etc.), one or more temperature sensors (e.g., configured to measure the temperature of at least a part of the device and / or the environment), one or more contact sensors, one or more proximity sensors (e.g., configured to detect whether the device is currently being worn), one or more electrical impedance sensors (e.g., configured to measure the electrical impedance of a user), a gaze detector, an optical tracker, an electro-potential tracker (such as an electrooculogram (EOG) sensor), a video-based eye tracker, an infrared / near-infrared sensor, a passive light sensor, or one or more eye-tracking sensors such as any other technology capable of determining where or what a person is looking at or gazing at.

[0024] In some embodiments, the system and method can interact with an extended reality device using an input device. The term input device can include any physical device configured to receive input from a user or the user's environment and provide data to a computing device. The data provided to the computing device may be in digital and / or analog form. In one embodiment, the input device can store the input received from the user in a memory device accessible by a processing device, and the processing device can access the data stored for analysis. In another embodiment, the input device can provide data directly to the processing device, for example, via a bus or via another communication system configured to transfer data from the input device to the processing device. In some examples, the input received by the input device can include key presses, tactile input data, motion data, position data, gesture-based input data, direction data, or any other data for providing for computation. Some examples of input devices can include buttons, keys, keyboards, computer mice, touch pads, touch screens, joysticks, or any other mechanism through which input can be received. Another example of an input device can include an integrated computing interface device that includes at least one physical component for receiving input from a user. The integrated computing interface device can include at least a memory, a processing device, and at least one physical component for receiving input from a user. In one example, the integrated computing interface device can further include a digital network interface that enables digital communication with other computing devices. In one example, the integrated computing interface device can further include a physical component for outputting information to the user. In some examples, all components of the integrated computing interface device can be included in a single housing, but in other examples, the components can be distributed between two or more housings.Some non-limiting examples of physical components for receiving input from a user that may be included in an integrated computing interface device include at least one of buttons, keys, keyboards, touch pads, touch screens, joysticks, or any other mechanism or sensor through which computing information may be received. Some non-limiting examples of physical components for outputting information to a user include at least one of optical indicators (such as LED indicators), screens, touch screens, beepers, audio speakers, or any other audio, video, or tactile device that provides an output perceivable by a human.

[0025] In some embodiments, the image data may be captured using one or more image sensors. In some examples, the image sensors may be included in extended reality devices, wearable devices, wearable extended reality devices, input devices, the user's environment, and the like. In some examples, the image data may be read from memory, received from an external device, or generated (e.g., using a generative model). Some non-limiting examples of image data may include images, grayscale images, color images, 2D images, 3D images, videos, 2D videos, 3D videos, frames, scenes, data obtained from other image data, and the like. In some examples, the image data may be encoded in any analog or digital format. Some non-limiting examples of such formats may include raw format, compressed format, uncompressed format, irreversible format, reversible format, JPEG, GIF, PNG, TIFF, BMP, NTSC, PAL, SECAM, MPEG, MPEG-4 Part 14, MOV, WMV, FLV, AVI, AVCHD, WebM, MKV, and the like.

[0026] In some embodiments, the extended reality device can receive, for example, a digital signal from an input device. The term digital signal refers to a series of digital values that are discrete in time. The digital signal can represent, for example, sensor data, text data, audio data, video data, virtual data, or any other form of data that provides perceptible information. Consistent with the present disclosure, the digital signal can be configured to cause the extended reality device to present virtual content. In one embodiment, the virtual content can be presented in a selected orientation. In this embodiment, the digital signal can indicate the position and angle of the viewpoint in an environment such as an extended reality environment. Specifically, the digital signal can include an encoding of the position and angle in six degrees of freedom coordinates (e.g., forward / backward, up / down, left / right, yaw, pitch, and roll). In another embodiment, the digital signal can include an encoding of the position as three-dimensional coordinates (e.g., x, y, and z) and an encoding of the angle as a vector resulting from the encoded position. Specifically, the digital signal can indicate the orientation and angle of the presented virtual content in the absolute coordinates of the environment, for example, by encoding the yaw, pitch, and roll of the virtual content relative to a standard default angle. In another embodiment, the digital signal can indicate the orientation and angle of the presented virtual content with respect to the viewpoint of another object (e.g., a virtual object, a physical object, etc.), for example, by encoding the yaw, pitch, and roll of the virtual content with respect to the direction corresponding to the viewpoint or the direction corresponding to another object. In another embodiment, such a digital signal can include one or more projections of the virtual content, for example, in a form ready for presentation (e.g., an image, a video, etc.). For example, each of such projections can correspond to a specific orientation or a specific angle. In another embodiment, the digital signal can include a representation of the virtual content, for example, by encoding an object in a three-dimensional array of voxels, a polygon mesh, or any other form in which the virtual content can be presented.

[0027] In some embodiments, the digital signal can be configured to cause an extended reality device to present virtual content. The term virtual content can include any type of data representation that can be presented to a user by an extended reality device. Virtual content can include virtual objects, non-animated virtual content, animated virtual content configured to change over time or in response to a trigger, virtual 2D content, virtual 3D content, virtual overlays on a portion of the physical environment or on a physical object, virtual additions to the physical environment or physical objects, virtual promotional content, virtual representations of physical objects, virtual representations of the physical environment, virtual documents, virtual characters or personas, virtual computer screens, virtual widgets, or any other form for virtually presenting information. Consistent with the present disclosure, virtual content can include any visual presentation rendered by a computer or processing device. In one embodiment, the virtual content includes virtual objects that are visual presentations rendered by a computer within a limited area and configured to represent a particular type of object (such as non-animated virtual objects, animated virtual objects, virtual furnishings, virtual decorative objects, virtual widgets, or other virtual representations). The rendered visual presentation can be changed, for example, to mimic a change in the appearance of a physical object or to reflect a change to a status object or a change in the viewing angle of an object. In another embodiment, the virtual content can include a virtual display (also referred to herein as a "virtual display screen" or "virtual screen") such as a virtual computer screen, virtual tablet screen, or virtual smartphone screen that is configured to display information generated by an operating system, and the operating system can be configured to receive text data from a physical keyboard and / or virtual keyboard and display text content within the virtual display screen. In an example shown in FIG. 1, the virtual content can include a virtual environment that includes a virtual computer screen and a plurality of virtual objects.In some examples, the virtual display can be a virtual object that mimics and / or extends the functionality of a physical display screen. For example, the virtual display can be presented within an extended reality environment (such as a mixed reality environment, an augmented reality environment, a virtual reality environment, etc.) using an extended reality device. In one example, the virtual display can present content generated by a normal operating system that is presented equally on a physical display screen. In one example, text content input using a keyboard (e.g., using a physical keyboard, using a virtual keyboard, etc.) can be presented on the virtual display in real time as the text content is typed. In one example, a virtual cursor can be presented on the virtual display, and the virtual cursor can be controlled by a pointing device (such as a physical pointing device, a virtual pointing device, a computer mouse, a joystick, a touchpad, a physical touch controller, etc.). In one example, one or more windows of a graphical user interface operating system can be presented on the virtual display. In another example, the content presented on the virtual display can be interactive, i.e., it can change in response to user actions. In yet another example, the presentation of the virtual display may or may not include the presentation of a screen frame.

[0028] Some of the disclosed embodiments may include and / or access a data structure or database. The terms data structure and database consistent with the present disclosure may include any set of data values and the relationships between them. The data may be stored linearly, horizontally, hierarchically, relationally, non-relationally, unidimensionally, multidimensionally, operationally, in an ordered manner, in an unordered manner, in an object-oriented manner, in a centralized manner, in a decentralized manner, in a distributed manner, in a custom manner, or in any manner that enables data access. By way of non-limiting example, data structures may include arrays, associative arrays, linked lists, binary trees, balanced trees, heaps, stacks, queues, sets, hash tables, records, tagged unions, entity-relationship models, graphs, hypergraphs, matrices, tensors, and the like. For example, data structures may include XML databases, RDBMS databases, SQL databases, or NoSQL alternatives for data storage / retrieval such as, for example, MongoDB, Redis, Couchbase, Datastax Enterprise Graph, Elastic Search, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, and Neo4J. The data structure may be a component of the disclosed system or a remote computing component (e.g., a cloud-based data structure). The data within the data structure may be stored in contiguous or non-contiguous memory. Further, the data structure does not require that information be located in the same place. It may be distributed across multiple servers, which may be owned or operated by the same or different entities, for example. Thus, the term data structure in the singular includes multiple data structures.

[0029] In some embodiments, the system can determine a confidence level in the received input or any determined value. The term confidence level refers to any indication, numerical value, or the like of a level (e.g., within a predetermined range) indicating the amount of confidence the system has in the determined data. For example, the confidence level can have a value from 1 to 10. Alternatively, the confidence level may be represented as a percentage or any other numerical or non-numerical indication. In some cases, the system may compare the confidence level to a threshold. The term threshold can refer to a reference value, level, point, or range of values. During operation, when the confidence level of the determined data exceeds the threshold (or, depending on the particular use case, falls below it), the system can follow a first course of action, and when the confidence level falls below it (or, depending on the particular use case, exceeds it), the system can follow a second course of action. The value of the threshold can be predetermined for each type of object being inspected or can be dynamically selected based on various considerations.

[0030] System Overview Referring now to FIG. 1, a user using an exemplary extended reality system consistent with various embodiments of the present disclosure is shown. FIG. 1 is an exemplary representation of only one embodiment, and it should be understood that within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. As shown in the illustration, user 100 is sitting behind table 102 that supports keyboard 104 and mouse 106. Keyboard 104 is connected by wire 108 to wearable extended reality device 110 that displays virtual content to user 100. As an alternative to or in addition to wire 108, keyboard 104 may be wirelessly connected to wearable extended reality device 110. For purposes of illustration, the wearable extended reality device is shown as a pair of smart glasses, but as described above, wearable extended reality device 110 may be any type of head-mounted device used to present extended reality to user 100. The virtual content displayed by wearable extended reality device 110 includes virtual screen 112 (also referred to herein as a "virtual display screen" or "virtual display") and a plurality of virtual widgets 114. Virtual widgets 114A-114D are displayed adjacent to virtual screen 112, and virtual widget 114E is displayed on table 102. User 100 can use keyboard 104 to input text into document 116 displayed on virtual screen 112 and can use mouse 106 to control virtual cursor 118. In one example, virtual cursor 118 can be moved to any location within virtual screen 112. In another example, virtual cursor 118 can be moved to any location within virtual screen 112 and can also be moved to any one of virtual widgets 114A-114D, but cannot be moved to virtual widget 114E. In yet another example, virtual cursor 118 can be moved to any location within virtual screen 112 and can also be moved to any one of virtual widgets 114A-114E.In an additional example, the virtual cursor 118 can move to any location within the extended reality environment that includes the virtual screen 112 and the virtual widgets 114A - 114E. In yet another example, the virtual cursor can move only on all available surfaces (i.e., virtual or physical surfaces) or on selected surfaces within the extended reality environment. Alternatively, or in addition, the user 100 can use hand gestures recognized by the wearable extended reality device 110 to interact with any one of the virtual widgets 114A - 114E or a selected virtual widget. For example, the virtual widget 114E may be an interactive widget (e.g., a virtual slider controller) that can be operated with a hand gesture.

[0031] FIG. 2 shows an example of a system 200 that provides an extended reality (XR) experience to a user such as user 100. It should be understood that FIG. 2 is an exemplary representation of just one embodiment, and within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. System 200 may be computer-based and may include computer system components, wearable devices, workstations, tablets, handheld computing devices, memory devices, and / or an internal network connecting these components. System 200 may include or be connected to various network computing resources (e.g., servers, routers, switches, network connections, storage devices, etc.) to support the services provided by system 200. Consistent with the present disclosure, system 200 can include an input unit 202, an XR unit 204, a mobile communication device 206, and a remote processing unit 208. Remote processing unit 208 can include a server 210 coupled to one or more physical or virtual storage devices such as data structure 212. System 200 may also include or be connected to a communication network 214 that facilitates communication and data exchange between the various system components and the various entities associated with system 200.

[0032] In accordance with the present disclosure, the input unit 202 can include one or more devices that can receive input from the user 100. In one embodiment, the input unit 202 can include a text input device such as a keyboard 104. The text input device can include all possible types of devices and mechanisms for inputting text information into the system 200. Examples of text input devices can include mechanical keyboards, membrane keyboards, flexible keyboards, QWERTY keyboards, Dvorak keyboards, Colemak keyboards, coded keyboards, wireless keyboards, keypads, key-based control panels, or other arrangements of control keys, visual input devices, or any other mechanism that can input text whether provided in a physical form or presented virtually. In one embodiment, the input unit 202 can also include a pointing input device such as a mouse 106. The pointing input device can include all possible types of devices and mechanisms for inputting two-dimensional or three-dimensional information into the system 200. In one example, two-dimensional input from the pointing input device can be used to interact with virtual content presented via the XR unit 204. Examples of pointing input devices can include computer mice, trackballs, touchpads, trackpads, touchscreens, joysticks, pointing sticks, styli, light pens, or any other physical or virtual input mechanism. In one embodiment, the input unit 202 can also include a graphical input device such as a touch screen configured to detect contact, movement, or interruption of movement. The graphical input device can use any of a plurality of touch sensitivity technologies including capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more contact points, but is not limited thereto. In one embodiment, the input unit 202 can also include one or more voice input devices such as a microphone.The voice input device can include all possible types of devices and mechanisms for inputting voice data to facilitate voice-enabled functions such as voice recognition, voice replication, digital recording, and telephone functions. In one embodiment, the input unit 202 can also include one or more image input devices, such as an image sensor, configured to capture image data. In one embodiment, the input unit 202 can also include one or more tactile gloves configured to capture hand movement and gesture data. In one embodiment, the input unit 202 can also include one or more proximity sensors configured to detect the presence and / or movement of objects within a selected area near the sensor.

[0033] According to some embodiments, the system can include at least one sensor configured to detect and / or measure characteristics associated with a user, the user's actions, or the user's environment. An example of the at least one sensor is sensor 216 included in input unit 202. Sensor 216 can be an acceleration sensor, a touch sensor, a light sensor, an infrared sensor, an audio sensor, an image sensor, a proximity sensor, a positioning sensor, a gyroscope, a temperature sensor, a biosensor, or any other sensing device to facilitate related functions. Sensor 216 may be integrated or connected with the input device, or may be separated from the input device. In one example, a thermometer may be included in mouse 106 to measure the body temperature of user 100. In another example, a positioning sensor may be integrated with keyboard 104 to determine the movement of user 100 relative to keyboard 104. Such a positioning sensor can be implemented using one of the following technologies: Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Galileo Global Navigation System, Beidou Navigation System, other Global Navigation Satellite Systems (GNSS), Indian Regional Navigation Satellite System (IRNSS), Local Positioning System (LPS), Real-Time Location System (RTLS), Indoor Positioning System (IPS), Wi-Fi based positioning system, cellular triangulation, image-based positioning technology, indoor positioning technology, outdoor positioning technology, or any other positioning technology.

[0034] According to some embodiments, the system can include one or more sensors for identifying the position and / or movement of physical devices (physical input devices, physical computing devices, keyboard 104, mouse 106, wearable extended reality device 110, etc.). The one or more sensors may be included in the physical device or may be external to the physical device. In some examples, an image sensor external to the physical device (e.g., an image sensor included in another physical device) can be used to capture image data of the physical device, and the image data can be analyzed to identify the position and / or movement of the physical device. For example, the image data can be analyzed using a visual object tracking algorithm to identify the movement of the physical device, and can be analyzed using a visual object detection algorithm to identify the position of the physical device (e.g., the position relative to the image sensor, the position in a global coordinate system, etc.), and so on. In some examples, an image sensor included in the physical device can be used to capture image data, and the image data can be analyzed to identify the position and / or movement of the physical device. For example, the image data can be analyzed using a visual odometry algorithm to identify the position of the physical device, and can be analyzed using an egomotion algorithm to identify the movement of the physical device, and so on. In some examples, positioning sensors such as indoor positioning sensors or outdoor positioning sensors can be included in the physical device and used to determine the position of the physical device. In some examples, motion sensors such as accelerometers or gyroscopes can be included in the physical device and used to determine the movement of the physical device. In some examples, a physical device such as a keyboard or a mouse can be configured to be positioned on a physical surface. Such a physical device can include an optical mouse sensor (also known as a non-mechanical tracking engine) directed at the physical surface, and the output of the optical mouse sensor can be analyzed to determine the movement of the physical device relative to the physical surface.

[0035] Consistent with the present disclosure, the XR unit 204 may include a wearable extended reality device configured to present virtual content to the user 100. An example of a wearable extended reality device is the wearable extended reality device 110. Additional examples of wearable extended reality devices may include virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, or any other device capable of generating extended reality content. Some non-limiting examples of such devices may include Nreal Light, Magic Leap One, Varjo, Quest 1 / 2, Vive, and the like. In some embodiments, the XR unit 204 can present virtual content to the user 100. Generally, extended reality devices may include all composite environments of reality and virtuality generated by computer technology and wearables, as well as human-machine interactions. As described above, the term "extended reality" (XR) refers to a superset that includes all regions from "complete reality" to "complete virtuality". This includes typical forms such as augmented reality (AR), mixed reality (MR), virtual reality (VR), and regions interpolated between them. Therefore, it should be noted that the terms "XR device", "AR device", "VR device", and "MR device" may be used interchangeably herein and may refer to any of the various devices listed above.

[0036] In accordance with the present disclosure, the system can exchange data with various communication devices associated with a user, such as mobile communication device 206. The term "communication device" is intended to include all possible types of devices that can exchange data using a digital communication network, an analog communication network, or any other communication network configured to transmit data. In some examples, communication devices can include smartphones, tablets, smartwatches, personal digital assistants, desktop computers, laptop computers, IoT devices, dedicated terminals, wearable communication devices, and any other device that enables data communication. In some cases, mobile communication device 206 can complement or replace input unit 202. Specifically, mobile communication device 206 can be associated with a physical touch controller that can function as a pointing input device. Additionally, mobile communication device 206 can also implement, for example, a virtual keyboard and be used to replace a text input device. For example, if user 100 leaves table 102 and walks into the break room with a smart glass, this user can receive an email that requires a quick response. In this case, this user can choose to use their smartwatch as an input device and type a response to the email while the email is virtually presented by the smart glass.

[0037] Consistent with the present disclosure, embodiments of the system may include the use of a cloud server. The term "cloud server" refers to a computer platform that provides services via a network such as the Internet. In the exemplary embodiment shown in FIG. 2, the server 210 can use virtual machines that may not correspond to individual hardware. For example, computing and / or memory capabilities can be implemented by allocating an appropriate portion of the desired computing / memory capabilities from a scalable repository such as a data center or a distributed computing environment. Specifically, in one embodiment, the remote processing unit 208 can be used with the XR unit 204 to provide virtual content to the user 100. In one exemplary configuration, the server 210 can be a cloud server that functions as an operating system (OS) for wearable extended reality devices. In one example, the server 210 can implement the methods described herein using custom hardwired logic, one or more application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), firmware, and / or program logic that combines the server 210 into a dedicated machine with a computer system.

[0038] In some embodiments, the server 210 can access the data structure 212 to determine, for example, virtual content to display for the user 100. The data structure 212 can utilize volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, other types of storage devices or tangible or non-transitory computer-readable media, or any medium or mechanism for storing information. The data structure 212 can be part of the server 210 as shown, or can be separate from the server 210. If the data structure 212 is not part of the server 210, the server 210 can exchange data with the data structure 212 via a communication link. The data structure 212 can include one or more memory devices that store data and instructions used to execute one or more features of the disclosed methods. In one embodiment, the data structure 212 can include any of a plurality of suitable data structures ranging from small data structures hosted on a workstation to large data structures distributed among data centers. The data structure 212 can also include any combination of one or more data structures controlled by a memory controller device (e.g., a server) or software.

[0039] In accordance with the present disclosure, the communication network can be any type of network (including infrastructure) that supports communication, exchanges information, and / or facilitates the exchange of information between components of the system. For example, the communication network 214 within the system 200 can include, for example, a telephone network, an extranet, an intranet, the Internet, satellite communication, offline communication, wireless communication, transponder communication, a local area network (LAN), a wireless network (e.g., a Wi-Fi / 802.11 network), a wide area communication network (WAN), a virtual private network (VPN), a digital communication network, an analog communication network, or any other mechanism or combination of mechanisms that enables data transmission.

[0040] The components and arrangements of the system 200 shown in FIG. 2 are intended to be exemplary only, as system components used to implement the disclosed processes and features may vary, and are not intended to limit any embodiments.

[0041] FIG. 3 is a block diagram of an exemplary configuration of the input unit 202. It should be understood that FIG. 3 is an exemplary representation of only one embodiment, and within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. In the embodiment of FIG. 3, the input unit 202 can directly or indirectly access a bus 300 (or other communication mechanism) that interconnects subsystems and components for transferring information within the input unit 202. For example, the bus 300 can interconnect a memory interface 310, a network interface 320, an input interface 330, a power supply 340, an output interface 350, a processing device 360, a sensor interface 370, and a database 380.

[0042] The memory interface 310 shown in FIG. 3 can be used to access software products and / or data stored in a non-transitory computer-readable medium. Generally, a non-transitory computer-readable storage medium refers to any kind of physical memory that can store information or data readable by at least one processor. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, any other optical data storage media, any physical media with a pattern of holes, PROM, EPROM, flash EPROM or any other flash memory, NVRAM, caches, registers, any other memory chip or cartridge, and networked versions thereof. The terms “memory” and “computer-readable storage medium” can refer to multiple structures, such as multiple memories or computer-readable storage media located within an input unit or at remote locations. Additionally, one or more computer-readable storage media can be utilized when implementing a computer-implemented method. Thus, the term computer-readable storage medium should be understood to include tangible items and exclude carrier waves and transient signals. In the specific embodiment shown in FIG. 3, the memory interface 310 can be used to access software products and / or data stored in a memory device such as memory device 311. Memory device 311 can include high-speed random access memory and / or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and / or flash memory (e.g., NAND, NOR). Consistent with the present disclosure, the components of memory device 311 can be distributed across two or more units of system 200 and / or two or more memory devices.

[0043] The memory device 311 shown in FIG. 3 may include software modules for executing processes consistent with the present disclosure. In particular, the memory device 311 may include an input determination module 312, an output determination module 313, a sensor communication module 314, a virtual content determination module 315, a virtual content communication module 316, and a database access module 317. Modules 312-317 may include software instructions for execution by at least one processor (e.g., processing device 360) associated with the input unit 202. The input determination module 312, the output determination module 313, the sensor communication module 314, the virtual content determination module 315, the virtual content communication module 316, and the database access module 317 may cooperate to perform various operations. For example, the input determination module 312 may be able to determine text using data received from, for example, the keyboard 104. Thereafter, the output determination module 313 may be able to cause the most recently input text to be presented on a dedicated display 352 physically or wirelessly coupled to, for example, the keyboard 104. In this way, when the user 100 types, a preview of the typed text can be viewed without constantly moving the head up and down to view the virtual screen 112. The sensor communication module 314 may be able to receive data from various sensors to determine the status of the user 100. Thereafter, the virtual content determination module 315 may be able to determine the virtual content to be displayed based on the received input and the determined status of the user 100. For example, the determined virtual content may be a virtual presentation of the most recently input text on a virtual screen that is virtually located adjacent to the keyboard 104. The virtual content communication module 316 may be able to obtain virtual content (e.g., another user's avatar) not determined by the virtual content determination module 315. The search for virtual content may be from the database 380, from a remote processing unit 208, or from any other source.

[0044] In some embodiments, the input determination module 312 can adjust the operation of the input interface 330 to receive pointer input 331, text input 332, audio input 333, and XR-related input 334. Details regarding pointer input, text input, and audio input have been described above. The term "XR-related input" can include any type of data that can cause a change in the virtual content presented to the user 100. In one embodiment, the XR-related input 334 can include image data of the user 100, wearable extended reality devices (e.g., detected gestures of the user 100's hand). In another embodiment, the XR-related input 334 can include wireless communication indicating the presence of another user in proximity to the user 100. Consistent with the present disclosure, the input determination module 312 can receive different types of input data simultaneously. Thereafter, the input determination module 312 can further apply various rules based on the type of input detected. For example, pointer input may be prioritized over voice input.

[0045] In some embodiments, the output determination module 313 can adjust the operation of the output interface 350 to generate an output using the optical indicator 351, the display 352, and / or the speaker 353. Generally, the output generated by the output determination module 313 does not include virtual content presented by the wearable extended reality device. Instead, the output generated by the output determination module 313 includes various outputs related to the operation of the input unit 202 and / or the XR unit 204. In one embodiment, the optical indicator 351 can include an optical indicator indicating the status of the wearable extended reality device. For example, the optical indicator may display green light when the wearable extended reality device 110 is connected to the keyboard 104 and may blink when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 352 can be used to display operation information. For example, the display may present an error message when the wearable extended reality device is inoperable. In another embodiment, the speaker 353 can be used to output sound, for example, when the user 100 desires to play some music for another user.

[0046] In some embodiments, the sensor communication module 314 can adjust the operation of the sensor interface 370 to receive sensor data from one or more sensors integrated with or connected to the input device. The one or more sensors can include an audio sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374 (e.g., a temperature sensor, an ambient light detector, etc.), and other sensors 375. In one embodiment, the data received from the sensor communication module 314 can be used to determine the physical orientation of the input device. The physical orientation of the input device can indicate the user's state and can be determined based on a combination of tilt movement, roll movement, and lateral movement. Thereafter, the physical orientation of the input device is used by the virtual content determination module 315 to modify the display parameters of the virtual content to match the user's state (e.g., attentive, sleepy, active, sitting, standing, reclined backward, reclined forward, walking, moving, riding, etc.).

[0047] In some embodiments, the virtual content determination module 315 can determine the virtual content to be displayed by the wearable extended reality device. The virtual content can be determined based on data from the input determination module 312, the sensor communication module 314, and other sources (e.g., the database 380). In some embodiments, determining the virtual content can include determining the distance, size, and orientation of the virtual object. The determination of the position of the virtual object can be determined based on the type of the virtual object. Specifically, with respect to the example shown in FIG. 1, since the virtual widget 114E is a virtual controller (e.g., a volume bar), the virtual content determination module 315 can determine to arrange the four virtual widgets 114A-114D on both sides of the virtual screen 112 and arrange the virtual widget 114E on the table 102. The determination of the position of the virtual object can be further determined based on the user's preference. For example, in the case of a left-handed user, the virtual content determination module 315 can determine to arrange the virtual volume bar to the left of the keyboard 104, and in the case of a right-handed user, the virtual content determination module 315 can determine to arrange the virtual volume bar to the right of the keyboard 104.

[0048] In some embodiments, the virtual content communication module 316 can adjust the operation of the network interface 320 to obtain data presented as virtual content to the user 100 from one or more sources. The one or more sources can include other XR units 204, the user's mobile communication device 206, a remote processing unit 208, publicly available information, and the like. In one embodiment, the virtual content communication module 316 can communicate with the mobile communication device 206 to provide a virtual representation of the mobile communication device 206. For example, the virtual representation can enable the user 100 to read messages and interact with applications installed on the mobile communication device 206. The virtual content communication module 316 can also adjust the operation of the network interface 320 to share virtual content with other users. In one example, the virtual content communication module 316 can use data from the input determination module to identify a trigger (e.g., the trigger can include a user gesture) and transfer the content from the virtual display to a physical display (e.g., a television) or to the virtual display of a different user.

[0049] In some embodiments, the database access module 317 can cooperate with the database 380 to search for stored data. The retrieved data can include, for example, privacy levels associated with various virtual objects, relationships between virtual objects and physical objects, user preferences, the user's past behavior, and the like. As described above, the virtual content determination module 315 can use the data stored in the database 380 to determine virtual content. The database 380 can include separate databases, such as, for example, a vector database, a raster database, a tile database, a viewport database, and / or a user input database. The data stored in the database 380 can be received from the modules 314-317 or other components of the system 200. Further, the data stored in the database 380 can be provided as input using data input, data transfer, or data upload.

[0050] Modules 312 - 317 can be implemented in software, hardware, firmware, or a mix of any of these. In some embodiments, any one or more of Modules 312 - 317 and the data associated with database 380 can be stored in XR unit 204, mobile communication device 206, or remote processing unit 208. The processing device of system 200 can be configured to execute the instructions of Modules 312 - 317. In some embodiments, aspects of Modules 312 - 317 can be implemented in hardware, software (including one or more signal processing and / or application specific integrated circuits), firmware, or any combination of these, executable by one or more processors, either alone or in various combinations with each other. Specifically, Modules 312 - 317 can be configured to interact with each other and / or with other modules of system 200 to perform functions consistent with some of the disclosed embodiments. For example, input unit 202 can execute instructions including an image processing algorithm on data from XR unit 204 to determine the movement of user 100's head. Further, with respect to input unit 202, or components of input unit 202, each function described throughout this specification can correspond to a set of instructions for performing that function. These instructions need not be implemented as separate software programs, procedures, or modules. Memory device 311 can include additional modules and instructions, or fewer modules and instructions. For example, memory device 311 can store an operating system such as ANDROID, iOS, UNIX, OSX, WINDOWS, DARWIN, RTXC, LINUX, or an embedded operating system such as VXWorkS. The operating system can include instructions for handling basic system services and performing hardware - dependent tasks.

[0051] The network interface 320 shown in FIG. 3 can provide bidirectional data communication to a network such as the communication network 214. In one embodiment, the network interface 320 may include an Integrated Services Digital Network (ISDN) card, a cellular modem, a satellite modem, or a modem for providing a data communication connection via the Internet. As another example, the network interface 320 may include a Wireless Local Area Network (WLAN) card. In another embodiment, the network interface 320 may include an Ethernet port connected to a high-frequency receiver and transmitter, and / or an optical (e.g., infrared) receiver and transmitter. The specific design and implementation of the network interface 320 may depend on one or more communication networks in which the input unit 202 is intended to operate. For example, in some embodiments, the input unit 202 may include a network interface 320 designed to operate on a GSM network, a GPRS network, an EDGE network, a Wi-Fi or WiMax network, and a Bluetooth network. In any such implementation, the network interface 320 may be configured to transmit and receive electrical, electromagnetic, or optical signals that carry digital data streams or digital signals representing various types of information.

[0052] The input interface 330 shown in FIG. 3 can receive inputs from various input devices, such as a keyboard, a mouse, a touchpad, a touch screen, one or more buttons, a joystick, a microphone, an image sensor, and any other device configured to detect physical or virtual inputs. The received input can be in at least one form of text, sound, voice, hand gesture, body gesture, tactile information, and any other type of physical or virtual input generated by the user. In the illustrated embodiment, the input interface 330 can receive pointer input 331, text input 332, audio input 333, and XR-related input 334. In additional embodiments, the input interface 330 may be an integrated circuit that can function as a bridge between the processing device 360 and any of the input devices listed above.

[0053] The power supply 340 shown in FIG. 3 supplies electrical energy to the input unit 202 and can optionally also supply power to the XR unit 204. Generally, the power supply included in any device or system in the present disclosure can be one or more batteries (e.g., lead-acid battery, lithium-ion battery, nickel-metal hydride battery, nickel-cadmium battery), one or more capacitors, one or more connections to an external power source, one or more power converters, or any combination thereof, but is not limited thereto, and can be any device capable of repeatedly storing, distributing, or transmitting power. Referring to the example shown in FIG. 3, the power supply may be portable, which means that the input unit 202 can be easily carried by hand (e.g., the total weight of the power supply 340 may be less than 1 pound). Due to the portability of the power supply, the user 100 can use the input unit 202 in various situations. In other embodiments, the power supply 340 may be associated with a connection to an external power source (such as a power grid) that can be used to charge the power supply 340. Additionally, the power supply 340 may be configured to charge one or more batteries included in the XR unit 204. For example, extended reality glasses (e.g., wearable extended reality device 110) can be charged (e.g., wirelessly or not wirelessly) when the glasses are placed on or in proximity to the input unit 202.

[0054] The output interface 350 shown in FIG. 3 can output from various output devices, for example, using an optical indicator 351, a display 352, and / or a speaker 353. In one embodiment, the output interface 350 may be an integrated circuit that can function as a bridge between the processing device 360 and at least one of the output devices listed above. The optical indicator 351 may include one or more light sources, for example, an LED array associated with various colors. The display 352 may include a screen (e.g., an LCD or dot matrix screen) or a touch screen. The speaker 353 may include audio headphones, a hearing aid type device, a speaker, bone conduction headphones, an interface that provides tactile cues, a vibrotactile stimulator, and the like.

[0055] The processing device 360 shown in FIG. 3 may include at least one processor configured to execute a computer program, application, method, process, or other software to implement the embodiments described in this disclosure. Generally, the processing device included in any device or system in this disclosure may include one or more integrated circuits, microchips, microcontrollers, microprocessors, all or part of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or other circuits suitable for executing instructions or performing logical operations. The processing device may include at least one processor configured to implement the functions of the disclosed methods, such as a microprocessor manufactured by Intel (trademark). The processing device may include a single-core or multi-core processor that executes parallel processes simultaneously. In one example, the processing device may be a single-core processor configured using virtual processing technology. The processing device may implement virtual machine technology or other technologies to provide the ability to execute, control, operate, manage, and store multiple software processes, applications, programs, etc. In another example, the processing device may include a multi-core processor configuration (e.g., dual-core, quad-core, etc.) configured to provide a parallel processing function to enable devices associated with the processing device to execute multiple processes simultaneously. It should be understood that other types of processor configurations may be implemented to provide the capabilities disclosed herein.

[0056] The sensor interface 370 shown in FIG. 3 can acquire sensor data from various sensors, such as an audio sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374, and other sensors 375. In one embodiment, the sensor interface 370 may be an integrated circuit that can function as a bridge between the processing device 360 and at least one of the sensors listed above.

[0057] The audio sensor 371 may include one or more audio sensors configured to capture audio by converting sound into digital information. Some examples of audio sensors can include microphones, unidirectional microphones, bidirectional microphones, cardioid microphones, omnidirectional microphones, on-board microphones, wired microphones, wireless microphones, or any combination of the above. Consistent with the present disclosure, the processing device 360 can modify the presentation of virtual content based on data received from the audio sensor 371 (e.g., voice commands).

[0058] The image sensor 372 may include one or more image sensors configured to capture visual information by converting light into image data. Consistent with the present disclosure, the image sensor may be included in any device or system in the present disclosure and may be any device capable of detecting optical signals in the near-infrared, infrared, visible, and ultraviolet spectra and converting them into electrical signals. Examples of image sensors may include digital cameras, phone cameras, semiconductor charge-coupled devices (CCDs), active pixel sensors in complementary metal-oxide-semiconductor (CMOS), or N-type metal-oxide-semiconductor (NMOS, live MOS). The electrical signals may be used to generate image data. Consistent with the present disclosure, the image data may include pixel data streams, digital images, digital video streams, data obtained from captured images, and data that may be used to construct one or more 3D images, sequences of 3D images, 3D videos, or virtual 3D representations. The image data obtained by the image sensor 372 may be transmitted to any processing device of the system 200 by wired or wireless transmission. For example, the image data may be processed to detect objects, detect events, detect actions, detect faces, detect people, recognize known persons, or for any other information that may be used by the system 200. Consistent with the present disclosure, the processing device 360 may be able to modify the presentation of virtual content based on the image data received from the image sensor 372.

[0059] The motion sensor 373 may include one or more motion sensors configured to measure the motion of the input unit 202 or the motion of an object within the environment of the input unit 202. Specifically, the motion sensor can detect the motion of an object within the environment of the input unit 202, measure the speed of an object within the environment of the input unit 202, measure the acceleration of an object within the environment of the input unit 202, detect the motion of the input unit 202, measure the speed of the input unit 202, measure the acceleration of the input unit 202, etc., and can perform at least one of these. In some embodiments, the motion sensor 373 may include one or more accelerometers configured to detect an appropriate change in acceleration and / or measure an appropriate acceleration of the input unit 202. In other embodiments, the motion sensor 373 may include one or more gyroscopes configured to detect a change in the orientation of the input unit 202 and / or measure information regarding the orientation of the input unit 202. In other embodiments, the motion sensor 373 may include one or more that use an image sensor, a LIDAR sensor, a radar sensor, or a proximity sensor. For example, by analyzing the captured image, the processing device can determine the motion of the input unit 202 using, for example, an ego-motion algorithm. Additionally, the processing device can determine the motion of an object within the environment of the input unit 202 using, for example, an object tracking algorithm. Consistent with the present disclosure, the processing device 360 can modify the presentation of virtual content based on the determined motion of the input unit 202 or the determined motion of an object within the environment of the input unit 202. For example, make the virtual display follow the motion of the input unit 202.

[0060] The environmental sensor 374 can include one or more sensors from different types configured to capture data reflecting the environment of the input unit 202. In some embodiments, the environmental sensor 374 is configured to perform at least one of measuring chemical properties in the environment of the input unit 202, measuring changes in chemical properties in the environment of the input unit 202, detecting the presence of chemical substances in the environment of the input unit 202, and measuring the concentration of chemical substances in the environment of the input unit 202, and can include one or more chemical sensors. Examples of such chemical properties can include pH level, toxicity, and temperature. Examples of such chemical substances can include electrolytes, specific enzymes, specific hormones, specific proteins, smoke, carbon dioxide, carbon monoxide, oxygen, ozone, hydrogen, and hydrogen sulfide. In other embodiments, the environmental sensor 374 can include one or more temperature sensors configured to detect changes in the temperature of the environment of the input unit 202 and / or measure the temperature of the environment of the input unit 202. In other embodiments, the environmental sensor 374 can include one or more barometers configured to detect changes in the atmospheric pressure in the environment of the input unit 202 and / or measure the atmospheric pressure in the environment of the input unit 202. In other embodiments, the environmental sensor 374 can include one or more light sensors configured to detect changes in ambient light in the environment of the input unit 202. Consistent with the present disclosure, the processing device 360 can modify the presentation of virtual content based on the input from the environmental sensor 374. For example, automatically reducing the brightness of virtual content when the environment of user 100 becomes dark.

[0061] The other sensor 375 may include a weight sensor, a light sensor, a resistance sensor, an ultrasonic sensor, a proximity sensor, a biosensor, or other sensing devices to facilitate related functions. In certain embodiments, the other sensor 375 may include one or more positioning sensors configured to obtain positioning information of the input unit 202, detect a change in the position of the input unit 202, and / or measure the position of the input unit 202. Alternatively, the GPS software may enable the input unit 202 to access an external GPS receiver (e.g., connect via a serial port or Bluetooth). Consistent with the present disclosure, the processing device 360 can modify the presentation of the virtual content based on the input from the other sensor 375. For example, personal information is presented only after the user 100 is identified using data from the biosensor.

[0062] The components and arrangements shown in FIG. 3 are not intended to limit any embodiments. As will be appreciated by those skilled in the art having the benefit of this disclosure, numerous variations and / or modifications may be made to the illustrated configuration of the input unit 202. For example, in all cases, not all components may be essential for the operation of the input unit. Any component may be disposed in any suitable portion of the input unit, and the components may be rearranged in various configurations while providing the functions of the various embodiments. For example, some input units may not include all of the elements as shown in the input unit 202.

[0063] FIG. 4 is a block diagram of an exemplary configuration of the XR unit 204. It should be understood that FIG. 4 is an exemplary representation of only one embodiment, and within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. In the embodiment of FIG. 4, the XR unit 204 can access directly or indirectly a bus 400 (or other communication mechanism) that interconnects subsystems and components for transferring information within the XR unit 204. For example, the bus 400 can interconnect a memory interface 410, a network interface 420, an input interface 430, a power supply 440, an output interface 450, a processing device 460, a sensor interface 470, and a database 480.

[0064] The memory interface 410 shown in FIG. 4 is assumed to have the same functions as the memory interface 310 described in detail above. The memory interface 410 can be used to access software products and / or data stored in a non-transitory computer-readable medium or in a memory device such as the memory device 411. The memory device 411 may include software modules for executing a process consistent with the present disclosure. In particular, the memory device 411 may include an input determination module 412, an output determination module 413, a sensor communication module 414, a virtual content determination module 415, a virtual content communication module 416, and a database access module 417. Modules 412-417 may include software instructions for execution by at least one processor (e.g., the processing device 460) associated with the XR unit 204. The input determination module 412, the output determination module 413, the sensor communication module 414, the virtual content determination module 415, the virtual content communication module 416, and the database access module 417 may cooperate to perform various operations. For example, the input determination module 412 can determine a user interface (UI) input received from the input unit 202. At the same time, the sensor communication module 414 can receive data from various sensors to determine the status of the user 100. The virtual content determination module 415 can determine the virtual content to be displayed based on the received input and the determined status of the user 100. The virtual content communication module 416 can search for virtual content not determined by the virtual content determination module 415. The search for virtual content may be from the database 380, the database 480, the mobile communication device 206, or the remote processing unit 208. Based on the output of the virtual content determination module 415, the output determination module 413 can cause a change in the virtual content displayed to the user 100 by the projector 454.

[0065] In some embodiments, the input determination module 412 can adjust the operation of the input interface 430 to receive gesture input 431, virtual input 432, audio input 433, and UI input 434. Consistent with the present disclosure, the input determination module 412 can receive different types of input data simultaneously. In one embodiment, the input determination module 412 can apply various rules based on the type of detected input. For example, gesture input may be prioritized over virtual input. In some embodiments, the output determination module 413 can adjust the operation of the output interface 450 to generate an output using the optical indicator 451, display 452, speaker 453, and projector 454. In one embodiment, the optical indicator 451 may include an optical indicator indicating the status of the wearable extended reality device. For example, the optical indicator may display green light when the wearable extended reality device 110 is connected to the input unit 202 and may blink when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 452 can be used to display operation information. In another embodiment, the speaker 453 may include bone conduction headphones used to output sound to the user 100. In another embodiment, the projector 454 can present virtual content to the user 100.

[0066] The operations of the sensor communication module, virtual content determination module, virtual content communication module, and database access module have been described above with reference to FIG. 3, and the details are not repeated here. Modules 412-417 can be implemented in software, hardware, firmware, a mixture of any of these, etc.

[0067] The network interface 420 shown in FIG. 4 is assumed to have the same functions as the network interface 320 described in detail above. The specific design and implementation of the network interface 420 may depend on the communication network in which the XR unit 204 is intended to operate. For example, in some embodiments, the XR unit 204 is configured to be selectively connectable to the input unit 202 by wire. When connected by wire, the network interface 420 can enable communication with the input unit 202, and when not connected by wire, the network interface 420 can enable communication with the mobile communication device 206.

[0068] The input interface 430 shown in FIG. 4 is assumed to have the same functions as the input interface 330 described in detail above. In this case, the input interface 430 can communicate with an image sensor to obtain a gesture input 431 (e.g., the finger of the user 100 pointing to a virtual object), communicate with other XR units 204 to obtain a virtual input 432 (e.g., a virtual object shared with the XR unit 204, or a gesture of an avatar detected in a virtual environment), communicate with a microphone to obtain an audio input 433 (e.g., a voice command), and communicate with the input unit 202 to obtain a UI input 434 (e.g., virtual content determined by the virtual content determination module 315).

[0069] The power supply 440 shown in FIG. 4 is assumed to have the same functions as the power supply 340 described above, and only supplies electrical energy to supply power to the XR unit 204. In some embodiments, the power supply 440 can be charged by the power supply 340. For example, the power supply 440 may be wirelessly changed when the XR unit 204 is placed on or near the input unit 202.

[0070] The output interface 450 shown in FIG. 4 is assumed to have the same functions as the output interface 350 described in detail above. In this case, the output interface 450 can output from the optical indicator 451, the display 452, the speaker 453, and the projector 454. The projector 454 can be any device, apparatus, instrument, etc. that can project (or direct) light to display virtual content on a surface. This surface may be part of the XR unit 204, part of the user 100's eye, or part of an object close to the user 100. In one embodiment, the projector 454 can include an illumination unit that concentrates light within a limited solid angle by one or more mirrors and lenses and provides a high value of luminous intensity in a defined direction.

[0071] The processing device 460 shown in FIG. 4 is assumed to have the same functions as the output processing device 360 described in detail above. When the XR unit 204 is connected to the input unit 202, the processing device 460 can cooperate with the processing device 360. Specifically, the processing device 460 can implement virtual machine technology or other technologies to provide the ability to execute, control, operate, manipulate, store, etc. multiple software processes, applications, programs, etc. It should be understood that other types of processor configurations can be implemented to provide the capabilities disclosed herein.

[0072] The sensor interface 470 shown in FIG. 4 is assumed to have the same functions as the sensor interface 370 described in detail above. Specifically, the sensor interface 470 can communicate with the audio sensor 471, the image sensor 472, the motion sensor 473, the environmental sensor 474, and other sensors 475. The operations of the audio sensor, the image sensor, the motion sensor, the environmental sensor, and other sensors have been described above with reference to FIG. 3, and the details are not repeated here. It should be understood that other types and combinations of sensors can be used to provide the capabilities disclosed herein.

[0073] The components and arrangements shown in FIG. 4 are not intended to limit any embodiment. As will be understood by those skilled in the art having the benefit of this disclosure, numerous variations and / or modifications may be made to the illustrated configuration of the XR unit 204. For example, in all cases, not all components are necessarily essential for the operation of the XR unit 204. Any component may be arranged in any suitable part of the system 200, and the components may be rearranged in various configurations while providing the functions of various embodiments. For example, some XR units may not include all of the elements within the XR unit 204 (e.g., the wearable extended reality device 110 may not have the optical indicator 451).

[0074] FIG. 5 is a block diagram of an exemplary configuration of the remote processing unit 208. It should be understood that FIG. 5 is merely an exemplary representation of one embodiment, and within the scope of this disclosure, some of the illustrated elements may be omitted and other elements may be added. In the embodiment of FIG. 5, the remote processing unit 208 may include a server 210 that directly or indirectly accesses a bus 500 (or other communication mechanism) that interconnects subsystems and components for transferring information within the server 210. For example, the bus 500 can interconnect a memory interface 510, a network interface 520, a power supply 540, a processing device 560, and a database 580. The remote processing unit 208 may also include one or more data structures. For example, data structures 212A, 212B, and 212C.

[0075] The memory interface 510 shown in FIG. 5 is assumed to have the same functions as the memory interface 310 described in detail above. The memory interface 510 can be used to access software products and / or data stored in a non-transitory computer-readable medium or in other memory devices such as the memory devices 311, 411, 511, or the data structures 212A, 212B, and 212C. The memory device 511 may include software modules for executing processes consistent with the present disclosure. In particular, the memory device 511 may include a shared memory module 512, a node registration module 513, a load balancing module 514, one or more computing nodes 515, an internal communication module 516, an external communication module 517, and a database access module (not shown). Modules 512-517 may include software instructions for execution by at least one processor (e.g., the processing device 560) associated with the remote processing unit 208. The shared memory module 512, the node registration module 513, the load balancing module 514, the computing module 515, and the external communication module 517 may cooperate to perform various operations.

[0076] The shared memory module 512 may enable information sharing between the remote processing unit 208 and other components of the system 200. In some embodiments, the shared memory module 512 may be configured to allow the processing device 560 (and other processing devices within the system 200) to access, retrieve, and store data. For example, using the shared memory module 512, the processing device 560 can perform at least one of executing a software program stored in the memory device 511, the database 580, or the data structures 212A-C, storing information in the memory device 511, the database 580, or the data structures 212A-C, or retrieving information from the memory device 511, the database 580, or the data structures 212A-C.

[0077] The node registration module 513 can be configured to track the availability of one or more computing nodes 515. In some examples, the node registration module 513 can be implemented as a software program, such as a software program executed by one or more computing nodes 515, a hardware solution, or a combined software and hardware solution. In some implementations, the node registration module 513 can communicate with one or more computing nodes 515 using, for example, the internal communication module 516. In some examples, one or more computing nodes 515 can notify the node registration module 513 of the status of these nodes by, for example, sending a message at startup, shutdown, at regular intervals, at a selected time, in response to a query received from the node registration module 513, or at any other determined time. In some examples, the node registration module 513 can query the status of one or more computing nodes 515 by, for example, sending a message at startup, at regular intervals, at a selected time, or at any other determined time.

[0078] The load balancing module 514 can be configured to divide the workload among one or more computing nodes 515. In some examples, the load balancing module 514 can be implemented as a software program such as a software program executed by one or more of the computing nodes 515, a hardware solution, or a combined software and hardware solution. In some implementations, the load balancing module 514 can interact with the node registration module 513 to obtain information regarding the availability of one or more of the computing nodes 515. In some implementations, the load balancing module 514 can communicate with one or more of the computing nodes 515 using, for example, the internal communication module 516. In some examples, one or more of the computing nodes 515 can notify the load balancing module 514 of the status of these nodes by, for example, responding to a query received from the load balancing module 514 at startup, shutdown, at regular intervals, at a selected time, or at any other determined time, or by sending a message at any other determined time. In some examples, the load balancing module 514 can inquire about the status of one or more of the computing nodes 515 by, for example, sending a message at startup, at regular intervals, at a preselected time, or at any other determined time.

[0079] The internal communication module 516 may be configured to receive and / or transmit information from one or more components of the remote processing unit 208. For example, control signals and / or synchronization signals may be transmitted and / or received through the internal communication module 516. In one embodiment, input information for a computer program, output information of a computer program, and / or intermediate information of a computer program may be transmitted and / or received through the internal communication module 516. In another embodiment, information received through the internal communication module 516 may be stored in the memory device 511, the database 580, the data structures 212A-C, or other memory devices within the system 200. For example, information retrieved from the data structure 212A may be transmitted using the internal communication module 516. In another example, input data may be received using the internal communication module 516 and stored in the data structure 212B.

[0080] The external communication module 517 may be configured to receive and / or transmit information from one or more components of the system 200. For example, control signals may be transmitted and / or received through the external communication module 517. In one embodiment, information received through the external communication module 517 may be stored in the memory device 511, the database 580, the data structures 212A-C, and / or any memory device within the system 200. In another embodiment, information retrieved from any of the data structures 212A-C may be transmitted to the XR unit 204 using the external communication module 517. In another embodiment, input data may be transmitted and / or received using the external communication module 517. Examples of such input data may include data received from the input unit 202, information captured from the environment of the user 100 using one or more sensors (e.g., audio sensor 471, image sensor 472, motion sensor 473, environmental sensor 474, other sensor 475), and the like.

[0081] In some embodiments, the aspects of modules 512-517 can be implemented in hardware, (including one or more signal processing and / or application specific integrated circuits), software, firmware, or any combination thereof, executable by one or more processors, either alone or in various combinations with each other. Specifically, modules 512-517 can be configured to interact with each other and / or with other modules of system 200 to perform functions consistent with the embodiments of the present disclosure. Memory device 511 can include additional modules and instructions, or fewer modules and instructions.

[0082] The network interface 520, power supply 540, processing device 560, and database 580 shown in FIG. 5 are assumed to have functions similar to those of the similar elements described above with reference to FIGS. 4 and 5. The specific design and implementation of the above-described components can vary based on the implementation of system 200. Additionally, remote processing unit 208 can include more or fewer components. For example, remote processing unit 208 can include an input interface configured to receive input directly from one or more input devices.

[0083] In accordance with the present disclosure, a processing device of the system 200 (e.g., a processor within the mobile communication device 206, a processor within the server 210, a processor within a wearable extended reality device such as the wearable extended reality device 110, and / or a processor within an input device associated with the wearable extended reality device 110 such as the keyboard 104) can use machine learning algorithms to implement any of the methods disclosed herein. In some embodiments, a machine learning algorithm (also referred to as a machine learning model in the present disclosure) can be trained using training examples, for example, as described below. Some non-limiting examples of such machine learning algorithms include classification algorithms, data regression algorithms, image segmentation algorithms, visual detection algorithms (object detectors, face detectors, person detectors, motion detectors, edge detectors, etc.), visual recognition algorithms (face recognition, person recognition, object recognition, etc.), speech recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbor algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recurrent neural network algorithms, linear machine learning models, non-linear machine learning models, ensemble algorithms, and the like. For example, a trained machine learning algorithm can include an inference model such as a prediction model, a classification model, a data regression model, a clustering model, a segmentation model, an artificial neural network (deep neural network, convolutional neural network, recurrent neural network, etc.), a random forest, a support vector machine, and the like. In some examples, a training example can include an exemplary input together with a desired output corresponding to the exemplary input. Further, in some examples, training a machine learning algorithm using training examples can generate a trained machine learning algorithm, and the trained machine learning algorithm can be used to estimate an output for an input not included in the training examples.In some examples, the engineers, scientists, processes, and machines that train a machine learning algorithm may further use validation examples and / or test examples. For example, the validation examples and / or test examples can include exemplary inputs along with the desired outputs corresponding to the exemplary inputs, and the trained machine learning algorithm and / or the intermediate trained machine learning algorithm can be used to estimate the outputs for the exemplary inputs of the validation examples and / or test examples, the estimated outputs can be compared with the corresponding desired outputs, and the trained machine learning algorithm and / or the intermediate trained machine learning algorithm can be evaluated based on the results of the comparison. In some examples, the machine learning algorithm can have parameters and hyperparameters, the hyperparameters can be set manually by a person or automatically by a process external to the machine learning algorithm (such as a hyperparameter search algorithm), and the parameters of the machine learning algorithm can be set by the machine learning algorithm based on training examples. In some implementations, the hyperparameters may be set based on training examples and validation examples, and the parameters may be set based on training examples and the selected hyperparameters. For example, given the hyperparameters, the parameters can be conditionally independent from the validation examples.

[0084] In some embodiments, a trained machine learning algorithm (also referred to herein as a machine learning model and a trained machine learning model) can be used to analyze an input and generate an output, for example, as described below. In some examples, a trained machine learning algorithm can be used as an inference model that generates an output inferred when an input is provided. For example, a trained machine learning algorithm can include a classification algorithm, the input can include a sample, and the inferred output can include a classification of the sample (inferred label, inferred tag, etc.). In another example, a trained machine learning algorithm can include a regression model, the input can include a sample, and the inferred output can include an inferred value corresponding to the sample. In yet another example, a trained machine learning algorithm can include a clustering model, the input can include a sample, and the inferred output can include an assignment of the sample to at least one cluster. In an additional example, a trained machine learning algorithm can include a classification algorithm, the input can include an image, and the inferred output can include a classification of an item shown in the image. In yet another example, a trained machine learning algorithm can include a regression model, the input can include an image, and the inferred output can include an inferred value corresponding to an item shown in the image (estimated properties of the item, such as size, volume, age of a person shown in the image, distance from an item shown in the image, etc.). In an additional example, a trained machine learning algorithm can include an image segmentation model, the input can include an image, and the inferred output can include a segmentation of the image. In yet another example, a trained machine learning algorithm can include an object detector, the input can include an image, and the inferred output can include one or more detected objects in the image and / or one or more locations of an object in the image.In some examples, a trained machine learning algorithm can include one or more equations, and / or one or more functions, and / or one or more rules, and / or one or more procedures, the input can be used as input to the equation and / or function, and / or rule, and / or procedure, and the inferred output can be based on the output of the equation and / or function, and / or rule, and / or procedure (e.g., selecting one of the outputs of the equation and / or function, and / or rule, and / or procedure, using a statistical measure of the output of the equation and / or function, and / or rule, and / or procedure, etc.).

[0085] In accordance with the present disclosure, the processing device of system 200 can analyze image data captured by an image sensor (e.g., image sensor 372, image sensor 472, or any other image sensor) to implement any of the methods disclosed herein. In some embodiments, analyzing the image data can include analyzing the image data to obtain preprocessed image data and then analyzing the image data and / or the preprocessed image data to obtain a desired result. Those skilled in the art will recognize that the following are examples and that image data can be preprocessed using other types of preprocessing methods. In some examples, the image data can be preprocessed by transforming the image data using a transformation function to obtain transformed image data, and the preprocessed image data can include the transformed image data. For example, the transformed image data can include one or more convolutions of the image data. For example, the transformation function can include one or more image filters such as a low-pass filter, a high-pass filter, a band-pass filter, an all-pass filter, etc. In some examples, the transformation function can include a non-linear function. In some examples, the image data can be preprocessed by smoothing at least a portion of the image data, e.g., using a Gaussian convolution, using a median filter, etc. In some examples, the image data can be preprocessed to obtain a different representation of the image data. For example, the preprocessed image data can include a representation of at least a portion of the image data in the frequency domain, a discrete Fourier transform of at least a portion of the image data, a discrete wavelet transform of at least a portion of the image data, a time / frequency representation of at least a portion of the image data, a representation of at least a portion of the image data in a low dimension, an irreversible representation of at least a portion of the image data, a reversible representation of at least a portion of the image data, a time series of any of the above, any combination of the above, etc. In some examples, the image data can be preprocessed to extract edges, and the preprocessed image data can include information based on and / or related to the extracted edges. In some examples, the image data can be preprocessed to extract image features from the image data.Some non-limiting examples of such image features may include information based on and / or related to edges, corners, blobs, ridges, Scale-Invariant Feature Transform (SIFT) features, temporal features, and the like. In some examples, analyzing the image data may include calculating at least one convolution of at least a portion of the image data and using the at least one calculated convolution to calculate at least one resulting value and / or perform determinations, identifications, recognitions, classifications, etc.

[0086] In accordance with another aspect of the present disclosure, the processing device of system 200 can analyze image data to implement any of the methods disclosed herein. In some embodiments, analyzing the image may include analyzing the image data and / or pre-processed image data using one or more rules, functions, procedures, artificial neural networks, object detection algorithms, face detection algorithms, visual event detection algorithms, action detection algorithms, motion detection algorithms, background subtraction algorithms, inference models, and the like. Some non-limiting examples of such inference models may include manually pre-programmed inference models, classification models, regression models, results of training algorithms such as machine learning algorithms and / or deep learning algorithms for training examples, where the training examples may include examples of data instances, and in some cases, the data instances may be labeled with corresponding desired labels and / or results. In some embodiments, analyzing the image data (e.g., by the methods, steps, and modules described herein) may include analyzing pixels, voxels, point clouds, range data, etc. included in the image data.

[0087] Convolution can include convolutions of any dimension. A one-dimensional convolution is a function that transforms an original sequence of numbers into a transformed sequence of numbers. A one-dimensional convolution can be defined by a column of scalars. Each particular value of the transformed sequence can be determined by calculating a linear combination of the values of a subsequence of the original sequence corresponding to the particular value. The resulting value of the calculated convolution can include any value of the transformed sequence. Similarly, an n-dimensional convolution is a function that transforms an original n-dimensional array into a transformed array. An n-dimensional convolution can be defined by an n-dimensional array of scalars (known as the kernel of the n-dimensional convolution). Each particular value of the transformed array can be determined by calculating a linear combination of the values within the n-dimensional region of the original array corresponding to the particular value. The resulting value of the calculated convolution can include any value within the transformed array. In some examples, an image can include one or more components (such as color components, depth components, etc.), and each component can include a two-dimensional array of pixel values. In one example, calculating the convolution of an image can include calculating a two-dimensional convolution on one or more components of the image. In another example, calculating the convolution of an image can include stacking arrays from various components to create a three-dimensional array and calculating a three-dimensional convolution on the resulting three-dimensional array. In some examples, a video can include one or more components (such as color components, depth components, etc.), and each component can include a three-dimensional array of pixel values (having two spatial axes and one temporal axis). In one example, calculating the convolution of a video can include calculating a three-dimensional convolution on one or more components of the video. In another example, calculating the convolution of a video can include stacking arrays from various components to create a four-dimensional array and calculating a four-dimensional convolution on the resulting four-dimensional array.

[0088] The following detailed description refers to the accompanying drawings. As far as possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. Although some exemplary embodiments are described herein, modifications, adaptations, and other implementations are possible. For example, substitutions, additions, or modifications may be made to the components shown in the drawings, and the exemplary methods described herein may be modified by substituting, rearranging, removing, or adding steps to the disclosed methods. Accordingly, the following detailed description is not limited to the disclosed embodiments and examples, but includes the general principles described and illustrated herein, in addition to the general principles encompassed by the appended claims.

[0089] The present disclosure is directed to systems and methods for providing an extended reality environment to a user. The term "extended reality environment", which may also be referred to as "extended reality", "extended reality space", or "extended environment", refers to any type of combined real and virtual environment that is at least partially generated by computer technology, and the interaction between humans and machines. The extended reality environment can be a fully simulated virtual environment or a combined real and virtual environment that a user can perceive from various perspectives. In some examples, the user can interact with elements of the extended reality environment. One non-limiting example of an extended reality environment can be a virtual reality environment, also known as "virtual reality" or "virtual environment". An immersive virtual reality environment can be a simulated non-physical environment that provides the user with the perception of being present within the virtual environment. Another non-limiting example of an extended reality environment can be an augmented reality environment, also known as "augmented reality" or "augmented environment". An augmented reality environment can include a live direct or indirect view of a physical real-world environment enhanced with virtual computer-generated perceptual information, such as virtual objects with which the user can interact. Another non-limiting example of an extended reality environment is a mixed reality environment, also known as "mixed reality" or "mixed environment". A mixed reality environment can be a hybrid of a physical real-world and a virtual environment in which physical objects and virtual objects coexist and can interact in real time. In some examples, both augmented reality environments and mixed reality environments can include a combination of the real world and the virtual world, real-time interaction, and accurate 3D registration of virtual and real objects. In some examples, both augmented reality environments and mixed reality environments can include structured overlay sensory information that can be added to the physical environment. In other examples, both augmented reality environments and mixed reality environments can include disruptive virtual content that can mask at least a portion of the physical environment.

[0090] In some embodiments, the system and method can provide an extended reality environment using extended reality devices. The term "extended reality device" can include any type of device or system that enables a user to perceive and / or interact with an extended reality environment. The extended reality device may enable a user to perceive and / or interact with the extended reality environment through one or more sensory modalities. Some non-limiting examples of such sensory modalities can include vision, hearing, touch, proprioception, and smell. An example of an extended reality device is a virtual reality device that enables a user to perceive and / or interact with a virtual reality environment. Another example of an extended reality device is an augmented reality device that enables a user to perceive and / or interact with an augmented reality environment. Yet another example of an extended reality device is a mixed reality device that enables a user to perceive and / or interact with a mixed reality environment.

[0091] Consistent with one aspect of the present disclosure, an extended reality device may be a wearable device, such as a head-mounted device, e.g., smart glasses, smart contact lenses, a headset, or any other device worn by a human for the purpose of presenting extended reality to the human. Other extended reality devices may include a holographic projector or any other device or system capable of providing augmented reality (AR), virtual reality (VR), mixed reality (MR), or any immersive experience. Typical components of a wearable extended reality device may include a stereoscopic head-mounted display, a stereoscopic head-mounted sound system, head motion tracking sensors (such as gyroscopes, accelerometers, magnetometers, image sensors, structured light sensors, etc.), a head-mounted projector, eye tracking sensors, and at least one of the additional components described below. Consistent with another aspect of the present disclosure, the extended reality device may be a non-wearable extended reality device. Specifically, the non-wearable extended reality device may include a multi-projection environment device. In some embodiments, the extended reality device may be configured to change the viewing perspective of the extended reality environment in response to the movement of the user, and in particular, in response to the movement of the user's head. In one example, a wearable extended reality device may change the field of view of the extended reality environment in response to a change in the user's head pose, such as by changing the orientation of the space without changing the user's spatial position within the extended reality environment. In another example, a non-wearable extended reality device may change the spatial position of the user within the extended reality environment in response to a change in the position of the user in the real world, such as by changing the spatial position of the user within the extended reality environment without changing the direction of the field of view with respect to the spatial position.

[0092] According to some embodiments, the extended reality device may include a digital communication device configured to perform at least one of receiving virtual content data configured to enable presentation of virtual content, transmitting virtual content for sharing with at least one external device, receiving context data from at least one external device, transmitting context data to at least one external device, transmitting usage data indicating usage of the extended reality device, and transmitting data based on information captured using at least one sensor included in the extended reality device. In additional embodiments, the extended reality device may include memory for storing at least one of virtual data configured to enable presentation of virtual content, context data, usage data indicating usage of the extended reality device, sensor data based on information captured using at least one sensor included in the wearable extended reality device, software instructions configured to cause a processing device to present virtual content, software instructions configured to cause a processing device to collect and analyze context data, software instructions configured to cause a processing device to collect and analyze usage data, and software instructions configured to cause a processing device to collect and analyze sensor data. In additional embodiments, the extended reality device may include a processing device configured to perform at least one of rendering virtual content, collecting and analyzing context data, collecting and analyzing usage data, and collecting and analyzing sensor data. In additional embodiments, the extended reality device may include one or more sensors.One or more sensors may include one or more image sensors (e.g., configured to capture images and / or video of a user of the device or the user's environment), one or more motion sensors (such as accelerometers, gyroscopes, magnetometers), one or more positioning sensors (GPS, outdoor positioning sensors, indoor positioning sensors, etc.), one or more temperature sensors (e.g., configured to measure the temperature of at least a portion of the device and / or the environment), one or more contact sensors, one or more proximity sensors (e.g., configured to detect whether the device is currently being worn), one or more electrical impedance sensors (e.g., configured to measure the electrical impedance of a user), a gaze detector, an optical tracker, an electro-potential tracker (such as an electrooculogram (EOG) sensor), a video-based eye tracker, an infrared / near-infrared sensor, a passive light sensor, or one or more eye-tracking sensors such as any other technique capable of determining where a person is looking or gazing.

[0093] In some embodiments, the system and method can interact with an extended reality device using an input device. The term input device can include any physical device configured to receive input from a user or the user's environment and provide data to a computing device. The data provided to the computing device may be in digital and / or analog form. In one embodiment, the input device can store the input received from the user in a memory device accessible by a processing device, and the processing device can access the data stored for analysis. In another embodiment, the input device can provide data directly to the processing device, for example, via a bus or via another communication system configured to transfer data from the input device to the processing device. In some examples, the input received by the input device can include key presses, tactile input data, motion data, position data, gesture-based input data, direction data, or any other data for providing for computation. Some examples of input devices can include buttons, keys, keyboards, computer mice, touch pads, touch screens, joysticks, or any other mechanism through which input can be received. Another example of an input device can include an integrated computing interface device that includes at least one physical component for receiving input from a user. The integrated computing interface device can include at least a memory, a processing device, and at least one physical component for receiving input from a user. In one example, the integrated computing interface device can further include a digital network interface that enables digital communication with other computing devices. In one example, the integrated computing interface device can further include a physical component for outputting information to the user. In some examples, all components of the integrated computing interface device can be included in a single housing, but in other examples, the components can be distributed between two or more housings.Some non-limiting examples of physical components for receiving input from a user that may be included in an integrated computing interface device include at least one of a button, a key, a keyboard, a touchpad, a touch screen, a joystick, or any other mechanism or sensor through which computing information may be received. Some non-limiting examples of physical components for outputting information to a user include at least one of a light indicator (such as an LED indicator), a screen, a touch screen, a beeper, an audio speaker, or any other audio, video, or tactile device that provides an output perceivable by a human.

[0094] In some embodiments, image data may be captured using one or more image sensors. In some examples, the image sensors may be included in an extended reality device, a wearable device, a wearable extended reality device, an input device, a user's environment, and the like. In some examples, the image data may be read from a memory, received from an external device, or generated (e.g., using a generative model). Some non-limiting examples of image data may include an image, a grayscale image, a color image, a 2D image, a 3D image, a video, a 2D video, a 3D video, a frame, a scene, data obtained from other image data, and the like. In some examples, the image data may be encoded in any analog or digital format. Some non-limiting examples of such formats may include raw format, compressed format, uncompressed format, irreversible format, reversible format, JPEG, GIF, PNG, TIFF, BMP, NTSC, PAL, SECAM, MPEG, MPEG-4 Part 14, MOV, WMV, FLV, AVI, AVCHD, WebM, MKV, and the like.

[0095] In some embodiments, the extended reality device can receive, for example, a digital signal from an input device. The term digital signal can refer to a series of digital values that are temporally discrete. The digital signal can represent, for example, sensor data, text data, audio data, video data, virtual data, or any other form of data that provides perceptible information. Consistent with the present disclosure, the digital signal can be configured to cause the extended reality device to present virtual content. In one embodiment, the virtual content can be presented in a selected orientation. In this embodiment, the digital signal can indicate the position and angle of the viewpoint in an environment such as an extended reality environment. Specifically, the digital signal can include an encoding of the position and angle in six degrees of freedom coordinates (e.g., forward / backward, up / down, left / right, yaw, pitch, and roll). In another embodiment, the digital signal can include an encoding of the position as three-dimensional coordinates (e.g., x, y, and z) and an encoding of the angle as a vector resulting from the encoded position. Specifically, the digital signal can indicate the orientation and angle of the presented virtual content in the absolute coordinates of the environment, for example, by encoding the yaw, pitch, and roll of the virtual content relative to a standard default angle. In another embodiment, the digital signal can indicate the orientation and angle of the presented virtual content with respect to the viewpoint of another object (e.g., a virtual object, a physical object, etc.), for example, by encoding the yaw, pitch, and roll of the virtual content with respect to the direction corresponding to the viewpoint or the direction corresponding to another object. In another embodiment, such a digital signal can include one or more projections of the virtual content, for example, in a form ready for presentation (e.g., an image, a video, etc.). For example, each of such projections can correspond to a specific orientation or a specific angle. In another embodiment, the digital signal can include a representation of the virtual content, for example, by encoding an object in a three-dimensional array of voxels, a polygon mesh, or any other form in which the virtual content can be presented.

[0096] In some embodiments, the digital signal may be configured to cause an extended reality device to present virtual content. The term virtual content may include any type of data representation that can be presented to a user by an extended reality device. Virtual content may include virtual objects, non-animated virtual content, animated virtual content configured to change over time or in response to a trigger, virtual two-dimensional content, virtual three-dimensional content, virtual overlays on a portion of a physical environment or on a physical object, virtual additions to a physical environment or physical object, virtual promotional content, virtual representations of physical objects, virtual representations of physical environments, virtual documents, virtual characters or personas, virtual computer screens, virtual widgets, or any other form for virtually presenting information. Consistent with the present disclosure, virtual content may include any visual presentation rendered by a computer or processing device. In one embodiment, the virtual content includes virtual objects that are visual presentations rendered by a computer within a limited area and configured to represent a particular type of object (such as a non-animated virtual object, an animated virtual object, a virtual fixture, a virtual decorative object, a virtual widget, or other virtual representation). The rendered visual presentation can be changed, for example, to mimic a change in the appearance of a physical object, to reflect a change to a status object, or to change the viewing angle of an object. In another embodiment, the virtual content can include a virtual display (also referred to herein as a "virtual display screen" or "virtual screen") such as a virtual computer screen, a virtual tablet screen, or a virtual smartphone screen that is configured to display information generated by an operating system, and the operating system can be configured to receive text data from a physical keyboard and / or a virtual keyboard and display text content within the virtual display screen. In an example shown in FIG. 1, the virtual content can include a virtual environment that includes a virtual computer screen and a plurality of virtual objects.In some examples, the virtual display can be a virtual object that mimics and / or extends the functionality of a physical display screen. For example, the virtual display can be presented within an extended reality environment (such as a mixed reality environment, an augmented reality environment, a virtual reality environment, etc.) using an extended reality device. In one example, the virtual display can present content generated by a normal operating system that is presented equally on a physical display screen. In one example, text content input using a keyboard (e.g., using a physical keyboard, using a virtual keyboard, etc.) can be presented on the virtual display in real time as the text content is typed. In one example, a virtual cursor can be presented on the virtual display, and the virtual cursor can be controlled by a pointing device (such as a physical pointing device, a virtual pointing device, a computer mouse, a joystick, a touchpad, a physical touch controller, etc.). In one example, one or more windows of a graphical user interface operating system can be presented on the virtual display. In another example, the content presented on the virtual display can be interactive, i.e., it can change in response to user actions. In yet another example, the presentation of the virtual display may or may not include the presentation of a screen frame.

[0097] Some of the disclosed embodiments may include and / or access a data structure or database. The terms data structure and database consistent with this disclosure may include any set of data values and the relationships among them. The data may be stored linearly, horizontally, hierarchically, relationally, non-relationally, unidimensionally, multi-dimensionally, operationally, in an ordered manner, in an unordered manner, in an object-oriented manner, in a centralized manner, in a decentralized manner, in a distributed manner, in a custom manner, or in any manner that enables data access. By way of non-limiting example, data structures may include arrays, associative arrays, linked lists, binary trees, balanced trees, heaps, stacks, queues, sets, hash tables, records, tagged unions, entity-relationship models, graphs, hypergraphs, matrices, tensors, and the like. For example, data structures may include XML databases, RDBMS databases, SQL databases, or NoSQL alternatives for data storage / retrieval such as MongoDB, Redis, Couchbase, Datastax Enterprise Graph, Elastic Search, Splunk, Solr, Cassandra, Amazon DynamoDB, Scylla, HBase, and Neo4J. The data structure may be a component of the disclosed system or a remote computing component (e.g., a cloud-based data structure). The data within the data structure may be stored in contiguous or non-contiguous memory. Further, the data structure does not require that information be located in the same place. It may be distributed across multiple servers, which may be owned or operated by the same or different entities. Thus, the term data structure in the singular includes multiple data structures.

[0098] In some embodiments, the system can determine a confidence level in a received input or any determined value. The term confidence level refers to any indication, numerical value, or otherwise, of the level (e.g., within a predetermined range) of the amount of confidence the system has in the determined data. For example, the confidence level can have a value from 1 to 10. Alternatively, the confidence level can be represented as a percentage or any other numerical or non - numerical indication. In some cases, the system may compare the confidence level to a threshold. The term threshold can refer to a reference value, level, point, or range of values. During operation, when the confidence level of the determined data exceeds (or, depending on the particular use case, falls below) the threshold, the system can follow a first course of action, and when the confidence level falls below (or, depending on the particular use case, exceeds) it, the system can follow a second course of action. The value of the threshold can be predetermined for each type of object being inspected or can be dynamically selected based on various considerations.

[0099] Referring now to FIG. 1, there is shown a user using an exemplary extended reality system consistent with an embodiment of the present disclosure. FIG. 1 is an exemplary representation of only one embodiment, and it should be understood that some of the elements shown are omitted and other elements may be added within the scope of the present disclosure. As shown, user 100 is sitting behind table 102 that supports keyboard 104 and mouse 106. Keyboard 104 is connected by wire 108 to wearable extended reality device 110 that displays virtual content to user 100. As an alternative to or in addition to wire 108, keyboard 104 may be wirelessly connected to wearable extended reality device 110. For illustrative purposes, the wearable extended reality device is shown as a pair of smart glasses, but as described above, wearable extended reality device 110 may be any type of head-mounted device used to present extended reality to user 100. The virtual content displayed by wearable extended reality device 110 includes virtual screen 112 (also referred to herein as a "virtual display screen" or "virtual display") and a plurality of virtual widgets 114. Virtual widgets 114A - 114D are displayed adjacent to virtual screen 112, and virtual widget 114E is displayed on table 102. User 100 can use keyboard 104 to input text into document 116 displayed on virtual screen 112 and can use mouse 106 to control virtual cursor 118. In one example, virtual cursor 118 can be moved to any location within virtual screen 112. In another example, virtual cursor 118 can be moved to any location within virtual screen 112 and can also be moved to any one of virtual widgets 114A - 114D, but cannot be moved to virtual widget 114E. In yet another example, virtual cursor 118 can be moved to any location within virtual screen 112 and can also be moved to any one of virtual widgets 114A - 114E.In an additional example, the virtual cursor 118 can move to any location within the extended reality environment that includes the virtual screen 112 and the virtual widgets 114A - 114E. In yet another example, the virtual cursor can move only on all available surfaces (i.e., virtual or physical surfaces), or on selected surfaces within the extended reality environment. Alternatively, or in addition, the user 100 can use hand gestures recognized by the wearable extended reality device 110 to interact with any one of the virtual widgets 114A - 114E, or with a selected virtual widget. For example, the virtual widget 114E may be an interactive widget (e.g., a virtual slider controller) that can be operated with a hand gesture.

[0100] FIG. 2 shows an example of a system 200 that provides an extended reality (XR) experience to a user such as user 100. It should be understood that FIG. 2 is an exemplary representation of just one embodiment, and within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. System 200 may be computer-based and may include computer system components, wearable devices, workstations, tablets, handheld computing devices, memory devices, and / or an internal network connecting these components. System 200 may include or be connected to various network computing resources (e.g., servers, routers, switches, network connections, storage devices, etc.) to support the services provided by system 200. Consistent with the present disclosure, system 200 may include an input unit 202, an XR unit 204, a mobile communication device 206, and a remote processing unit 208. Remote processing unit 208 may include a server 210 coupled to one or more physical or virtual storage devices such as data structure 212. System 200 may also include or be connected to a communication network 214 that facilitates communication and data exchange between the various system components and the various entities associated with system 200.

[0101] In accordance with the present disclosure, the input unit 202 can include one or more devices that can receive input from the user 100. In one embodiment, the input unit 202 can include a text input device such as a keyboard 104. The text input device can include all possible types of devices and mechanisms for inputting text information into the system 200. Examples of text input devices can include mechanical keyboards, membrane keyboards, flexible keyboards, QWERTY keyboards, Dvorak keyboards, Colemak keyboards, coded keyboards, wireless keyboards, keypads, key-based control panels, or other arrangements of control keys, visual input devices, or any other mechanism that can input text whether provided in physical form or presented virtually. In one embodiment, the input unit 202 can also include a pointing input device such as a mouse 106. The pointing input device can include all possible types of devices and mechanisms for inputting two-dimensional or three-dimensional information into the system 200. In one example, two-dimensional input from the pointing input device can be used to interact with virtual content presented via the XR unit 204. Examples of pointing input devices can include computer mice, trackballs, touchpads, trackpads, touchscreens, joysticks, pointing sticks, styli, light pens, or any other physical or virtual input mechanism. In one embodiment, the input unit 202 can also include a graphical input device such as a touchscreen configured to detect contact, movement, or interruption of movement. The graphical input device can use any of a plurality of touch sensitivity technologies including capacitive, resistive, infrared, and surface acoustic wave technologies, as well as other proximity sensor arrays or other elements for determining one or more contact points, but is not limited thereto. In one embodiment, the input unit 202 can also include one or more voice input devices such as a microphone.The voice input device can include all possible types of devices and mechanisms for inputting voice data to facilitate voice-enabled functions such as voice recognition, voice replication, digital recording, and telephone functions. In one embodiment, the input unit 202 can also include one or more image input devices, such as an image sensor, configured to capture image data. In one embodiment, the input unit 202 can also include one or more tactile gloves configured to capture hand movement and gesture data. In one embodiment, the input unit 202 can also include one or more proximity sensors configured to detect the presence and / or movement of objects within a selected area near the sensor.

[0102] According to some embodiments, the system can include at least one sensor configured to detect and / or measure characteristics associated with a user, the user's actions, or the user's environment. An example of the at least one sensor is sensor 216 included in input unit 202. Sensor 216 can be a motion sensor, a touch sensor, an optical sensor, an infrared sensor, an audio sensor, an image sensor, a proximity sensor, a positioning sensor, a gyroscope, a temperature sensor, a biometric sensor, or any other sensing device to facilitate related functions. Sensor 216 may be integrated or connected with the input device, or may be separated from the input device. In one example, a thermometer may be included in mouse 106 to measure the body temperature of user 100. In another example, a positioning sensor may be integrated with keyboard 104 to determine the movement of user 100 relative to keyboard 104. Such a positioning sensor can be implemented using one of the following technologies: Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), Galileo Global Navigation System, Beidou Navigation System, other Global Navigation Satellite Systems (GNSS), Indian Regional Navigation Satellite System (IRNSS), Local Positioning System (LPS), Real-Time Location System (RTLS), Indoor Positioning System (IPS), Wi-Fi based positioning system, cellular triangulation, image-based positioning technology, indoor positioning technology, outdoor positioning technology, or any other positioning technology.

[0103] According to some embodiments, the system can include one or more sensors for identifying the position and / or movement of physical devices (such as physical input devices, physical computing devices, keyboard 104, mouse 106, wearable extended reality device 110, etc.). The one or more sensors may be included in the physical device or may be external to the physical device. In some examples, an image sensor external to the physical device (e.g., an image sensor included in another physical device) can be used to capture image data of the physical device, and the image data can be analyzed to identify the position and / or movement of the physical device. For example, the image data can be analyzed using a visual object tracking algorithm to identify the movement of the physical device, and can be analyzed using a visual object detection algorithm to identify the position of the physical device (e.g., the position relative to the image sensor, the position in a global coordinate system, etc.), and so on. In some examples, an image sensor included in the physical device can be used to capture image data, and the image data can be analyzed to identify the position and / or movement of the physical device. For example, the image data can be analyzed using a visual odometry algorithm to identify the position of the physical device, and can be analyzed using an egomotion algorithm to identify the movement of the physical device, and so on. In some examples, positioning sensors such as indoor positioning sensors or outdoor positioning sensors can be included in the physical device and can be used to determine the position of the physical device. In some examples, motion sensors such as accelerometers or gyroscopes can be included in the physical device and can be used to determine the movement of the physical device. In some examples, a physical device such as a keyboard or a mouse can be configured to be positioned on a physical surface. Such a physical device can include an optical mouse sensor (also known as a non-mechanical tracking engine) directed at the physical surface, and the output of the optical mouse sensor can be analyzed to determine the movement of the physical device relative to the physical surface.

[0104] In accordance with the present disclosure, the XR unit 204 may include a wearable extended reality device configured to present virtual content to the user 100. An example of a wearable extended reality device is the wearable extended reality device 110. Additional examples of wearable extended reality devices may include virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, or any other device capable of generating extended reality content. Some non-limiting examples of such devices may include Nreal Light, Magic Leap One, Varjo, Quest 1 / 2, Vive, and the like. In some embodiments, the XR unit 204 may be capable of presenting virtual content to the user 100. Generally, extended reality devices may include all composite environments of reality and virtuality generated by computer technology and wearables, as well as human-machine interactions. As described above, the term "extended reality" (XR) refers to a superset that includes all regions from "complete reality" to "complete virtuality". This includes typical forms such as augmented reality (AR), mixed reality (MR), virtual reality (VR), and regions interpolated between them. Therefore, it should be noted that the terms "XR device", "AR device", "VR device", and "MR device" may be used interchangeably herein and may refer to any of the various devices listed above.

[0105] In accordance with the present disclosure, the system can exchange data with various communication devices associated with a user, such as mobile communication device 206. The term "communication device" is intended to include all possible types of devices that can exchange data using a digital communication network, an analog communication network, or any other communication network configured to transmit data. In some examples, communication devices can include smartphones, tablets, smartwatches, personal digital assistants, desktop computers, laptop computers, IoT devices, dedicated terminals, wearable communication devices, and any other device that enables data communication. In some cases, mobile communication device 206 can complement or replace input unit 202. Specifically, mobile communication device 206 can be associated with a physical touch controller that can function as a pointing input device. Additionally, mobile communication device 206 can also implement, for example, a virtual keyboard and be used to replace a text input device. For example, if user 100 leaves table 102 and walks into the break room with a smart glass, this user can receive an email that requires a quick response. In this case, this user can choose to use their smartwatch as an input device and type a response to the email while the email is being virtually presented by the smart glass.

[0106] In accordance with the present disclosure, embodiments of the system may include the use of a cloud server. The term "cloud server" refers to a computer platform that provides services over a network such as the Internet. In the exemplary embodiment shown in FIG. 2, the server 210 can use virtual machines that may not correspond to individual hardware. For example, the computing and / or memory capabilities can be implemented by allocating an appropriate portion of the desired computing / memory capabilities from a scalable repository such as a data center or a distributed computing environment. Specifically, in one embodiment, the remote processing unit 208 can be used with the XR unit 204 to provide virtual content to the user 100. In one exemplary configuration, the server 210 can be a cloud server that functions as an operating system (OS) for wearable extended reality devices. In one example, the server 210 can implement the methods described herein using custom hardwired logic, one or more application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), firmware, and / or program logic that combines the server 210 into a dedicated machine with a computer system.

[0107] In some embodiments, server 210 can access data structure 212 to determine virtual content for displaying, for example, user 100. Data structure 212 can utilize volatile or non-volatile, magnetic, semiconductor, tape, optical, removable, non-removable, other types of storage devices or tangible or non-transitory computer-readable media, or any medium or mechanism for storing information. As shown in the figure, data structure 212 may be part of server 210 or may be separate from server 210. If data structure 212 is not part of server 210, server 210 can exchange data with data structure 212 via a communication link. Data structure 212 may include one or more memory devices that store data and instructions used to execute one or more features of the disclosed methods. In one embodiment, data structure 212 may include any of a plurality of suitable data structures ranging from small data structures hosted on a workstation to large data structures distributed among data centers. Data structure 212 may also include any combination of one or more data structures controlled by a memory controller device (e.g., a server) or software.

[0108] In accordance with the present disclosure, the communication network can be any type of network (including infrastructure) that supports communication, exchanges information, and / or facilitates the exchange of information between components of the system. For example, communication network 214 within system 200 can include, for example, a telephone network, an extranet, an intranet, the Internet, satellite communication, offline communication, wireless communication, transponder communication, a local area network (LAN), a wireless network (e.g., a Wi-Fi / 802.11 network), a wide area communication network (WAN), a virtual private network (VPN), a digital communication network, an analog communication network, or any other mechanism or combination of mechanisms that enables data transmission.

[0109] The components and arrangements of the system 200 shown in FIG. 2 are merely illustrative and are not intended to limit the disclosed embodiments, since the system components used to implement the disclosed processes and features may vary.

[0110] FIG. 3 is a block diagram of an exemplary configuration of the input unit 202. It should be understood that FIG. 3 is merely an exemplary representation of one embodiment, and within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. In the embodiment of FIG. 3, the input unit 202 can directly or indirectly access a bus 300 (or other communication mechanism) that interconnects subsystems and components for transferring information within the input unit 202. For example, the bus 300 can interconnect a memory interface 310, a network interface 320, an input interface 330, a power supply 340, an output interface 350, a processing device 360, a sensor interface 370, and a database 380.

[0111] The memory interface 310 shown in FIG. 3 can be used to access software products and / or data stored in a non-transitory computer-readable medium. Generally, a non-transitory computer-readable storage medium refers to any kind of physical memory that can store information or data readable by at least one processor. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, non-volatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, any other optical data storage media, any physical media with a pattern of holes, PROM, EPROM, flash EPROM or any other flash memory, NVRAM, caches, registers, any other memory chip or cartridge, and networked versions thereof. The terms "memory" and "computer-readable storage medium" can refer to multiple structures, such as multiple memories or computer-readable storage media located within the input unit or at remote locations. Additionally, one or more computer-readable storage media can be utilized when implementing a computer-implemented method. Thus, the term "computer-readable storage medium" should be understood to include tangible items and exclude carrier waves and transient signals. In the particular embodiment shown in FIG. 3, the memory interface 310 can be used to access software products and / or data stored in a memory device such as the memory device 311. The memory device 311 can include high-speed random access memory and / or non-volatile memory, such as one or more magnetic disk storage devices, one or more optical storage devices, and / or flash memory (e.g., NAND, NOR). Consistent with the present disclosure, the components of the memory device 311 can be distributed across two or more units of the system 200 and / or two or more memory devices.

[0112] The memory device 311 shown in FIG. 3 may include software modules for executing processes consistent with the present disclosure. In particular, the memory device 311 may include an input determination module 312, an output determination module 313, a sensor communication module 314, a virtual content determination module 315, a virtual content communication module 316, and a database access module 317. Modules 312-317 may include software instructions for execution by at least one processor (e.g., processing device 360) associated with the input unit 202. The input determination module 312, the output determination module 313, the sensor communication module 314, the virtual content determination module 315, the virtual content communication module 316, and the database access module 317 may cooperate to perform various operations. For example, the input determination module 312 can determine text using data received from, for example, the keyboard 104. Thereafter, the output determination module 313 can cause the most recently input text to be presented on a dedicated display 352 physically or wirelessly coupled to the keyboard 104, for example. In this way, when the user 100 types, a preview of the typed text can be viewed without constantly moving the head up and down to view the virtual screen 112. The sensor communication module 314 can receive data from various sensors to determine the status of the user 100. Thereafter, the virtual content determination module 315 can determine the virtual content to be displayed based on the received input and the determined status of the user 100. For example, the determined virtual content may be a virtual presentation of the most recently input text on a virtual screen that is virtually located adjacent to the keyboard 104. The virtual content communication module 316 can obtain virtual content (e.g., another user's avatar) that is not determined by the virtual content determination module 315. The search for virtual content may be from the database 380, from the remote processing unit 208, or from any other source.

[0113] In some embodiments, the input determination module 312 can adjust the operation of the input interface 330 to receive pointer input 331, text input 332, audio input 333, and XR-related input 334. Details regarding pointer input, text input, and audio input have been described above. The term "XR-related input" can include any type of data that can cause a change in the virtual content presented to the user 100. In one embodiment, the XR-related input 334 can include image data of the user 100, wearable extended reality devices (e.g., detected gestures of the user 100's hand). In another embodiment, the XR-related input 334 can include wireless communication indicating the presence of another user in proximity to the user 100. Consistent with the present disclosure, the input determination module 312 can receive different types of input data simultaneously. Thereafter, the input determination module 312 can further apply various rules based on the type of input detected. For example, pointer input may be prioritized over voice input.

[0114] In some embodiments, the output determination module 313 can adjust the operation of the output interface 350 to generate an output using the optical indicator 351, the display 352, and / or the speaker 353. Generally, the output generated by the output determination module 313 does not include virtual content presented by the wearable extended reality device. Instead, the output generated by the output determination module 313 includes various outputs related to the operation of the input unit 202 and / or the XR unit 204. In one embodiment, the optical indicator 351 may include an optical indicator indicating the status of the wearable extended reality device. For example, the optical indicator may display a green light when the wearable extended reality device 110 is connected to the keyboard 104, and may blink when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 352 may be used to display operation information. For example, the display may present an error message when the wearable extended reality device is inoperable. In another embodiment, the speaker 353 may be used to output sound, for example, when the user 100 desires to play some music for another user.

[0115] In some embodiments, the sensor communication module 314 can adjust the operation of the sensor interface 370 to receive sensor data from one or more sensors integrated with or connected to the input device. The one or more sensors can include an audio sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374 (e.g., a temperature sensor, an ambient light detector, etc.), and other sensors 375. In one embodiment, the data received from the sensor communication module 314 can be used to determine the physical orientation of the input device. The physical orientation of the input device can indicate the user's state and can be determined based on a combination of tilt movement, roll movement, and lateral movement. The physical orientation of the input device is then used by the virtual content determination module 315 to modify the display parameters of the virtual content to match the user's state (e.g., attentive, sleepy, active, sitting, standing, reclined backward, reclined forward, walking, moving, riding, etc.).

[0116] In some embodiments, the virtual content determination module 315 can determine the virtual content to be displayed by the wearable extended reality device. The virtual content can be determined based on data from the input determination module 312, the sensor communication module 314, and other sources (e.g., the database 380). In some embodiments, determining the virtual content can include determining the distance, size, and orientation of the virtual object. The determination of the position of the virtual object can be determined based on the type of the virtual object. Specifically, with respect to the example shown in FIG. 1, since the virtual widget 114E is a virtual controller (e.g., a volume bar), the virtual content determination module 315 can determine to arrange the four virtual widgets 114A-114D on both sides of the virtual screen 112 and arrange the virtual widget 114E on the table 102. The determination of the position of the virtual object can be further determined based on the user's preference. For example, in the case of a left-handed user, the virtual content determination module 315 can determine to arrange the virtual volume bar to the left of the keyboard 104, and in the case of a right-handed user, the virtual content determination module 315 can determine to arrange the virtual volume bar to the right of the keyboard 104.

[0117] In some embodiments, the virtual content communication module 316 can adjust the operation of the network interface 320 to obtain data presented as virtual content to the user 100 from one or more sources. The one or more sources can include other XR units 204, the user's mobile communication device 206, remote processing units 208, publicly available information, and the like. In one embodiment, the virtual content communication module 316 can communicate with the mobile communication device 206 to provide a virtual representation of the mobile communication device 206. For example, the virtual representation can enable the user 100 to read messages and interact with applications installed on the mobile communication device 206. The virtual content communication module 316 can also adjust the operation of the network interface 320 to share virtual content with other users. In one example, the virtual content communication module 316 can use data from the input determination module to identify a trigger (e.g., the trigger can include a user gesture) and transfer content from the virtual display to a physical display (e.g., a television) or to the virtual display of a different user.

[0118] In some embodiments, the database access module 317 can cooperate with the database 380 to retrieve stored data. The retrieved data can include, for example, privacy levels associated with various virtual objects, relationships between virtual objects and physical objects, user preferences, the user's past behavior, and the like. As described above, the virtual content determination module 315 can use the data stored in the database 380 to determine virtual content. The database 380 can include separate databases, such as, for example, a vector database, a raster database, a tile database, a viewport database, and / or a user input database. The data stored in the database 380 can be received from the modules 314-317 or other components of the system 200. Further, the data stored in the database 380 can be provided as input using data input, data transfer, or data upload.

[0119] Modules 312 - 317 can be implemented in software, hardware, firmware, or a combination of any of these. In some embodiments, any one or more of Modules 312 - 317 and data associated with database 380 can be stored in XR unit 204, mobile communication device 206, or remote processing unit 208. The processing device of system 200 can be configured to execute the instructions of Modules 312 - 317. In some embodiments, aspects of Modules 312 - 317 can be implemented in hardware, (including one or more signal processing and / or application specific integrated circuits) software, firmware, or any combination thereof, executable by one or more processors, either alone or in various combinations with each other. Specifically, Modules 312 - 317 can be configured to interact with each other and / or with other modules of system 200 to perform functions consistent with the disclosed embodiments. For example, input unit 202 can execute instructions including an image processing algorithm on data from XR unit 204 to determine the movement of user 100's head. Further, with respect to input unit 202, or components of input unit 202, each function described throughout this specification can correspond to a set of instructions for performing said function. These instructions need not be implemented as separate software programs, procedures, or modules. Memory device 311 can include additional modules and instructions, or fewer modules and instructions. For example, memory device 311 can store an operating system such as ANDROID, iOS, UNIX, OSX, WINDOWS, DARWIN, RTXC, LINUX, or an embedded operating system such as VXWorkS. The operating system can include instructions for processing basic system services and performing hardware - dependent tasks.

[0120] The network interface 320 shown in FIG. 3 can provide bidirectional data communication to a network such as the communication network 214. In one embodiment, the network interface 320 can include an Integrated Services Digital Network (ISDN) card, a cellular modem, a satellite modem, or a modem for providing a data communication connection via the Internet. As another example, the network interface 320 can include a Wireless Local Area Network (WLAN) card. In another embodiment, the network interface 320 can include an Ethernet port connected to a high-frequency receiver and transmitter, and / or an optical (e.g., infrared) receiver and transmitter. The specific design and implementation of the network interface 320 can depend on the one or more communication networks in which the input unit 202 is intended to operate. For example, in some embodiments, the input unit 202 can include a network interface 320 designed to operate on a GSM network, a GPRS network, an EDGE network, a Wi-Fi or WiMax network, and a Bluetooth network. In any such implementation, the network interface 320 can be configured to transmit and receive electrical, electromagnetic, or optical signals that carry digital data streams or digital signals representing various types of information.

[0121] The input interface 330 shown in FIG. 3 can receive input from various input devices, such as a keyboard, a mouse, a touchpad, a touch screen, one or more buttons, a joystick, a microphone, an image sensor, and any other device configured to detect physical or virtual input. The received input can be in at least one form of text, sound, voice, hand gesture, body gesture, tactile information, and any other type of physical or virtual input generated by the user. In the illustrated embodiment, the input interface 330 can receive pointer input 331, text input 332, audio input 333, and XR-related input 334. In additional embodiments, the input interface 330 may be an integrated circuit that can function as a bridge between the processing device 360 and any of the input devices listed above.

[0122] The power supply 340 shown in FIG. 3 supplies electrical energy to the input unit 202 and can optionally also supply power to the XR unit 204. Generally, the power supply included in any device or system in the present disclosure can be one or more batteries (e.g., lead-acid battery, lithium-ion battery, nickel-metal hydride battery, nickel-cadmium battery), one or more capacitors, one or more connections to an external power supply, one or more power converters, or any combination thereof, but is not limited thereto, and can be any device capable of repeatedly storing, distributing, or transmitting power. Referring to the example shown in FIG. 3, the power supply may be portable, which means that the input unit 202 can be easily carried by hand (e.g., the total weight of the power supply 340 may be less than 1 pound). Due to the portability of the power supply, the user 100 can use the input unit 202 in various situations. In other embodiments, the power supply 340 may be associated with a connection to an external power supply (such as a power grid) that can be used to charge the power supply 340. Additionally, the power supply 340 may be configured to charge one or more batteries included in the XR unit 204. For example, an extended reality glass (e.g., a wearable extended reality device 110) can be charged (e.g., wirelessly or not wirelessly) when the glass is placed on or near the input unit 202.

[0123] The output interface 350 shown in FIG. 3 can output from various output devices, for example, using an optical indicator 351, a display 352, and / or a speaker 353. In one embodiment, the output interface 350 may be an integrated circuit that can function as a bridge between the processing device 360 and at least one of the output devices listed above. The optical indicator 351 may include one or more light sources, for example, an LED array associated with various colors. The display 352 may include a screen (e.g., an LCD or dot matrix screen) or a touch screen. The speaker 353 may include audio headphones, a hearing aid type device, a speaker, bone conduction headphones, an interface that provides tactile cues, a vibrotactile stimulator, etc.

[0124] The processing device 360 shown in FIG. 3 may include at least one processor configured to execute a computer program, application, method, process, or other software to implement the embodiments described in this disclosure. Generally, the processing device included in any device or system in this disclosure may include one or more integrated circuits, microchips, microcontrollers, microprocessors, all or part of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), or other circuits suitable for executing instructions or performing logical operations. The processing device may include at least one processor configured to implement the functions of the disclosed methods, such as a microprocessor manufactured by Intel (trademark). The processing device may include a single-core or multi-core processor that executes parallel processes simultaneously. In one example, the processing device may be a single-core processor configured using virtual processing technology. The processing device may implement virtual machine technology or other technologies to provide the ability to execute, control, operate, manage, and store multiple software processes, applications, programs, etc. In another example, the processing device may include a multi-core processor configuration (e.g., dual-core, quad-core, etc.) configured to provide a parallel processing function to enable devices associated with the processing device to execute multiple processes simultaneously. It should be understood that other types of processor configurations may be implemented to provide the capabilities disclosed herein.

[0125] The sensor interface 370 shown in FIG. 3 can acquire sensor data from various sensors, such as an audio sensor 371, an image sensor 372, a motion sensor 373, an environmental sensor 374, and other sensors 375. In one embodiment, the sensor interface 370 may be an integrated circuit that can function as a bridge between the processing device 360 and at least one of the sensors listed above.

[0126] The audio sensor 371 may include one or more audio sensors configured to capture audio by converting sound into digital information. Some examples of audio sensors can include a microphone, a unidirectional microphone, a bidirectional microphone, a cardioids microphone, an omnidirectional microphone, an on-board microphone, a wired microphone, a wireless microphone, or any combination of the above. Consistent with the present disclosure, the processing device 360 can modify the presentation of virtual content based on data received from the audio sensor 371 (e.g., voice commands).

[0127] The image sensor 372 may include one or more image sensors configured to capture visual information by converting light into image data. Consistent with the present disclosure, the image sensor may be included in any device or system in the present disclosure and may be any device capable of detecting optical signals in the near-infrared, infrared, visible, and ultraviolet spectra and converting them into electrical signals. Examples of image sensors may include digital cameras, phone cameras, semiconductor charge-coupled devices (CCDs), active pixel sensors in complementary metal-oxide-semiconductors (CMOS), or N-type metal-oxide-semiconductors (NMOS, live MOS). The electrical signals may be used to generate image data. Consistent with the present disclosure, the image data may include pixel data streams, digital images, digital video streams, data obtained from captured images, and data that may be used to construct one or more 3D images, sequences of 3D images, 3D videos, or virtual 3D representations. The image data acquired by the image sensor 372 may be transmitted, by wired or wireless transmission, to any processing device of the system 200. For example, the image data may be processed to detect an object, detect an event, detect an action, detect a face, detect people, recognize a known person, or for any other information that may be used by the system 200. Consistent with the present disclosure, the processing device 360 may be able to modify the presentation of virtual content based on the image data received from the image sensor 372.

[0128] The motion sensor 373 may include one or more motion sensors configured to measure the motion of the input unit 202 or the motion of an object within the environment of the input unit 202. Specifically, the motion sensor can perform at least one of detecting the motion of an object within the environment of the input unit 202, measuring the speed of an object within the environment of the input unit 202, measuring the acceleration of an object within the environment of the input unit 202, detecting the motion of the input unit 202, measuring the speed of the input unit 202, measuring the acceleration of the input unit 202, etc. In some embodiments, the motion sensor 373 may include one or more accelerometers configured to detect an appropriate change in acceleration and / or measure an appropriate acceleration of the input unit 202. In other embodiments, the motion sensor 373 may include one or more gyroscopes configured to detect a change in the orientation of the input unit 202 and / or measure information regarding the orientation of the input unit 202. In other embodiments, the motion sensor 373 may include one or more that use an image sensor, a LIDAR sensor, a radar sensor, or a proximity sensor. For example, by analyzing the captured image, the processing device can determine the motion of the input unit 202 using, for example, an egomotion algorithm. Additionally, the processing device can determine the motion of an object within the environment of the input unit 202 using, for example, an object tracking algorithm. Consistent with the present disclosure, the processing device 360 can modify the presentation of the virtual content based on the determined motion of the input unit 202 or the determined motion of an object within the environment of the input unit 202. For example, cause the virtual display to follow the motion of the input unit 202.

[0129] The environmental sensor 374 can include one or more sensors from different types configured to capture data reflecting the environment of the input unit 202. In some embodiments, the environmental sensor 374 is configured to perform at least one of measuring chemical properties in the environment of the input unit 202, measuring changes in chemical properties in the environment of the input unit 202, detecting the presence of chemical substances in the environment of the input unit 202, and measuring the concentration of chemical substances in the environment of the input unit 202, and can include one or more chemical sensors. Examples of such chemical properties can include pH level, toxicity, and temperature. Examples of such chemical substances can include electrolytes, specific enzymes, specific hormones, specific proteins, smoke, carbon dioxide, carbon monoxide, oxygen, ozone, hydrogen, and hydrogen sulfide. In other embodiments, the environmental sensor 374 can include one or more temperature sensors configured to detect changes in the temperature of the environment of the input unit 202 and / or measure the temperature of the environment of the input unit 202. In other embodiments, the environmental sensor 374 can include one or more barometers configured to detect changes in the atmospheric pressure in the environment of the input unit 202 and / or measure the atmospheric pressure in the environment of the input unit 202. In other embodiments, the environmental sensor 374 can include one or more light sensors configured to detect changes in ambient light in the environment of the input unit 202. Consistent with the present disclosure, the processing device 360 can modify the presentation of virtual content based on the input from the environmental sensor 374. For example, automatically reducing the brightness of virtual content when the environment of user 100 becomes dark.

[0130] Other sensors 375 may include a weight sensor, a light sensor, a resistance sensor, an ultrasonic sensor, a proximity sensor, a biosensor, or other sensing devices to facilitate related functions. In certain embodiments, other sensors 375 may include one or more positioning sensors configured to obtain positioning information of the input unit 202, detect changes in the position of the input unit 202, and / or measure the position of the input unit 202. Alternatively, the GPS software may enable the input unit 202 to access an external GPS receiver (e.g., connect via a serial port or Bluetooth). Consistent with the present disclosure, the processing device 360 can modify the presentation of virtual content based on inputs from other sensors 375. For example, personal information is presented only after identifying the user 100 using data from a biosensor.

[0131] The components and arrangements shown in FIG. 3 are not intended to limit the disclosed embodiments. As will be appreciated by those skilled in the art having the benefit of this disclosure, numerous variations and / or modifications may be made to the illustrated configuration of the input unit 202. For example, in all cases, not all components may be essential for the operation of the input unit. Any component may be arranged in any suitable part of the input unit, and the components may be rearranged in various configurations while providing the functions of the disclosed embodiments. For example, some input units may not include all of the elements as shown in the input unit 202.

[0132] FIG. 4 is a block diagram of an exemplary configuration of the XR unit 204. It should be understood that FIG. 4 is an exemplary representation of only one embodiment, and within the scope of the present disclosure, some of the illustrated elements may be omitted and other elements may be added. In the embodiment of FIG. 4, the XR unit 204 can access directly or indirectly a bus 400 (or other communication mechanism) that interconnects subsystems and components for transferring information within the XR unit 204. For example, the bus 400 can interconnect a memory interface 410, a network interface 420, an input interface 430, a power supply 440, an output interface 450, a processing device 460, a sensor interface 470, and a database 480.

[0133] The memory interface 410 shown in FIG. 4 is assumed to have the same functions as those of the memory interface 310 described in detail above. The memory interface 410 can be used to access software products and / or data stored in a non-transitory computer-readable medium or in a memory device such as the memory device 411. The memory device 411 may include software modules for executing processes consistent with the present disclosure. In particular, the memory device 411 may include an input determination module 412, an output determination module 413, a sensor communication module 414, a virtual content determination module 415, a virtual content communication module 416, and a database access module 417. Modules 412 to 417 may include software instructions for execution by at least one processor (e.g., the processing device 460) associated with the XR unit 204. The input determination module 412, the output determination module 413, the sensor communication module 414, the virtual content determination module 415, the virtual content communication module 416, and the database access module 417 may cooperate to perform various operations. For example, the input determination module 412 can determine a user interface (UI) input received from the input unit 202. At the same time, the sensor communication module 414 can receive data from various sensors to determine the status of the user 100. The virtual content determination module 415 can determine the virtual content to be displayed based on the received input and the determined status of the user 100. The virtual content communication module 416 can search for virtual content not determined by the virtual content determination module 415. The search for virtual content may be from the database 380, the database 480, the mobile communication device 206, or the remote processing unit 208. Based on the output of the virtual content determination module 415, the output determination module 413 can cause a change in the virtual content displayed to the user 100 by the projector 454.

[0134] In some embodiments, the input determination module 412 can adjust the operation of the input interface 430 to receive gesture input 431, virtual input 432, audio input 433, and UI input 434. Consistent with the present disclosure, the input determination module 412 can receive different types of input data simultaneously. In one embodiment, the input determination module 412 can apply various rules based on the type of detected input. For example, gesture input may be prioritized over virtual input. In some embodiments, the output determination module 413 can adjust the operation of the output interface 450 to generate outputs using the optical indicator 451, display 452, speaker 453, and projector 454. In one embodiment, the optical indicator 451 can include an optical indicator indicating the status of the wearable extended reality device. For example, the optical indicator may display green light when the wearable extended reality device 110 is connected to the input unit 202 and may blink when the battery of the wearable extended reality device 110 is low. In another embodiment, the display 452 can be used to display operation information. In another embodiment, the speaker 453 can include bone conduction headphones used to output sound to the user 100. In another embodiment, the projector 454 can present virtual content to the user 100.

[0135] The operations of the sensor communication module, virtual content determination module, virtual content communication module, and database access module have been described above with reference to FIG. 3, and the details are not repeated here. Modules 412-417 can be implemented in software, hardware, firmware, a mixture of any of these, etc.

[0136] The network interface 420 shown in FIG. 4 is assumed to have the same functions as the network interface 320 described in detail above. The specific design and implementation of the network interface 420 may depend on the communication network in which the XR unit 204 is intended to operate. For example, in some embodiments, the XR unit 204 is configured to be selectively connectable to the input unit 202 by wire. When connected by wire, the network interface 420 can enable communication with the input unit 202, and when not connected by wire, the network interface 420 can enable communication with the mobile communication device 206.

[0137] The input interface 430 shown in FIG. 4 is assumed to have the same functions as the input interface 330 described in detail above. In this case, the input interface 430 can communicate with an image sensor to obtain a gesture input 431 (e.g., the finger of user 100 pointing to a virtual object), communicate with another XR unit 204 to obtain a virtual input 432 (e.g., a virtual object shared with the XR unit 204, or a gesture of an avatar detected in a virtual environment), communicate with a microphone to obtain an audio input 433 (e.g., a voice command), and communicate with the input unit 202 to obtain a UI input 434 (e.g., virtual content determined by the virtual content determination module 315).

[0138] The power supply 440 shown in FIG. 4 is assumed to have the same functions as the power supply 340 described above and only supplies electrical energy to power the XR unit 204. In some embodiments, the power supply 440 can be charged by the power supply 340. For example, the power supply 440 may be wirelessly changed when the XR unit 204 is placed on or near the input unit 202.

[0139] The output interface 450 shown in FIG. 4 is assumed to have the same functions as the output interface 350 described in detail above. In this case, the output interface 450 can output from the optical indicator 451, the display 452, the speaker 453, and the projector 454. The projector 454 can be any device, apparatus, instrument, etc. that can project (or direct) light to display virtual content on a surface. This surface may be part of the XR unit 204, part of the user 100's eye, or part of an object proximate to the user 100. In one embodiment, the projector 454 can include an illumination unit that concentrates light within a limited solid angle by one or more mirrors and lenses and provides a high value of luminous intensity in a defined direction.

[0140] The processing device 460 shown in FIG. 4 is assumed to have the same functions as the output processing device 360 described in detail above. When the XR unit 204 is connected to the input unit 202, the processing device 460 can cooperate with the processing device 360. Specifically, the processing device 460 can implement virtual machine technology or other technologies to provide the ability to execute, control, operate, manipulate, store, etc. a plurality of software processes, applications, programs, etc. It should be understood that other types of processor configurations can be implemented to provide the capabilities disclosed herein.

[0141] The sensor interface 470 shown in FIG. 4 is assumed to have the same functions as the sensor interface 370 described in detail above. Specifically, the sensor interface 470 can communicate with the audio sensor 471, the image sensor 472, the motion sensor 473, the environmental sensor 474, and other sensors 475. The operations of the audio sensor, the image sensor, the motion sensor, the environmental sensor, and other sensors have been described above with reference to FIG. 3, and the details are not repeated here. It should be understood that other types and combinations of sensors can be used to provide the capabilities disclosed herein.

[0142] The components and arrangements shown in FIG. 4 are not intended to limit the disclosed embodiments. As will be understood by those skilled in the art having the benefit of this disclosure, numerous variations and / or modifications may be made to the illustrated configuration of the XR unit 204. For example, in all cases, not all components are necessarily essential for the operation of the XR unit 204. Any component may be arranged in any suitable part of the system 200, and the components may be rearranged in various configurations while providing the functions of the disclosed embodiments. For example, some XR units may not include all of the elements within the XR unit 204 (e.g., the wearable extended reality device 110 may not have the optical indicator 451).

[0143] FIG. 5 is a block diagram of an exemplary configuration of the remote processing unit 208. It should be understood that FIG. 5 is merely an exemplary representation of one embodiment, and within the scope of this disclosure, some of the illustrated elements may be omitted and other elements may be added. In the embodiment of FIG. 5, the remote processing unit 208 may include a server 210 that directly or indirectly accesses a bus 500 (or other communication mechanism) that interconnects subsystems and components for transferring information within the server 210. For example, the bus 500 can interconnect a memory interface 510, a network interface 520, a power supply 540, a processing device 560, and a database 580. The remote processing unit 208 may also include one or more data structures. For example, data structures 212A, 212B, and 212C.

[0144] The memory interface 510 shown in FIG. 5 is assumed to have the same functions as the memory interface 310 described in detail above. The memory interface 510 can be used to access software products and / or data stored in a non-transitory computer-readable medium or in other memory devices such as memory devices 311, 411, 511, or data structures 212A, 212B, and 212C. The memory device 511 may include software modules for executing processes consistent with the present disclosure. In particular, the memory device 511 may include a shared memory module 512, a node registration module 513, a load balancing module 514, one or more computing nodes 515, an internal communication module 516, an external communication module 517, and a database access module (not shown). Modules 512-517 may include software instructions for execution by at least one processor (e.g., processing device 560) associated with the remote processing unit 208. The shared memory module 512, the node registration module 513, the load balancing module 514, the computing module 515, and the external communication module 517 may cooperate to perform various operations.

[0145] The shared memory module 512 may enable information sharing between the remote processing unit 208 and other components of the system 200. In some embodiments, the shared memory module 512 may be configured to allow the processing device 560 (and other processing devices within the system 200) to access, retrieve, and store data. For example, using the shared memory module 512, the processing device 560 can perform at least one of executing a software program stored in the memory device 511, the database 580, or the data structures 212A-C, storing information in the memory device 511, the database 580, or the data structures 212A-C, or retrieving information from the memory device 511, the database 580, or the data structures 212A-C.

[0146] The node registration module 513 may be configured to track the availability of one or more computing nodes 515. In some examples, the node registration module 513 may be implemented as a software program such as a software program executed by one or more computing nodes 515, a hardware solution, or a combined software and hardware solution. In some implementations, the node registration module 513 can communicate with one or more computing nodes 515 using, for example, the internal communication module 516. In some examples, one or more computing nodes 515 can notify the node registration module 513 of the status of these nodes by, for example, sending a message at startup, shutdown, at regular intervals, at a selected time, in response to a query received from the node registration module 513, or at any other determined time. In some examples, the node registration module 513 can inquire about the status of one or more computing nodes 515 by, for example, sending a message at startup, at regular intervals, at a selected time, or at any other determined time.

[0147] The load balancing module 514 can be configured to divide the workload among one or more computing nodes 515. In some examples, the load balancing module 514 can be implemented as a software program, such as a software program executed by one or more of the computing nodes 515, a hardware solution, or a combined software and hardware solution. In some implementations, the load balancing module 514 can interact with the node registration module 513 to obtain information regarding the availability of one or more of the computing nodes 515. In some implementations, the load balancing module 514 can communicate with one or more of the computing nodes 515 using, for example, the internal communication module 516. In some examples, one or more of the computing nodes 515 can notify the load balancing module 514 of the status of these nodes by, for example, responding to a query received from the load balancing module 514 at startup, shutdown, at regular intervals, at a selected time, or at any other determined time, or by sending a message at any other determined time. In some examples, the load balancing module 514 can inquire about the status of one or more of the computing nodes 515 by, for example, sending a message at startup, at regular intervals, at a preselected time, or at any other determined time.

[0148] The internal communication module 516 may be configured to receive and / or transmit information from one or more components of the remote processing unit 208. For example, control signals and / or synchronization signals may be transmitted and / or received through the internal communication module 516. In one embodiment, input information for a computer program, output information of a computer program, and / or intermediate information of a computer program may be transmitted and / or received through the internal communication module 516. In another embodiment, the information received through the internal communication module 516 may be stored in the memory device 511, the database 580, the data structures 212A-C, or other memory devices within the system 200. For example, the information retrieved from the data structure 212A may be transmitted using the internal communication module 516. In another example, input data may be received using the internal communication module 516 and stored in the data structure 212B.

[0149] The external communication module 517 may be configured to receive and / or transmit information from one or more components of the system 200. For example, control signals may be transmitted and / or received through the external communication module 517. In one embodiment, the information received through the external communication module 517 may be stored in the memory device 511, the database 580, the data structures 212A-C, and / or any memory device within the system 200. In another embodiment, the information retrieved from any of the data structures 212A-C may be transmitted to the XR unit 204 using the external communication module 517. In another embodiment, input data may be transmitted and / or received using the external communication module 517. Examples of such input data may include data received from the input unit 202, information captured from the environment of the user 100 using one or more sensors (e.g., audio sensor 471, image sensor 472, motion sensor 473, environmental sensor 474, other sensor 475), and the like.

[0150] In some embodiments, the aspects of modules 512-517 may be implemented in hardware, software (including one or more signal processing and / or application specific integrated circuits), firmware, or any combination thereof, executable by one or more processors, either alone or in various combinations with each other. Specifically, modules 512-517 may be configured to interact with each other and / or with other modules of system 200 to perform functions consistent with the disclosed embodiments. Memory device 511 may include additional modules and instructions, or fewer modules and instructions.

[0151] The network interface 520, power supply 540, processing device 560, and database 580 shown in FIG. 5 are assumed to have functions similar to those of the similar elements described above with reference to FIGS. 4 and 5. The specific design and implementation of the above-described components may vary based on the implementation of system 200. Additionally, remote processing unit 208 may include more or fewer components. For example, remote processing unit 208 may include an input interface configured to receive input directly from one or more input devices.

[0152] In accordance with the present disclosure, a processing device of system 200 (e.g., a processor within mobile communication device 206, a processor within server 210, a processor within a wearable extended reality device such as wearable extended reality device 110, and / or a processor within an input device associated with wearable extended reality device 110 such as keyboard 104) can use a machine learning algorithm to implement any of the methods disclosed herein. In some embodiments, a machine learning algorithm (also referred to as a machine learning model in the present disclosure) can be trained using training examples, such as when described below. Some non-limiting examples of such machine learning algorithms include classification algorithms, data regression algorithms, image segmentation algorithms, visual detection algorithms (object detectors, face detectors, person detectors, motion detectors, edge detectors, etc.), visual recognition algorithms (face recognition, person recognition, object recognition, etc.), speech recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbor algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recurrent neural network algorithms, linear machine learning models, non-linear machine learning models, ensemble algorithms, and the like. For example, a trained machine learning algorithm can include an inference model such as a prediction model, a classification model, a data regression model, a clustering model, a segmentation model, an artificial neural network (deep neural network, convolutional neural network, recurrent neural network, etc.), a random forest, a support vector machine, and the like. In some examples, a training example can include an exemplary input along with a desired output corresponding to the exemplary input. Further, in some examples, training a machine learning algorithm using training examples can generate a trained machine learning algorithm, and the trained machine learning algorithm can be used to estimate an output for an input not included in the training examples.In some examples, the engineers, scientists, processes, and machines that train machine learning algorithms may further use validation examples and / or test examples. For example, the validation examples and / or test examples can include exemplary inputs along with the desired outputs corresponding to the exemplary inputs, and the trained machine learning algorithm and / or the intermediate-trained machine learning algorithm can be used to estimate the outputs for the exemplary inputs of the validation examples and / or test examples, the estimated outputs can be compared to the corresponding desired outputs, and the trained machine learning algorithm and / or the intermediate-trained machine learning algorithm can be evaluated based on the results of the comparison. In some examples, the machine learning algorithm can have parameters and hyperparameters, the hyperparameters can be set manually by a person or automatically by a process external to the machine learning algorithm (such as a hyperparameter search algorithm), and the parameters of the machine learning algorithm can be set by the machine learning algorithm based on the training examples. In some implementations, the hyperparameters may be set based on the training examples and the validation examples, and the parameters may be set based on the training examples and the selected hyperparameters. For example, given the hyperparameters, the parameters can be conditionally independent from the validation examples.

[0153] In some embodiments, a trained machine learning algorithm (also referred to herein as a machine learning model and a trained machine learning model) can be used to analyze an input and generate an output, for example, when described below. In some examples, a trained machine learning algorithm can be used as an inference model that generates an output inferred when an input is provided. For example, a trained machine learning algorithm can include a classification algorithm, the input can include a sample, and the inferred output can include a classification of the sample (inferred label, inferred tag, etc.). In another example, a trained machine learning algorithm can include a regression model, the input can include a sample, and the inferred output can include an inferred value corresponding to the sample. In yet another example, a trained machine learning algorithm can include a clustering model, the input can include a sample, and the inferred output can include an assignment of the sample to at least one cluster. In an additional example, a trained machine learning algorithm can include a classification algorithm, the input can include an image, and the inferred output can include a classification of the item shown in the image. In yet another example, a trained machine learning algorithm can include a regression model, the input can include an image, and the inferred output can include an inferred value corresponding to the item shown in the image (estimated properties of the item such as size, volume, age of a person shown in the image, distance from an item shown in the image, etc.). In an additional example, a trained machine learning algorithm can include an image segmentation model, the input can include an image, and the inferred output can include a segmentation of the image. In yet another example, a trained machine learning algorithm can include an object detector, the input can include an image, and the inferred output can include one or more detected objects in the image and / or one or more locations of an object in the image.In some examples, the trained machine learning algorithm can include one or more equations, and / or one or more functions, and / or one or more rules, and / or one or more procedures, the input can be used as an input to the equation and / or function, and / or rule, and / or procedure, and the inferred output can be based on the output of the equation and / or function, and / or rule, and / or procedure (e.g., selecting one of the outputs of the equation and / or function, and / or rule, and / or procedure, using a statistical measure of the output of the equation and / or function, and / or rule, and / or procedure, etc.).

[0154] In accordance with the present disclosure, the processing device of system 200 can analyze image data captured by an image sensor (e.g., image sensor 372, image sensor 472, or any other image sensor) to implement any of the methods disclosed herein. In some embodiments, analyzing the image data can include analyzing the image data to obtain pre-processed image data and then analyzing the image data and / or the pre-processed image data to obtain a desired result. Those skilled in the art will recognize that the following are examples and that image data can be pre-processed using other types of pre-processing methods. In some examples, the image data can be pre-processed by using a conversion function to convert the image data to obtain converted image data, and the pre-processed image data can include the converted image data. For example, the converted image data can include one or more convolutions of the image data. For example, the conversion function can include one or more image filters such as a low-pass filter, a high-pass filter, a band-pass filter, an all-pass filter, etc. In some examples, the conversion function can include a non-linear function. In some examples, the image data can be pre-processed by smoothing at least a portion of the image data, e.g., using a Gaussian convolution, using a median filter, etc. In some examples, the image data can be pre-processed to obtain a different representation of the image data. For example, the pre-processed image data can include a representation of at least a portion of the image data in the frequency domain, a discrete Fourier transform of at least a portion of the image data, a discrete wavelet transform of at least a portion of the image data, a time / frequency representation of at least a portion of the image data, a representation of at least a portion of the image data in a low dimension, an irreversible representation of at least a portion of the image data, a reversible representation of at least a portion of the image data, a time series of any of the above, any combination of the above, etc. In some examples, the image data can be pre-processed to extract edges, and the pre-processed image data can include information based on and / or related to the extracted edges. In some examples, the image data can be pre-processed to extract image features from the image data.Some non-limiting examples of such image features may include information based on and / or related to edges, corners, blobs, ridges, Scale-Invariant Feature Transform (SIFT) features, temporal features, etc. In some examples, analyzing the image data may include calculating at least one convolution of at least a portion of the image data and using the at least one calculated convolution to calculate at least one result value and / or to perform determinations, identifications, recognitions, classifications, etc.

[0155] In accordance with other aspects of the present disclosure, the processing device of system 200 can analyze image data to implement any of the methods disclosed herein. In some embodiments, analyzing the image can include analyzing the image data and / or pre-processed image data using one or more rules, functions, procedures, artificial neural networks, object detection algorithms, face detection algorithms, visual event detection algorithms, action detection algorithms, motion detection algorithms, background subtraction algorithms, inference models, etc. Some non-limiting examples of such inference models can include manually pre-programmed inference models, classification models, regression models, results of training algorithms such as machine learning algorithms and / or deep learning algorithms for training examples, where the training examples can include examples of data instances, and in some cases, the data instances can be labeled with corresponding desired labels and / or results. In some embodiments, analyzing the image data (e.g., by the methods, steps, and modules described herein) can include analyzing pixels, voxels, point clouds, range data, etc. included in the image data.

[0156] Convolution can include convolution in any dimension. One-dimensional convolution is a function that transforms an original sequence of numbers into a transformed sequence of numbers. One-dimensional convolution can be defined by a column of scalars. Each specific value in the transformed sequence of numbers can be determined by calculating a linear combination of the values of a subsequence of the original sequence corresponding to the specific value. The resulting value of the calculated convolution can include any value in the transformed sequence of numbers. Similarly, n-dimensional convolution is a function that transforms an original n-dimensional array into a transformed array. n-dimensional convolution can be defined by an n-dimensional array of scalars (known as the kernel of the n-dimensional convolution). Each specific value in the transformed array can be determined by calculating a linear combination of the values within the n-dimensional region of the original array corresponding to the specific value. The resulting value of the calculated convolution can include any value within the transformed array. In some examples, an image can include one or more components (such as color components, depth components, etc.), and each component can include a two-dimensional array of pixel values. In one example, calculating the convolution of an image can include calculating a two-dimensional convolution for one or more components of the image. In another example, calculating the convolution of an image can include stacking arrays from various components to create a three-dimensional array and calculating a three-dimensional convolution on the resulting three-dimensional array. In some examples, a video can include one or more components (such as color components, depth components, etc.), and each component can include a three-dimensional array of pixel values (having two spatial axes and one temporal axis). In one example, calculating the convolution of a video can include calculating a three-dimensional convolution for one or more components of the video. In another example, calculating the convolution of a video can include stacking arrays from various components to create a four-dimensional array and calculating a four-dimensional convolution on the resulting four-dimensional array.

[0157] Some of the disclosed embodiments may include a system, method, and non-transitory computer-readable medium configured to enable content sharing among users of wearable extended reality devices. Content sharing may include the presentation of text, images, programs, or any other information by one entity to another entity. The content may be shared among users of wearable extended reality devices such that an entity wearing the device can view or otherwise access the content available via another wearable extended reality device. Content sharing may be enabled in many different ways. In some embodiments, the enabling may occur through a link between two extended reality devices. In one example, the link may include a direct and / or indirect communication link between two extended reality devices. In another example, a system (such as a centralized system) may communicate with each of two extended reality devices to, for example, provide content for presentation to each of the two extended reality devices and receive location and / or orientation information associated with the extended reality devices, and / or any other information regarding presenting the content. The link may include a linkage within a data structure or database of the system or maintained by the system. In one non-limiting example, the detected proximity of two entities may trigger the automatic or selectable sharing of content. In other embodiments, one entity may select content for sharing with either a group or an individual, or may associate the content with a physical or virtual location and enable anyone accessing that location to view the content. In some embodiments, content sharing may be enabled by permission, and viewing of shared content may be enabled only for entities having appropriate permissions. In yet other embodiments, rules may be defined as to who can share content. In further exemplary embodiments, sharing may be enabled via a request sent from one device to another device.Content sharing may, in some embodiments, be initiated as a result of signals from one or more sensors. For example, one or more image sensors may detect a gesture indicating an intention to share content. Alternatively, a proximity sensor may trigger the ability to share when two wearable extended reality devices are in proximity to each other. There are many different ways in which content sharing may be enabled consistent with the present disclosure, and the present disclosure is not limited to any particular one.

[0158] Some of the disclosed embodiments may include a computer-readable medium that, when executed by at least one processor, causes the at least one processor to establish a link between a first wearable extended reality device and a second wearable extended reality device. In some embodiments, establishing a link can refer to any means of communicating between the first wearable extended reality device and the second extended wearable reality device. In some embodiments, there may be a formal protocol that defines how the first wearable extended reality device and the second wearable extended reality device transmit and receive data. Such protocols can include, but are not limited to, Transmission Control Protocol / Internet Protocol (TCP / IP), Bluetooth, infrared, near field communication, ultra-wideband, WiFi, Zig-Bee, short-range communication protocols, and / or long-range communication protocols. Such communication means between the first wearable extended reality device and the second wearable extended reality device can include, but are not limited to, a local area network (LAN), a wireless local area network (WLAN), a virtual private network (VPN), an indirect communication link, and / or a directed communication link. The processor can cause the first wearable extended reality device to initiate communication with the second wearable extended reality device via one or more of the protocols identified above. In some examples, establishing a link can refer to a linkage in a data structure and / or database in, for example, a system that adjusts an extended reality environment, a system that communicates with both the first and second wearable extended reality devices, and / or a system that provides content for presentation to both the first and second wearable extended reality devices.

[0159] Some of the disclosed embodiments may include establishing a link (e.g., a communication link, a data structure, and / or a link in a database) and enabling information exchange when a first wearable extended reality device is detected in proximity to a second wearable extended reality device. Such instructions may be configured based on a threshold distance, which may vary among users of the extended reality devices. For example, a user of a first wearable extended reality device may prefer to initiate a link with a second wearable extended reality device at a shorter threshold distance, such as 1 meter or 2 meters, when a particular type of user, such as a family member, is nearby. In contrast, the same user of the first wearable extended reality device may prefer to initiate a link with a second wearable extended reality device at a longer threshold distance to facilitate communication with a colleague or team member. The user may freely update the threshold distance setting according to the user's preference.

[0160] In some examples, the communication link between wearable extended reality devices may be via an intermediate device. The intermediate device may include any networking device disposed between a remote access server (RAS) and an RAS client. The intermediate device can assist in providing connectivity between two separate wearable extended reality devices. The intermediate device can provide an additional layer of security before a communication link between two wearable extended reality devices is established. For example, if a security protocol is not met, the intermediate device can prevent the establishment of a connection between the two extended reality devices.

[0161] Some of the disclosed embodiments may include presenting first virtual content through a first wearable extended reality device. The virtual content can take a variety of different forms and can include documents, photos, videos, virtual characters, and any other sharable media that can be transmitted wirelessly. In some examples, the virtual content may be presented as part of an extended reality environment. In other examples, the virtual content may be displayed on a physical screen such as a television, tablet, laptop, or smartphone, or may be displayed on a virtual screen through one or more wearable extended reality devices or the like.

[0162] Some of the disclosed embodiments may include obtaining a first command for displaying the first virtual content via a second wearable extended reality device. The first command may refer to a signal received from at least one processor for displaying the virtual content via the second wearable extended reality device. In another example, the first command may refer to a command obtained from the user and / or in response to the user's action. The non-transitory computer-readable medium may be configured to share the content and may include instructions for at least one processor to transmit a signal to the first wearable extended reality device for displaying the virtual content.

[0163] In some of the disclosed embodiments, obtaining a first command to present virtual content may include an intent to share in the data captured by the first wearable extended reality device. The intent to share may refer to the user desiring to present the virtual content to another user or exchange the virtual content with another user. In one example, a machine learning model may be trained using training examples to identify the intent to share by analyzing the captured data. Examples of such training examples may include sample capture data along with a label indicating whether the sample capture data corresponds to an intent to share and / or parameters of the intent to share (e.g., an intent to share specific content, an intent to share content presented in a specific area, an intent to share content with a specific entity, and / or an intent to share content in a specific manner). The trained machine learning model may be used to analyze the data captured by the first wearable extended reality device to identify the intent to share and / or parameters of the intent to share. For example, the user's intent to share may be identified in the captured image data. Some non-limiting examples of such captured image data may include image data (such as one or more images and / or one or more videos) captured using an image sensor included in the first wearable extended reality device, an image sensor included in a second wearable extended reality device, or an image sensor included in a device within the environment of the first and second wearable extended reality devices (the device may be different from the first and second wearable extended reality devices, such as another wearable extended reality device, a fixed camera mounted in a room, and / or any other device capable of capturing image or video data). The user's intent to share may be identified by analyzing the image data to detect movement or a change in position or orientation of an object within the captured image data.For example, the user's sharing intention can be identified by using a visual gesture recognition algorithm to analyze image data, or by detecting the movement of the user's hand (e.g., the user is waving their hand). In another example, the user's sharing intention can be identified by using, for example, a visual object tracking algorithm to detect that the user has moved hardware within one or more captured images. Additionally, such a sharing intention can be captured when the user moves their finger, nods, touches an extended reality device, or gestures to another user who desires to share content in some other way. In some examples, a machine learning model can be trained using training examples to identify a sharing intention by analyzing images and / or videos. Examples of such training examples can include sample images and / or sample videos along with labels indicating whether the sample images and / or sample videos correspond to a sharing intention and / or parameters of a sharing intention (e.g., an intention to share specific content, an intention to share content presented in a specific area, an intention to share content with a specific entity, an intention to share content in a specific way, and / or an intention to share content at a specific time). The trained machine learning model can be used to analyze the captured image data to identify a sharing intention and / or parameters of a sharing intention. In some examples, at least a portion of the image data can be convolved to obtain a computed convolution result value. In one example, in response to a first computed convolution result value, a sharing intention for a first virtual content may be identified, and in response to a second computed convolution result value, a sharing intention for the first virtual content may not be identified. In another example, in response to a first computed convolution result value, a sharing intention for one virtual content may be identified, and in response to a second computed convolution result value, a sharing intention for another virtual content may be identified.

[0164] As an example, FIG. 6 shows the sharing intention of a first extended reality device user 610 via a sharing operation 612. For example, as shown in FIG. 6, the gesture by the first user 610 reflects the intention to share virtual content 614 with a second extended reality device user 616.

[0165] Hardware that can be used to capture a user's sharing intention may include a pointer, keyboard, mouse, joystick, or any other object designed to facilitate sharing data. Such a sharing intention can be captured when the user moves hardware such as a mouse or joystick towards another user. In some embodiments, the user's sharing intention can also be determined when the user waves a wand, presses a button on a keyboard, or gestures towards another user in other ways. In some embodiments, the identification of the sharing intention based on data captured from the hardware may be configured based on the user's preference and may not be limited to moving the hardware towards a second user.

[0166] The user's preference refers to how a particular wearable extended reality device user chooses to configure their device to share virtual content. For example, a user may prefer to share virtual content via hand movements, while another user may prefer to share content via a pointer or joystick. Alternatively, the user may be allowed to define gesture preferences for sharing content.

[0167] In some embodiments, the sharing intent may also be identifiable in the captured audio data. For example, the captured audio data may include audio data captured using an audio sensor included in a first wearable extended reality device, using an audio sensor included in a second wearable extended reality device, and / or using an audio sensor included in another device. In one example, a user can verbally initiate sharing via an audio command. The command may include, for example, "Share content", "Transmit content", "Display content", "Present content", or any other combination of words that can be interpreted as a request to share content with another user, but is not limited thereto. Additionally, the verbal command may specifically name a second user, for example, in the form of "Share content with User X". The verbal command may also be configured based on the user's preferences. In one example, the captured audio data can be analyzed using an audio recognition algorithm to identify an audio command corresponding to the sharing intent and / or parameters of the sharing intent.

[0168] As described above, the preferences of each individual wearable extended reality device user may be stored in a non-transitory computer-readable medium capable of controlling content sharing.

[0169] In some embodiments, the shared intent may be identified in the captured positioning data. Such positioning data may be based on the position and / or orientation of the first wearable extended reality device user, based on the position and / or orientation of the first wearable extended reality device, and / or based on the position and / or orientation of a device associated with the first wearable extended reality device (e.g., another device used by the user of the first wearable extended reality device such as a keyboard, glove, and / or pointing device). For example, the shared intent may be identified when the user positions their body and the wearable extended reality device close to another user's body. As another example, the shared intent may be identified when the user stands up from a sitting position, leans towards another user, or positions their body or extended reality device in another way to indicate the shared intent. The shared intent based on the positioning data may also be configured based on the user's preferences. In one example, the position and / or orientation may be determined by analyzing data captured using sensors included in the first wearable extended reality device, and / or sensors included in different devices such as positioning sensors (e.g., accelerometers, gyroscopes, and / or GPS sensors) and / or image sensors (e.g., by analyzing captured image data using an ego-motion algorithm and / or a visual object tracking algorithm using an image sensor).

[0170] In some embodiments, obtaining the first command may include identifying a shared intent associated with an action performed in an extended reality environment (such as a virtual reality environment, an augmented reality environment, and / or a mixed reality environment). As described in the following paragraphs, actions performed in an extended reality environment may include sharing content via a shared bubble or a shared ring, or by automatically sharing content with another wearable extended reality device user when a link is established between two or more wearable extended reality devices.

[0171] In some embodiments, identifying a shared intent may include determining that a first user of a first wearable extended reality device has changed the orientation of a virtual screen presenting first virtual content toward a second user of a second wearable extended reality device. In one example, determining that a first user of a first wearable extended reality device has changed the orientation of the virtual screen toward a second user may include determining that the virtual screen is oriented toward the second user and determining that the virtual screen is oriented toward the second user as a result of an action by the first user (and not, for example, as a result of movement of the second user). In one example, determining whether the virtual screen is oriented toward the second user as a result of an action by the first user may be based on an analysis of movement of the second user and / or movement of the virtual screen and / or input leading to movement of the virtual screen. For example, a system that adjusts the extended reality environment, a system that communicates with both the first and second wearable extended reality devices, a system that provides content for presentation to both the first and second wearable extended reality devices, or at least one of the first wearable extended reality devices may be able to determine that the orientation of the virtual screen has changed toward the second user. Based on the preferences of the first wearable extended reality device user, this may indicate an intent to share with the second user. In some examples, the first angle may be the angle between the surface corresponding to the virtual screen and the direction associated with the user of the first wearable extended reality device, the second angle may be the angle between the surface corresponding to the virtual screen and the direction associated with the user of the second wearable extended reality device, and the determination that the orientation of the virtual screen presenting the first virtual content has changed toward the second user of the second wearable extended reality may be based on the first angle and the second angle.Some non-limiting examples of such directions associated with a user may include the direction of the user's head, the direction of the user's body, the user's gaze and / or eye direction, the user's fixation direction, the user's line of sight direction, the user's visual axis direction, the user's pupil axis direction, the user's optical axis direction, and / or the direction associated with the orientation of the user's wearable extended reality device. In one example, a function of a first angle and a second angle can be calculated and compared to a threshold to determine whether the orientation of a virtual screen presenting first virtual content has changed towards a second user. In another example, a machine learning model can be trained using training examples to determine whether the orientation of a virtual screen is changed towards a user based on an angle and / or a distance. Examples of such training examples can include a sample angle of a sample virtual screen with respect to a sample user, and / or a sample distance of the sample virtual screen from the sample user, along with a label indicating whether the sample virtual screen is oriented towards each of the sample users. The trained machine learning model can be used to analyze a first angle, a second angle, and optionally the distance of the user from the virtual screen to determine that the virtual screen is oriented towards the second user.

[0172] Furthermore, the user of the second wearable extended reality device can be located within the proximity of the virtual screen, whether it is a physical threshold distance or a virtual proximity. In one example, a shared intent can be determined when the user of the first wearable extended reality device leans or nods towards the user of the second wearable extended reality device, such that a change in orientation can be captured. The user of the second wearable extended reality device can also be a representation of the user, such as an avatar. Thus, even if the second user is not physically located next to the first user, the first user can share content based on the change in orientation.

[0173] In some embodiments, identifying a shared intent may include determining that a first user of a first wearable extended reality device has moved a first virtual content towards a virtual shared space surrounding a second user of a second wearable extended reality device. In one example, the virtual shared space surrounding the second user may be referred to as a shared bubble. For example, at least one processor may be configured to detect an action that may take the form of a gesture, a drag of the first virtual content, or other physical movement made by the first user. At least one processor may be configured to determine the movement of the first virtual content towards the virtual shared space based on the detected gesture, drag, or other physical movement. This sharing action may also be based on the natural movement of people leaning towards others who desire to share content. Such movement may be detected by one or more sensors within at least one extended reality device or via sensors associated with another device.

[0174] The virtual shared space in the above embodiments known as the shared bubble can be automatically determined in countless different ways. For example, the virtual shared space may be determined based on the position and / or movement of the second user and / or the second wearable extended reality device. For example, the virtual shared space may include the space within a selected distance from the second user and / or the second wearable extended reality device. In this embodiment, when the first user is within a specific distance of the second user, the virtual content may be automatically shared with the second user. In another example, the shared bubble may be elongated in the direction of movement. In yet another example, the shared bubble may be larger in the horizontal direction than in the vertical direction.

[0175] The shared bubble may be the space around the user. This space may be defined in various different ways. In some embodiments, the size and / or orientation of the bubble surrounding one or more users may depend on the number of users, the positions of the users, the positions of other people not belonging to the bubble, the orientation of the users, the orientation of other people not belonging to the bubble, the identity of the users, the identity of users not belonging to the bubble, the positions of non-animated objects in the vicinity of the users, the types of non-animated objects in the vicinity of the users, the status of non-animated objects in the vicinity of the users, the movement patterns of the users, the movement patterns of other people not belonging to the bubble, the preferences of the users, the past behavior of the users, the gestures of the users, and / or other factors related to the proximity of the users to each other. In one example, one or more phase correlation functions of these parameters may be able to define the space of the shared bubble.

[0176] When the virtual shared bubbles of two or more users collide, the two or more users may be automatically provided with an opportunity to create a new shared bubble in which virtual content items can be shared between the extended realities of the two or more users. At least one processor may be configured to determine that there is an overlap between the virtual shared bubbles, and in response, at least one processor may suggest to the users of the overlapping bubbles that virtual content can be shared between these users. The wearable extended reality devices of the users may also be configured to automatically share content when they overlap with the bubble of another user.

[0177] As an example, FIG. 7A shows a pair of users 714 and 716. As shown in FIG. 7A, the first wearable extended reality device user 714 can have an associated virtual shared bubble 710, and the second wearable extended reality device user 716 can have an associated virtual shared bubble 718. The virtual shared bubble 710 of the first wearable extended reality device user 714 can contact and overlap with the virtual shared bubble 718 of the second wearable extended reality device user 716, for example, when the first wearable extended reality device user 714 moves towards the second user 716. FIG. 7B shows the condition where the virtual shared bubble 710 overlaps with the virtual shared bubble 718 via the overlapping region 720. The virtual content 712 can be automatically shared with the second wearable extended reality device user 716 when the shared bubbles 710 and 718 overlap, for example, in the overlapping region 720, and / or when the volume of the overlapping region 720 exceeds a selected volume threshold. In one example, the volume threshold can be selected as a function of the volume of the virtual shared bubble 710 and / or the volume of the virtual shared bubble 718, the volumes of the users 714, 716, and / or the volume of the virtual content 712, by configuration, by the user.

[0178] As an example, FIGS. 8A and 8B show the combined virtual shared bubble 810 after the shared bubbles from the first extended reality device user 812 and the second extended reality device user 816 overlap, as shown in FIG. 8A. The virtual content 814 may be automatically shared when these virtual shared bubbles overlap, as shown in FIG. 8B, but the virtual content 814 may also be automatically shared when the first user 812 leans towards the second user 816 (for example, when the first user 812 leans towards the second user 816 when the virtual shared bubbles overlap). The combined shared bubble 810 can shrink in size as the users approach each other.

[0179] The virtual shared space can also be determined based on the usage mode of the second wearable extended reality device, the user, the second wearable extended reality device, and / or the objects surrounding the second wearable extended reality device. For example, the extended reality device can be configured to automatically share content when the user enters a room, passes by the desk of a second user, or walks within the vicinity of any other object in the room. Additionally, the usage mode may be configured to automatically share content with some users and not with others.

[0180] The usage mode may refer to various settings for the user to configure their extended reality system (e.g., their wearable extended reality device) based on the various tasks the user desires to accomplish. For example, one usage mode may only apply to activities performed at work. In this example, certain types of content, including documents and emails, can be shared between two wearable extended reality device users at a certain distance. In another example, the usage mode may only apply to activities performed at home. The user may, for example, desire to share videos but not documents or emails.

[0181] In some embodiments, identifying a shared intent may also include determining that a first user of a first wearable extended reality device has moved first virtual content toward a defined shared space. For example, in addition to, or instead of, automatically sharing content when two extended reality device users move toward each other, a user may create a shared space based on predefined parameters such as length, width, height, and / or position within a room. Instead of sharing content when the virtual shared bubbles of two users overlap, the shared intent may be determined when a user moves content toward and / or into such a predefined space. Encompassing each user and in contrast to the shared bubble described above, this shared intent may be referred to as a shared ring since its characteristics are predefined by the extended reality device users. The shared ring may be configured based on user preferences. In particular, the location and / or size of the shared ring may be configured by each user of the wearable extended reality device.

[0182] Some non-limiting examples of such virtual shared spaces can include a ring, rectangle, cube, sphere, convex shape, non-convex shape, smooth shape, irregular shape, or any other well-defined shape. In some examples, the virtual shared space may be fixed to a physical location, fixed to a physical object, created by a user, or created based on any other predefined characteristic. The virtual shared space (such as a shared ring) may have any two-dimensional or three-dimensional shape and is not limited to a ring. In one example, the virtual shared space (such as a shared ring) can be configured to be fixed to an object and move with the object. For example, the virtual shared space may be fixed to a keyboard, input device, output device, chair, or desk associated with a particular user, and when virtual content is moved into that virtual shared space, the virtual content may be shared with a particular user. In the example of a desk, the virtual shared space may correspond to at least a portion of the area on the top surface of the desk. In the example of a chair, the virtual shared space may correspond to a three-dimensional shape that includes at least a portion of the chair. In the example of a keyboard, the keyboard may be placed on a desk, and the virtual shared space may correspond to at least a portion of the area on the top surface of the desk, for example, a portion of the area on the top surface of the desk near the keyboard. Sharing the content may also be based on predefined privacy settings that can be configured by the extended reality device user. In some examples, the content may be shared between two users, shared publicly, shared among multiple users, or may include any other combination of public and private sharing. The type and level of sharing, and / or the identity of the users who can access the virtual content, can be determined based on the virtual shared space, the content, the users, or any other factor based on the location where the users are located. In the shared space, the movement of the virtual content can be the result of the actions of the user of the first wearable extended reality device. Such actions may take the form of, for example, a gesture, dragging of the first virtual content, or any other physical movement or movement involving the first virtual content.

[0183] As an example, FIG. 9 shows a shared ring 914, which can be a pre-defined virtual shared space created by a first wearable extended reality device user 910, a virtual shared space created by a user of another wearable extended reality device (such as one of the users 918 or a user of a different wearable extended reality device), a virtual shared space created by a user not using a wearable extended reality device, and / or an automatically generated virtual shared space. The first user 910 can place the virtual content 912 in a pre-defined area 914 such that the virtual content 912, which was previously visible only to the first wearable extended reality device user 910, can become shared content 916 visible to other extended reality device users 918.

[0184] In some examples, virtual content items can be shared based on people's natural movements and / or based on the user moving the virtual display screen and positioning this screen in a visible position to other users or participants. For example, hand movements or gestures may be recognized, and in response, virtual content items (such as virtual display screens) may be shared with relevant users. In some embodiments, all participants can view a virtual display screen docked at the same location. In some embodiments, virtual content items are simultaneously viewed by multiple participants in a public area and, in private mode, can be viewed as an additional / copied virtual display screen of a particular user docked at a location selected by the user. For example, a user may prefer to view a virtual content item from a closer distance, particularly if the virtual content item includes text, and may prefer to add a private overlay or any other configuration to the virtual content item. In one example, virtual content items with a lower level of privacy can be shared without user approval. In one example, public content items may be viewed in shared mode and private mode, and shared content may be viewed in private mode.

[0185] In some embodiments, for example, the sharing of virtual content can be triggered when the sharing bubbles of a first user and a second user overlap, or when the user shares the content within a pre-defined area, i.e., a sharing ring, or by any other means described herein. However, before virtual content can be shared between a first wearable extended reality device and a second wearable extended reality device, the non-transitory computer-readable medium can establish a link between the two devices. In some embodiments, virtual content can be automatically shared via a virtual shared space when a link is established between a first wearable extended reality device and a second extended reality device. The processor can automatically generate the sharing of virtual content when the first and second wearable extended reality devices are linked to communicate with each other. The shared virtual content may be selected based on the preferences of the users of the first and / or second extended reality devices. Whether a sharing is created immediately after establishing the link may be based on the user's preference, and the extended reality device may be configured for various uses.

[0186] For example, a user can configure a wearable extended reality device or system to automatically share content at work. In this example, the device may be configured to share content at a longer distance, such as when a colleague is within sight, and the content may be limited to documents and emails. In another example, a user can configure a wearable extended reality device or system to automatically share content at home. In this example, the device may be configured to automatically share content only at a shorter distance, such as when a family member is right next to the user. Such virtual content may be limited to videos and music and may not include documents or emails.

[0187] In some embodiments, the visualization of the virtual shared space can be presented by each of the first wearable extended reality device and the second wearable extended reality device. For example, on a typical computer screen, documents can be shared by dragging and dropping files between various folders such as Dropbox folders. In contrast, in some embodiments of the present disclosure, sharing virtual objects need not be limited to folders. Rather, virtual objects can be shared via virtual boxes, virtual windows, or otherwise marked areas. A virtual box may refer to a space on a virtual display presented by a first or second wearable extended reality device where multiple users can share virtual content. This sharing act may also occur in the virtual space of a three-dimensional space, and the act of sharing virtual objects between such folders, virtual boxes, or windows may be visible to users of both the first and second wearable extended reality devices. In some examples, the visualization of the virtual shared space may include a visual indication of the boundary of the virtual shared space, a visual indication of the volume of the virtual shared space, and / or a visual indication of the center of the virtual shared space. Some non-limiting examples of such visual indicators may include virtual coloring of a surface or volume, virtual highlighting of a surface or volume, and / or one or more virtual visual symbols at a selected location. When a user moves virtual content towards and / or into the visualized virtual shared space, sharing of the moved virtual content can be triggered. For example, a user can move virtual content using gestures, using a computer pointing device, using text input, and / or using voice commands.

[0188] In some embodiments, the location of the virtual shared space can be determined based on the location of physical objects within the environment of the first wearable extended reality device. For example, certain physical objects such as a whiteboard or other display surface may be authorized and linked to the virtual shared space so that anyone in the vicinity of the object can access the shared content. In some embodiments, the extended reality device user can select an object for defining the virtual shared space. For example, an artwork on a wall or a physical object on a table may be selected as a basis for the virtual shared space, and upon detection of the selected object, a virtual display screen may appear in an area adjacent to the selected object.

[0189] As an example, FIG. 10A shows a shared bubble where a second wearable extended reality device 1012 receives first virtual content 1014 from a second wearable extended reality device 1010 via a shared area 1016. Such virtual content can also be shared via a shared ring as shown in FIG. 9.

[0190] In another example, FIG. 10B shows presenting the received virtual content. Here, the second wearable extended reality device 1012 presents the first virtual content 1014 received from the first wearable extended reality device 1010. The virtual content can be presented via a wearable extended reality device on a screen such as a tablet or a TV, or via the device itself or any other physical object, as represented by the square in FIG. 10B.

[0191] In some embodiments, an indicator can be provided via a first wearable extended reality device when a first virtual content is presented via a second wearable extended reality device. The indicator can refer to any type of mark, annotation, or signal that notifies the user that the user's content is being displayed on another extended reality device. Additionally, the indicator can indicate whether the content is shared privately, publicly, with a specific individual (and / or the identity of a specific individual), with a specific group (and / or the identity of a specific group), and other public or private sharing options. As another example, the indicator can notify whether a user of the second wearable extended reality device has included an annotation or other comment in the first virtual content, or has edited or transformed the first virtual content. In one example, the indicator provided via the first wearable extended reality device can be an audible indicator. In another example, the indicator provided via the first wearable extended reality device can be a visual indicator. For example, the visual indicator can be present on or with the first virtual content. In another example, the visual indicator can include a visual modification to the presentation of the first virtual content.

[0192] There are many ways in which virtual content can be shared among users of wearable extended reality devices. The content can be shared via a shared bubble or a shared ring. A first user of a wearable extended reality device can share content with a second user of a wearable extended reality device via the shared bubble or the shared ring. In the above embodiment, the second user of the wearable extended reality device can display the content received from the first user. This content can be displayed in countless ways, whether via the second user's extended reality device, via a screen such as a tablet or a television, via a flat surface such as a wall, a window, or a chalkboard, and / or via any other physical object that the user may specify to present the virtual content. For example, after receiving virtual content from the first user, the second wearable extended reality device may display the content via the second device and / or via a television or a whiteboard.

[0193] In some embodiments, the first virtual content may be associated with at least one private virtual object and at least one public virtual object, and causing the virtual content to be transmitted for display by a second wearable extended reality device may include transmitting at least one public virtual object and avoiding transmission of at least one private virtual object. As described herein, a virtual object may refer to a virtual display screen, a virtual content item, a virtual two-dimensional object, a virtual three-dimensional object, a document, a media item, a photograph, a video, a virtual character, user-generated content, and / or a component of an enumerated object (e.g., a volume or brightness adjustment bar). Further, a public virtual object may include a virtual object that can be shared for viewing by other wearable extended reality device users and that may not be subject to any restrictions on sharing with other users. In contrast, a private virtual object may include virtual content that a wearable extended reality device user does not desire to share and / or is not permitted to share with any other user or any other user who is not part of a predefined group. In some examples, a user may be able to configure which virtual objects are public and which are private, or configure various privacy levels for various virtual objects. In other examples, for instance, based on an analysis of the visual and / or non-visual characteristics of a virtual object, it may be automatically determined whether the virtual object is private or public.

[0194] In the above embodiment, the user can specify that some or all parts of the virtual content cannot be shared with another user. For example, when the virtual content is visualized as a screen, the screen can have both a public window and a private window. The public window and the private window can each contain virtual objects. Thus, in some disclosed embodiments, the virtual content can be displayed such that only the public virtual objects are visible to the second wearable extended reality device user and the private virtual content remains hidden. The virtual content intended to be shared among wearable extended reality device users can consist of various virtual objects, some of which are private and some of which are public.

[0195] As an example, the virtual content may include a table or graph showing financial information associated with the user, and the user may not wish to share the financial information with other users. The user can specify the financial information within the virtual content as a private virtual object, thus preventing other users from viewing the virtual content. In another example, the virtual content can include a document with confidential exhibits. The user can specify the exhibits as private virtual objects, thus preventing other users from viewing the virtual objects.

[0196] In some embodiments, virtual content items can be shared at the same level of privacy without requiring user approval. The level of privacy may refer to which virtual content items can or cannot be shared with which specific wearable extended reality device users. The level of privacy may be established by dividing the virtual content of other users into one or more groups, and each group is enabled to visualize part or all of the virtual content. The number of privacy levels can be configured based on user preferences. For example, a part of the virtual content may be designated as content that can be shared with all other users. This may correspond to a low level of privacy. In another example, a portion of the virtual content may be designated as content that can be shared with users related to the users sharing the virtual content. This relationship may be based on family or professional relationships. For example, a user may designate part or all of the virtual content as content that can only be shared with immediate family members (e.g., spouse, child, or parent) or users within a specific professional group (e.g., an office team or department). This may correspond to a medium level of privacy. As another example, a user may designate part or all of the virtual content as content that can only be viewed by that user and not by other users. This may correspond to a high level of privacy. Although only three privacy levels (e.g., low, medium, high) are described herein, the present disclosure is not so limited, and it is contemplated that any number of privacy levels can be established consistent with the embodiments of the present disclosure.

[0197] In some embodiments, the virtual content item may require user approval before being shared at a privacy level lower than the current privacy level of the visual content item. In some examples, the determination of whether a virtual display screen needs to be shared may be based on the virtual size of the virtual display screen, the orientation of the virtual display screen, the location of the virtual display screen, and factors related to the virtual screen configuration.

[0198] In some embodiments, various sharing levels may be applied. Some non-limiting examples of such sharing levels include private (available only to the user), shared (available to those the user has chosen to share with), public (available to anyone within a particular physical space), or any other combination of public, private, or shared. In one example, there may be an owner of a particular physical space, and only the owner of the particular physical space can share the public content within that particular physical space content. For example, a company can share public content within the company's office, or a user can share public content within the user's home.

[0199] In some examples, private content items, including but not limited to virtual display screens and widgets, can be shared with other users in a way that does not enable the identification of some types of items. For example, items such as text, images, fine details, or any other media that a user may not wish to share may not be identified or may not be visible to other users. As an example, some or all of the virtual content may appear blurred to some users but clearly visible to other users, based on how the first wearable extended reality device user configures that device or the privacy level. Other ways of obscuring the virtual content may be used. In one example, when the first user approaches the second user, the first user can see the second user's private content as if it were being viewed through frosted glass, but cannot see the details of the content. Thus, the second user's content can be protected from the first user, who may not have the right to view the private document.

[0200] In some embodiments, the user may be communicated that certain content is about to be shared with other users, and the user may be provided with time to cancel the sharing before the specific content is shared. Additionally, the user can obtain information related to the user's objects being shared with other users, information related to the objects of other users being shared with the user, information related to these other users, and / or information related to the sharing levels of various users.

[0201] In some embodiments, a user can use gestures to select virtual content items, such as a shared virtual display screen, document, or other media, and send it to a physical screen. Such a physical screen can include, but is not limited to, for example, a TV, tablet, laptop, smartphone, smart board, board, a physical screen designated for content sharing, or any other physical screen. Gestures used by the user to share virtual content can include, for example, drag, pinch, swipe, or any other hand movement. As a result, the virtual content item may be shared, presented on the physical screen, or operated in any other way as desired by the user. In some examples, the physical screen may provide and / or guide a virtual shared space, for example, with the screen boundary as the shared area, and the virtual content item may be presented as a virtual layer and viewed by an extended reality device.

[0202] As an example, FIGS. 11A and 11B show sharing of public virtual content 1110. This content can be shared, for example, with a physical screen 1114, here a TV, as shown in FIG. 11B, by using gesture 1112 as shown in FIG. 11A. Such a gesture can be a wave of the hand, a point of the finger, or any body movement. In this embodiment, the virtual content shared on the physical screen may be public virtual content that can be viewed by all people, including those not wearing an extended reality device.

[0203] In some examples, a user can move their virtual content items, such as a virtual display screen or a document, to any of their personal physical screens. Such personal screens can include personal physical display screens, smartphone screens, or tablet screens. The user can move their content by using gestures. In response, the virtual content item may move to the physical display screen (i.e., may be presented using the physical display screen), or, for example, the virtual content item may be presented virtually on the physical display screen while remaining at the same privacy level without additional approval.

[0204] However, when sharing some content, additional approval may be required. For example, a user may wish to share private virtual content that has a higher privacy level, i.e., is confidential, with another user. Such virtual content may be password protected, accessible only to certain users, or otherwise made private. In this scenario, the sharing user may be prompted to enter a password, contact a system administrator, or otherwise request permission to share the virtual content.

[0205] As an example, FIGS. 12A and 12B illustrate sharing private virtual content 1210. Here, an extended reality device user can share their virtual content 1210 with their physical screen 1212 by using gesture 1214, as shown in FIG. 12B. This content remains at the same privacy level, i.e., no additional approval is required to share the private virtual content. In some examples, the user can initiate sharing of virtual content with the physical screen. Identification of one or more users exposed to the physical screen can be performed (e.g., based on location information associated with the users and / or based on analysis of an image of the room including the physical screen). Based on the identified one or more users, it can be determined whether the virtual content can be shared with the physical screen without additional approval, whether the virtual content can be shared with the physical screen upon additional approval, and / or whether the virtual content cannot be shared with the physical screen. In one example, after the virtual content is shared with the physical screen, (e.g., based on location information associated with additional users and / or based on analysis of an image of the room including the physical screen) a (potential or actual) exposure of the physical screen to additional users may be identified, and in response to the identified exposure of the physical screen to additional users, sharing of the virtual content with the physical screen may be stopped, or a prompt may be provided to the user sharing the virtual content. Such a prompt may include an indicator of the exposure and / or the additional users, and / or may be configured to enable the user sharing the virtual content to continue or stop sharing the virtual content.

[0206] In some embodiments, the privacy settings described above may be configured as part of the user's default settings. In some embodiments, causing the first virtual content to be displayed by the second wearable extended reality device may include modifying the first virtual content according to the default settings associated with the second wearable extended reality device. The default settings may be configured according to the preferences of the second wearable extended reality device user and / or the hardware available on the second wearable extended reality device and / or the environmental conditions associated with the second wearable extended reality device. For example, the second wearable extended reality device may operate at a lower or higher resolution than the first wearable extended reality device, and the rendering resolution of the first virtual content may be modified accordingly. In another example, the second wearable extended reality device may operate under ambient light conditions different from those of the first wearable extended reality device, and the intensity and / or opacity of the rendering of the first virtual content may be modified to compensate for the various ambient light conditions. In yet another example, the second wearable extended reality device may enable a different field of view than the first wearable extended reality device, and the size of the first virtual content may be modified to be adjusted to the different field of view. In some examples, the default settings may refer to how the wearable extended reality device is configured for use or how the wearable extended reality device is used. In some embodiments, the default settings may be configured to hide personal information such as passwords or credit card numbers, or alternatively, the default settings may include the user's preferred display parameters, i.e., where the content is displayed or how large the content appears.The default settings may be customizable by a user of the wearable extended reality device and may include, for example, privacy settings, proximity settings, gesture settings, or any other settings that can determine how the wearable extended reality device presents and / or shares virtual content.

[0207] In the above embodiments, the privacy settings can determine an initial privacy level associated with any virtual content that can be shared with another user. For example, the default privacy settings may allow virtual content to be shared with all users. As another example, the default privacy settings may prohibit virtual content from being shared with any other user. As another example, the default privacy settings may allow virtual content to be shared with a pre - defined group of users (e.g., the user's close relatives, or colleagues in a specific team or department associated with the user).

[0208] In the above embodiments, the sharing settings can determine how far away a user needs to be from another user before a sharing bubble is created, how large a pre - defined sharing ring is, and where this ring is presented, or the default physical object through which virtual content is presented. In the above embodiments, the gesture settings can determine which default user movements indicate an intention to share. For example, a user of a wearable extended reality device can configure the gesture settings so that content is automatically shared when the user waves a hand, but not when the user nods. Such privacy - sharing and gesture settings can be adjusted at any time according to the user's preferences and what the desired use of the wearable extended reality device is.

[0209] Some of the disclosed embodiments may include receiving second virtual content from a second wearable extended reality device for display via a first wearable extended reality device. The receiving may include the transmission of the content in any form. For example, the content may be received through direct communication of the content, such as through communication of network-based content or through other direct communication protocol connections between Bluetooth, Wi-Fi, NFC, or multiple extended reality devices or modems associated therewith. In some embodiments, a processor may receive virtual content from a second wearable extended reality device and instruct the first wearable extended reality device to display the virtual content. In some other examples, a processor may be able to receive virtual content from a second wearable extended reality device and, for example, provide the virtual content to the first wearable extended reality device for display of the virtual content. As described elsewhere herein, there are many ways in which a first wearable extended reality device may receive virtual content from a second wearable extended reality device. For example, the second wearable extended reality device may be able to share virtual content with the first wearable extended reality device via a shared bubble or a shared ring. As another example, the second wearable extended reality device may be able to share virtual content with the first wearable extended reality device as soon as a link between the two devices is established.

[0210] Some of the disclosed embodiments may include displaying multiple versions of virtual content. In some embodiments, the first virtual content can include two-dimensional virtual objects, and the second virtual content can include alternative versions of the two-dimensional objects. Alternative versions of the two-dimensional objects can include, for example, a scaled-down version of the two-dimensional object, an enlarged version of the two-dimensional object, a two-dimensional object having a different position or orientation, a two-dimensional object having a different color, texture, or shading, or a two-dimensional object whose some or all of its characteristics or functions are changed in some way. Alternative versions of the two-dimensional objects can be generated based on the preferences of the browsing user.

[0211] For example, the two-dimensional object may be a virtual document, and the alternative version of the two-dimensional virtual object may be an annotated or highlighted version of the virtual document. As another example, the two-dimensional object may be a presentation, and the alternative version of the two-dimensional object may have modified colors, graphics, and / or shapes in the presentation. As another example, the multiple versions of the virtual content can be composed of both public virtual objects and private virtual objects. For example, a first user of a first wearable extended reality device can send a document to a second user, and the second user can share the document with a larger group. However, some of the comments and annotations on the virtual object can remain hidden based on the privacy and / or document sharing settings of the first or second user.

[0212] In some embodiments, the first virtual content can include a 3D virtual object, and the second virtual content can include a 2D virtual object associated with the 3D object. When a user of a wearable extended reality device shares content with another user, such content can include multiple virtual objects. As described in the above embodiments, these virtual objects can include both 2D objects and 3D objects and can receive various privacy settings. The 3D virtual object can be a design model or a project mock-up, and the 2D virtual object associated with the 3D object can be a list of comments on the model. Similar to the 2D model, subsequent versions of the 3D virtual object can be configured to have various privacy settings.

[0213] For example, a project mock-up such as a bridge, building, or other scale or architectural model can be shared as a 3D virtual object with another user. A list of comments regarding the size, length, width, and any number of other arbitrary parameters related to the project mock-up, i.e., a 2D virtual object, can be associated with the project mock-up. In addition to the 2D object associated with the 3D object, the user can also include the 3D virtual object in the revised version. For example, in the case of a project mock-up, a user of a wearable extended reality device can share the first version of the mock-up and also share a revised version of the model that can have more components, a different size, or a different texture than the first version.

[0214] In response to receiving virtual content from a second wearable extended reality device, some disclosed embodiments may include presenting, via a first extended reality device, second virtual content received from the second wearable extended reality device. There are many ways in which virtual content can be shared among wearable extended reality device users. The content can be shared via a shared bubble or a shared ring. A second wearable extended reality device user can share content with a first wearable extended reality device user via the shared bubble or the shared ring. In the above embodiments, a first wearable extended reality device user can present the content received from the second user. This content can be presented in countless ways, for example, through the extended reality device of the second user, via a screen such as a tablet or a TV, or via a physical object such as a whiteboard or a chalkboard. In one embodiment, after a first user receives virtual content from a second user, the first user can present the content via the first device or via a TV or a whiteboard. A first wearable extended reality device user can also present the content received from the second user to other users near the first user.

[0215] As an example, FIGS. 13A and 13B show a shared bubble, where a first wearable extended reality device 1310 receives second virtual content 1314 from a second wearable extended reality device 1312 via a shared area 1316. Such virtual content can also be shared via a shared ring. FIG. 13B shows an example of presenting the received content 1314. Here, the first wearable extended reality device 1310 can present the second virtual content 1314 received from the second wearable extended reality device 1312. The virtual content can be presented via a wearable extended reality device, a screen such as a tablet or a TV, or any other physical object.

[0216] In another example, FIG. 14 shows a flowchart of an exemplary method 1410 for coordinating between a first wearable extended reality device and a second wearable extended reality device to display various virtual contents. Method 1410 can include step 1412 where at least one processor can establish a link between the first wearable extended reality device and the second wearable extended reality device, for example, for communication between the two devices or for control of the two devices. Method 1410 can also include step 1414 where the first virtual content can be presented to the first wearable extended reality device before, after, or at the same time as establishing a link with the second wearable extended reality device. Method 1410 can also include step 1416 which includes obtaining a command for displaying the first virtual content via the second wearable extended reality device. Method 1410 can also include step 1418 where, in some embodiments, the first virtual content can be caused to be transmitted for display to the second wearable extended reality device. Method 1410 can also include step 1420 where a second virtual content can be received from the second wearable extended reality device. Method 1410 can include step 1422 where the first wearable extended reality device can display the second virtual content.

[0217] Some of the disclosed embodiments can include operations for providing situation awareness to a user of a wearable extended reality device. Situation awareness can include one or more of the perception of elements in the environment, the understanding of the current situation, the prediction of future states, or any ability to identify, process, or understand elements of information regarding what is occurring in the environment. For example, situation awareness can include an understanding of one or more of location, area, point, geography, region, scene, setting, site, surroundings, topography, section, angle, inclination, dimension, quantity, width, capacity, content, diameter, height, intensity, length, size, ratio, range, volume, width, amplitude, age, date, presence, time, moment, opportunity, or any other condition associated with any part of the environment. In some examples, situation awareness can include an understanding of the state of another person within the user's environment. For example, situation awareness can include an understanding of whether another person is engaging with virtual content (such as a virtual display, virtual avatar, etc.).

[0218] Some embodiments can display virtual content through a first wearable extended reality device. The term virtual content can include any type of data representation that can be presented to a user, such as through an extended reality device or other presentation device. Virtual content can include virtual objects, non-animated virtual content, animated virtual content configured to change over time or in response to a trigger, virtual two-dimensional content, virtual three-dimensional content, virtual overlays on a part of the physical environment or on physical objects, virtual additions to the physical environment or physical objects, virtual promotion content, virtual representations of physical objects, virtual representations of the physical environment, virtual documents, virtual characters or personas, virtual computer screens, virtual widgets, or any other form for virtually presenting information. Consistent with the present disclosure, virtual content can include any visual presentation rendered by a computer or processing device. In one embodiment, the virtual content is rendered by a computer within a limited area and includes virtual objects that are visual presentations configured to represent a particular type of object (such as non-animated virtual objects, animated virtual objects, virtual furnishings, virtual decorative objects, virtual widgets, or other virtual representations). The rendered visual presentation can be changed, for example, to mimic a change in the appearance of a physical object or to reflect a change to a status object or a change in the viewing angle of an object.

[0219] Some embodiments may include detecting a second wearable extended reality device in proximity to a first wearable extended reality device. Proximity may refer to nearness, adjacency, simultaneity, continuity, affinity, approachability, or any other state or condition of being near. Proximity may be measured by any of a capacitive effect, an inductive effect, a magnetic effect, or an optical effect. Proximity may also be detected using, for example, one or more of radar, sonar, ultrasonic, optical fiber, or Hall effect technology, reflection of ionizing radiation, or any other technology capable of detecting the presence of nearby objects. Proximity may include distance, direction, or a combination of distance and direction. In other examples, the locations of the first and second wearable extended reality devices may be based on positioning data corresponding thereto. In one example, this location may be determined, for example, using indoor or outdoor positioning sensors included in each of the two wearable extended reality devices, from the localization of the two wearable extended reality devices in an image (e.g., an image captured using an image sensor in the environment of the two wearable extended reality devices and analyzed using an object detection algorithm to localize the wearable extended reality devices). In other examples, an image captured using an image sensor included in the first wearable extended reality device may be analyzed using an object detection algorithm to detect the second wearable extended reality device, and the distance between the two wearable extended reality devices may be determined, for example, based on the size of the second wearable extended reality device in the captured image, thereby determining whether the second wearable extended reality device is in proximity to the first wearable extended reality device.In other examples, an image captured using an image sensor included in a second wearable extended reality device may be analyzed using an object detection algorithm to detect a first wearable extended reality device, and the second wearable extended reality device may be considered to be in proximity to the first wearable extended reality device when the first wearable extended reality device is detected within the image captured by the second wearable extended reality device (and thus within the line of sight of the second wearable extended reality device).

[0220] In some embodiments, detecting a second wearable extended reality device in proximity to a first wearable extended reality device may be continuous. In some embodiments, detecting a second wearable extended reality device in proximity to a first wearable extended reality device may occur at regular or irregular time intervals. In some embodiments, detecting a second wearable extended reality device in proximity to a first wearable extended reality device may be triggered by an input. The input may be received from a user, a system, or any other information source that may trigger detection of proximity.

[0221] In certain embodiments, detecting a second wearable extended reality device proximate to a first wearable extended reality device may include detecting that the second wearable extended reality device is within a particular distance, width, height, radius, or other measure of separation from the first wearable extended reality device. For example, detecting a second wearable extended reality device proximate to a first wearable extended reality device may include detecting that the second wearable extended reality device is within 1 foot, 2 feet, 5 feet, or any distance from the first extended reality device. The measure of separation used to detect proximity may be input by a user, the system, or determined from any other source of information. For example, a user may input a value of 5 feet to trigger detection of proximity. In another embodiment, the system may include a default setting for triggering detection of proximity when a distance of 5 feet is detected between the first wearable extended reality device and the second wearable extended reality device. In one example, the measure of separation used to detect proximity may vary for different directions (e.g., one measure of separation for horizontal distance and a smaller measure of separation for vertical distance, or in another example, one measure of separation for upward distance and a smaller measure of separation for downward distance). In some examples, the second wearable extended reality device may be considered proximate to the first wearable extended reality device when the two wearable extended reality devices are in the same defined space (e.g., the same room, the same apartment, the same office, the same building, etc.). In some examples, the second wearable extended reality device may be considered proximate to the first wearable extended reality device when the first wearable extended reality device is within the line of sight of the second wearable extended reality device.

[0222] In some embodiments, proximity detection can be based on any reference point of each of the first wearable extended reality device and the second wearable extended reality device. For example, proximity can be detected based on the distance from any physical location of the wearable extended reality device. Alternatively, proximity can be detected based on the distance from the virtual location of a virtual object presented through the wearable extended reality device. In another example, proximity can be detected based on the distance from a reference point within a specific range of the wearable extended reality device.

[0223] Some embodiments may include establishing a link between the first wearable extended reality device and the second wearable extended reality device. The link can include one or more of a physical or non-physical attachment, or any other connection mode. In some embodiments, the link can be established using a physical attachment, including, for example, a wire, cable, or any other type of attachment means that requires a tangible connection between the first wearable extended reality device and the second wearable extended reality device. In some embodiments, the link can be established using non-physical means and can include wireless communication such as Bluetooth, ZigBee, Wi-Fi, or any other technology that implements the transfer of information between two or more points without using a conductor as a medium for the transfer.

[0224] Furthermore, the link can be a direct or indirect link between a first wearable extended reality device and a second extended reality device. A direct link between the first wearable extended reality device and the second wearable extended reality device can be an unobstructed connection between the first wearable extended reality device and the second wearable extended reality device. Alternatively, an indirect link between the first wearable extended reality device and the second wearable extended reality device can be an obstructed connection between the first wearable extended reality device and the second wearable extended reality device. This type of link can include a connection that connects a system connected to one or both of the first wearable extended reality device and the second wearable extended reality device. Such a system can include, for example, a laptop, a smartphone, or any other type of non-wearable device that can be connected to a wearable extended reality device. In one example, the link between the first wearable extended reality device and the second wearable extended reality device can include a direct and / or indirect communication link between the two extended reality devices. In another example, a system (such as a centralized system) can communicate with each of the first and second wearable extended reality devices to, for example, provide content for presentation to each of the two wearable extended reality devices and receive location and / or orientation information associated with the wearable extended reality devices. The link between the first wearable extended reality device and the second wearable extended reality device can include a linkage within a data structure or database of the system or maintained by the system.In some examples, establishing a link between a first wearable extended reality device and a second wearable extended reality device can refer to a linkage in a data structure and / or database in, for example, a system that adjusts an extended reality environment, a system that communicates with both the first and second wearable extended reality devices, a system that provides content for presentation to both the first and second wearable extended reality devices, and the like.

[0225] Some embodiments may include transmitting data representing at least a portion of virtual content in an obfuscated form to a second wearable extended reality device, where the obfuscated form provides an indication of the location of at least a portion of the virtual content in three-dimensional space without revealing the identity of the obfuscated form of the virtual content. In one example, the data may be transmitted from a first wearable extended reality device to a second wearable extended reality device, for example, through a direct or indirect communication link. In this example, the first wearable extended reality device may be able to process at least a portion of the virtual content to generate the obfuscated form, as described below. In another example, the data may be transmitted from a computerized system (e.g., a system that coordinates an extended reality environment, a system that communicates with both the first and second wearable extended reality devices, a system that provides content for presentation to both the first and second wearable extended reality devices, etc.) to the second wearable extended reality device. In this example, the computerized system may be able to process at least a portion of the virtual content to generate the obfuscated form, as described below. The obfuscated form may include presenting any portion of the virtual content in a way that obscures that portion of the virtual content. In certain embodiments, the obfuscated form that provides an indication of the location of at least a portion of the virtual content in three-dimensional space without revealing the identity of the virtual content may include presenting at least a portion of the virtual content by masking, editing, omitting, pixelating, blurring, or any other means of at least partially hiding any portion of the virtual content. In some examples, the obfuscated form may be able to provide a visual indication of the location of at least a portion of the virtual content in three-dimensional space and / or a visual indication of the type of at least a portion of the virtual content without revealing at least one detail of at least a portion of the virtual content.For example, an obfuscated form can provide a visual indication that at least a portion of the virtual content includes text content without revealing the exact words and / or exact characters of the text content. In another example, an obfuscated form can provide a visual indication that at least a portion of the virtual content includes graphic content without revealing the exact image and / or exact graphic of the graphic content.

[0226] The functionality of at least one processor as defined by the instructions is illustrated in FIGS. 15 and 16. For example, FIG. 15 is a diagram of an exemplary environment including a plurality of users using a plurality of wearable extended reality devices, consistent with some embodiments of the present disclosure. As shown in FIG. 15, a first user 1514 of a first wearable extended reality device 1515 may be located at a first location 1510. A second user 1516 of a second wearable extended reality device 1517 may be located at a second location 1512 that is not proximate to the first location 1510 of the first user 1514 (e.g., at a distance longer than a selected threshold, not in the same defined space, not in the same room, not in the line of sight, etc.). The first content 1522 may be presented to the first user 1514 in a first virtual display 1518, and the second content 1524 may be presented to the second user 1516 in a second virtual display 1520. Since the first wearable extended reality device 1515 is not proximate to the second wearable extended reality device 1517, none of the first virtual content 1522 can be transmitted to the second user 1516.

[0227] FIG. 16 is a diagram of an exemplary virtual display including a portion of a virtual display provided in an obscured form that is consistent with some embodiments of the present disclosure. As shown in FIG. 16, a first user 1614 of a first wearable extended reality device 1615 may be located at a first location 1610. A second user 1616 of a second...

Claims

1. A non-transitory computer-readable medium including instructions that, when executed by at least one processor, cause the at least one processor to perform operations for presenting virtual content to a plurality of viewers, the operations including: Receiving sensor data indicating a plurality of wearable extended reality devices located within a room; Receiving a command for sharing a virtual object with the plurality of wearable extended reality devices; Analyzing the sensor data to determine a first location within the room of a first wearable extended reality device, a second location within the room of a second wearable extended reality device, and a third location within the room of a third wearable extended reality device; Determining a position for displaying the virtual object within the room based on the determined first location, the determined second location, and the determined third location; Causing a first display of the virtual object to occur at the determined position through the first wearable extended reality device, the first display being rendered from a first perspective; Causing a second display of the virtual object to occur at the determined position through the second wearable extended reality device, the second display being rendered from a second perspective different from the first perspective; Causing a third display of the virtual object to occur at the determined position through the third wearable extended reality device, the third display being rendered from a third perspective different from the first perspective and the second perspective; Including; The operations further include determining lighting conditions within the room, and the position for displaying the virtual object is determined based on the lighting conditions and the determined first location, the second location, and the third location. A non-transitory computer-readable medium.

2. The operation further includes analyzing the sensor data to determine that at least one of the first wearable extended reality device, the second wearable extended reality device, or the third wearable extended reality device is virtually located in the room, and the determined location of the at least one wearable extended reality device virtually located in the room reflects the location of an avatar of at least one user. The non-transitory computer-readable medium according to claim 1.

3. The determined position for displaying the virtual object includes the determined orientation of the virtual object in the room, and the operation further includes analyzing additional sensor data to determine that at least two of the first wearable extended reality device, the second wearable extended reality device, or the third wearable extended reality device have changed their orientation, and adjusting the determined orientation of the virtual object is based on the change in orientation. The non-transitory computer-readable medium according to claim 1.

4. The determined position for displaying the virtual object includes the determined location of the virtual object, and the operation further includes analyzing additional sensor data to determine that at least two of the first wearable extended reality device, the second wearable extended reality device, or the third wearable extended reality device have changed their location, and adjusting the determined location of the virtual object is based on the change in location. The non-transitory computer-readable medium according to claim 1.

5. The operation further includes determining the identity of at least one of the users of the first wearable extended reality device, the second wearable extended reality device, or the third wearable extended reality device, and determining the position for displaying the virtual object is further based on the identity of the at least one user. The non-transitory computer-readable medium according to claim 1.

6. The operation further includes determining physical characteristics of at least one user of the first wearable extended reality device, the second wearable extended reality device, or the third wearable extended reality device, and determining the position for displaying the virtual object is further based on the physical characteristics of the at least one user. The non-transitory computer-readable medium according to claim 1.

7. The operation further includes determining a layout of the room including one or more physical locations of furnishings in the room, and the position for displaying the virtual object is further determined based on the layout of the room. The non-transitory computer-readable medium according to claim 1.

8. The operation further includes determining a type of the virtual object, and the position for displaying the virtual object is further determined based on the type of the virtual object. The non-transitory computer-readable medium according to claim 1.

9. The operation further includes analyzing the sensor data to identify physical objects in the room, and the position for displaying the virtual object is determined such that none of the first display, the second display, and the third display is blocked by the physical object. The non-transitory computer-readable medium according to claim 1.

10. The operation further includes analyzing the sensor data to identify physical objects in the room, and the position for displaying the virtual object is determined such that none of the first display, the second display, and the third display blocks the physical object. The non-transitory computer-readable medium according to claim 1.

11. The first display of the virtual object, the second display of the virtual object, and the third display of the virtual object are associated with a single version of the virtual object, and the operations further include detecting changes to the virtual object introduced by the user of the first wearable extended reality device, and updating the second display and the third display to reflect the changes to the virtual object introduced by the user of the first wearable extended reality device. The non-transitory computer-readable medium according to claim 1.

12. The first display of the virtual object, the second display of the virtual object, and the third display of the virtual object are associated with various versions of the virtual object, and the operations further include obtaining a profile associated with the user of the first wearable extended reality device, a profile associated with the user of the second wearable extended reality device, and a profile associated with the user of the third wearable extended reality device, and determining a personalized version of the virtual object based on the profiles associated with each user. The non-transitory computer-readable medium according to claim 1.

13. While the virtual object is displayed at the determined position through the first wearable extended reality device and the second wearable extended reality device, the operations further include receiving additional sensor data indicating a change in the status of the third wearable extended reality device, and repositioning the virtual object based on the determined first location of the first wearable extended reality device and the determined second location of the second wearable extended reality device. The non-transitory computer-readable medium according to claim 1.

14. While the virtual object is displayed at the determined position, the operation further includes receiving additional sensor data indicating that a fourth wearable extended reality device is in the room, determining a fourth location of the fourth wearable extended reality device, and repositioning the virtual object based on the determined first location, the determined second location, the determined third location, and the determined fourth location. The non-transitory computer-readable medium according to claim 1.

15. The operation is executed by the first wearable extended reality device, generating the first display includes generating a display signal, generating the second display includes transmitting data reflecting the virtual object to the second wearable extended reality device, and generating the third display includes transmitting data reflecting the virtual object to the third wearable extended reality device. The non-transitory computer-readable medium according to claim 1.

16. The virtual object is a virtual display configured to present text input using an input device, and determining the position for displaying the virtual display is further based on at least one of the location or orientation of the input device. The non-transitory computer-readable medium according to claim 1.

17. The virtual object is a virtual display configured to present text input using a physical keyboard, the physical keyboard is positioned on a physical surface, and determining the position for displaying the virtual display is further based on the characteristics of the physical surface. The non-transitory computer-readable medium according to claim 1.

18. A method for presenting virtual content to a plurality of viewers, the method comprising: Receiving sensor data indicating a plurality of wearable extended reality devices located in a room; Receiving a command for sharing a virtual object with the plurality of wearable extended reality devices; Analyzing the sensor data to determine a first location in the room of the first wearable extended reality device, a second location in the room of the second wearable extended reality device, and a third location in the room of the third wearable extended reality device; Determining a position for displaying the virtual object in the room based on the determined first location, the determined second location, and the determined third location; Causing a first display of the virtual object to occur at the determined position through the first wearable extended reality device, wherein the first display is rendered from a first perspective; Causing a second display of the virtual object to occur at the determined position through the second wearable extended reality device, wherein the second display is rendered from a second perspective different from the first perspective; Causing a third display of the virtual object to occur at the determined position through the third wearable extended reality device, wherein the third display is rendered from a third perspective different from the first perspective and the second perspective; Including; Further including determining lighting conditions in the room, wherein the position for displaying the virtual object is determined based on the lighting conditions and the determined first location, second location, and third location.

19. A system for presenting virtual content to a plurality of viewers, the system comprising: Comprising at least one processor, the at least one processor: Receiving sensor data indicating a plurality of wearable extended reality devices located in a room; Receiving a command for sharing the virtual object with the plurality of wearable extended reality devices; Analyze the sensor data to determine a first location in the room of the first wearable extended reality device, a second location in the room of the second wearable extended reality device, and a third location in the room of the third wearable extended reality device; Based on the determined first location, the determined second location, and the determined third location, determine a position for displaying the virtual object in the room; Cause a first display of the virtual object to occur at the determined position through the first wearable extended reality device, wherein the first display is rendered from a first perspective; Cause a second display of the virtual object to occur at the determined position through the second wearable extended reality device, wherein the second display is rendered from a second perspective different from the first perspective; Cause a third display of the virtual object to occur at the determined position through the third wearable extended reality device, wherein the third display is rendered from a third perspective different from the first perspective and the second perspective; Perform; Further perform determining lighting conditions in the room, and the position for displaying the virtual object is programmed to be determined based on the lighting conditions and the determined first location, second location, and third location.

Images